Agentic Lab engineers developed a specialized AI model for legal document analysis, specifically for extracting entities like contract details, payment events, and breach claims. The team opted for an on-premise solution to ensure data security and compliance with Russian regulations regarding personal data storage.
The new model, based on Qwen2.5-1.5B with a LoRA adapter, was trained on a dataset of 5,500 real legal documents. It achieves a macro-F1 score of 0.70, performing nearly on par with the Ministral 3 14B model, which scores 0.71. The model runs on A5000 hardware, operating 2.2 times faster than the previous setup on A100.
Initial attempts to balance the model using synthetic data failed because the team focused on balancing entity classes rather than document types. By generating synthetic procedural documents like claims and petitions, the team successfully addressed performance gaps, specifically increasing the accuracy of performance event extraction by 23 points.
The development was driven by the need to avoid sending sensitive legal materials to external APIs. Large models like Gemma 4 28b were found to consume excessive VRAM on A100 cards, hindering the performance of other concurrent tasks. The resulting compact model allows for efficient local processing without compromising data privacy.
