The initial RAG system for legal documents relied on a standard pipeline: document chunking, embeddings, vector search, and LLM processing. However, this approach struggled with precise data retrieval, such as specific contract numbers, dates, or names, where semantic search often failed to identify the correct document.
To improve accuracy, the developer implemented a hybrid search combining BM25 for exact matches and BGE-M3 for semantic relevance, using Reciprocal Rank Fusion to merge results. On a test set of 401 documents, this configuration achieved a Recall@10 metric of 0.935. An attempt to use a reranker failed to improve quality, reducing Recall@10 to 0.806 or 0.871 depending on the method, and was subsequently removed.
The system now incorporates a routing layer that directs queries to either SQL databases for structured data or the RAG pipeline. This routing is governed by deterministic rules, which currently pass 21 out of 21 regression tests. To ensure transparency, each query is assigned a trace_id to monitor the processing path and identify where potential errors occur.
The architecture separates fact extraction from decision-making. LLMs are used to extract facts and generate natural language responses, while deterministic code applies business rules to evaluate risks or contract terms. This approach prevents the model from hallucinating or inconsistently interpreting legal conditions, as critical numerical values are validated against source data.
The developer emphasizes that complex techniques like reranking or query rewriting do not guarantee performance improvements. Instead, components are only integrated if they solve a measurable problem. In cases where data is insufficient or contradictory, the system is designed to defer to human judgment rather than forcing an automated verdict.
