The updated LAB-AA benchmark, version 1.1, now incorporates hallucination checks to evaluate whether AI-generated legal outputs are supported by source documents. This metric complements existing criteria that assess model performance against expert-written rubrics.
The benchmark is managed by the evaluation firm Artificial Analysis and utilizes a private set of 120 tasks. This approach prevents models from being tuned specifically to pass known questions, ensuring the integrity of the leaderboard.
As of October 2026, top models on the leaderboard achieve criterion pass rates of 93–95%, while all-pass rates remain below 30%. The new update aims to identify instances where models satisfy rubric criteria but include unsupported claims.
Harvey originally open-sourced its Legal Agent Benchmark (LAB) on May 6, 2026, featuring over 1,200 tasks and 75,000 rubric criteria. Separately, Harvey released lab-core v1.1.0 on September 18, 2026, which introduced dual-judge evaluations using Claude Sonnet 4.6 and GPT-5.5.
Harvey reports that its Harvey Assistant has a hallucination rate of 0.2%, compared to 0.7–1.9% for foundation models. These figures are provided by the company and reflect the potential impact of error rates in legal workflows.
