Researchers developed a custom benchmark to evaluate LLMs for an AI assistant tasked with querying a laboratory database. The test set consists of 24 real-world questions categorized into navigation, data extraction, reasoning, traps, and honesty. The evaluation criteria require models to provide correct answers, cite sources, and demonstrate honesty by admitting when data is missing or unreliable.
The study tested 19 combinations of models and reasoning efforts across 450 runs. Results varied from 18 to 24 correct answers out of 24. Two configurations achieved a perfect score: Claude 3.5 Opus on high effort and DeepSeek Flash 4.1 on max effort. The cost for the full test set ranged from $0.12 to $4.94 per run.
DeepSeek models proved cost-effective, with the Flash 4.1 model on max effort achieving a perfect score for $0.63. In contrast, Claude 3.5 Opus on high effort reached the same score for $3.19 but demonstrated faster performance, with a median response time of 24 seconds compared to 2.5 minutes for the DeepSeek configuration.
The researchers emphasized that lower reasoning efforts often led to models stopping prematurely, while higher efforts improved performance in complex tasks. They recommended using fresh sessions for each question, logging all tool calls, and including specific trap questions to ensure the model does not hallucinate data or violate read-only constraints.
