An experiment tested 16 AI models and 7 search tools to evaluate their ability to retrieve accurate corporate data. The study focused on Bitrix24, using a seven-point checklist that included legal details, revenue, key personnel, pricing, product updates, webinars, and media mentions.
The research revealed that model performance varies significantly based on how they handle uncertainty. Some models generate confident but false information, while others correctly identify gaps in their knowledge. Gemini Flash demonstrated superior internal knowledge compared to other models when tested without internet access.
Search tool efficiency proved critical for accuracy. Exa outperformed other tools by providing full-text content, whereas DuckDuckGo and SearXNG relied on limited snippets, often leading to incomplete or inaccurate data extraction. The study also highlighted the risk of agents retrieving data from typosquatted domains.
Testing showed that search results are not always reproducible. A single model paired with the same search tool produced score variations of 3 to 6 points across multiple runs. Models like MiMo achieved high accuracy with zero confident hallucinations, while others relied on aggressive speculation to boost their scores.
Prompt engineering attempts to improve performance, such as adding specific instructions for webinar searches, proved unstable. Enhancing results for one category often led to a decline in overall completeness. The experiment concluded that domain validation is essential before deploying agents in production.
