Datalab has introduced OmniParseBench, an open-source evaluation framework designed to audit OCR document parsers. The tool contains 16,288 pass/fail tests and operates offline, requiring Python 3.11 or higher. It does not utilize API keys for scoring, as it relies on open-source code rather than model-based evaluation.
The benchmark calculates a final score based on the mean of three categories: text, tables, and layout. These are derived from five test types, including digit-for-digit number matching and KaTeX-based equation comparison. Each test includes metadata tags across five axes, such as language and structure, allowing for granular performance analysis.
The dataset includes 6,767 table-related tests across 1,000 pages and 10,089 English-language tests. Non-English content covers 1,931 tests from 321 documents. Answers are verified through 10 derivation methods, including JATS XML, PDF text layers, and human review.
Performance results indicate that Claude Sonnet 5.5 achieved top scores in text and tables, while Reducto led in layout performance. Datalab also provides tools to compare models based on latency and cost per 1,000 pages, measured using a sample of 200 pages.
