OmniParseBench is a benchmarking tool for document parsing, currently available in version 1.0.0rc1. The benchmark includes a dataset containing 16 288 tests across various document types, including PDF files and images.
The evaluation framework utilizes a scoring system based on unit tests that assess document structure, text content, and layout. The benchmark categorizes tests into families such as text, tables, and layout, using specific tags for languages and scripts.
Developers can use the tool to validate parser performance by comparing model outputs against gold-standard files. The system supports various test types, including exact text matching, mathematical equation verification, and table cell extraction.
The dataset is hosted on Hugging Face and is version-controlled via specific commits to ensure consistency as new parsers are added. The framework allows for granular testing, where individual changes in document content trigger specific verdict updates.
