WebPageBench is an open-source web benchmark designed to evaluate AI agents on six simulated, Russian-language websites. The platform tests agents on typical user tasks such as booking hotels, ordering groceries, or purchasing train tickets. Success is determined by analyzing event logs generated by the simulated sites rather than relying on model self-reporting or external judge models.
The benchmark includes 152 tasks, consisting of 65 manually created scenarios and 87 variations that test different UI implementations, such as dark mode or alternative input controls. The simulated sites are anonymized and feature neutral branding, with synthetic data used for catalogs, including 226 book titles and 277 grocery items. Images are sourced from open-license repositories.
Each task execution creates an isolated environment where every user action is recorded as a typed event with specific parameters. The primary metric, Event-Match Score (EMS), evaluates whether all task conditions are met. The system also tracks a 'Completion' metric to compare agent self-reporting against actual performance, revealing discrepancies in agent reliability.
The framework supports 33 UI variants across nine aspects, including date pickers, city selection, and guest counters. This allows developers to determine if an agent fails due to a lack of understanding or an inability to interact with specific interface elements. Currently, the public leaderboard features 24 model and harness combinations, with Gemini-3.8-flash paired with OpenManus showing high performance.
