Yandex and the School of Data Analysis have released MTGBench, a new benchmark designed to evaluate how neural networks handle complex tasks requiring sequential reasoning. The tool is now available on GitHub, and the accompanying research paper has been accepted for the EMNLP 2026 conference.
The benchmark utilizes seven text-based games, such as Wordle, Alias, and variations of guessing games, to test a model's ability to plan actions, incorporate new information, and avoid contradictions. Scoring is based on both the final accuracy and the number of steps taken to reach the solution, with all tasks provided in English and Russian.
Researchers tested 16 models, including GPT, Gemini, and Qwen. The results indicate that reasoning modes significantly improve performance in tasks involving symbols and hints, while gains in knowledge extraction were less consistent. Furthermore, models generally performed better in English than in Russian.
Despite language-specific performance gaps, top-tier models using reasoning modes outperformed human participants across all seven games. The developers intend for MTGBench to serve as a standard tool for researchers and developers to compare model capabilities and select optimal solutions for specific applications.
