Researchers from Stanford University and Google DeepMind have introduced TURTLE, a system designed to evolve AI benchmarks to keep pace with improving model capabilities. The project, detailed in an arXiv paper dated October 19, 2026, focuses on tool-use tasks where AI agents operate software across multiple steps.
TURTLE utilizes three agents: a proposer based on DeepMind’s AlphaEvolve, a solver model, and a feedback generator. The system modifies existing tasks by adding filters or extra steps while ensuring they remain solvable. A task is considered successful if the solver achieves at least a 50% success rate over a minimum of 10 attempts.
The team tested the system on benchmarks including AppWorld, τ-Bench, and AutomationBench using the Gemma 4 31B-it model. In AppWorld, the share of hard tasks increased by 165%, rising from 19.8% in the original set. Nearly all of the 735 evolved tasks resulted in solver success rates between 40% and 60%.
By automating the creation of multi-step tasks, TURTLE aims to reduce the manual labor required for benchmark development. The system ensures that tasks remain within a specific difficulty range, allowing for more precise measurement of model improvements as capabilities shift over time.
