Researchers from Microsoft and the City University of Hong Kong developed Taste-Bench to evaluate the decision-making quality of AI agents during long-term tasks. Unlike existing benchmarks that only measure final task completion, this tool assesses an agent's ability to choose optimal paths at critical decision points, or 'decision forks'.
The benchmark consists of 502 questions derived from thousands of coding and research task runs. It presents models with two potential paths, requiring them to identify the superior one based on historical data. The dataset includes parallel paths from independent attempts and correction-based detours where agents previously failed.
Testing 14 models from families including GPT, Claude, Grok, and Mistral revealed significant limitations. The top-performing model, GPT-5.6 Sol, achieved an accuracy of 59.7 percent, while most others performed significantly worse. Accuracy dropped to 21 percent when the evidence for a correct decision appeared only after extensive subsequent work.
The study demonstrated that 'taste' can be improved through training. By distilling judgment from a teacher model into a Qwen3.6-27B student model, researchers increased accuracy on unseen tasks from 30 percent to 47.9 percent. Integrating this student model as an advisor improved the success rate on SWE-bench Pro tasks from 14.6 percent to 33.7 percent.
