Meta FAIR, the University of Oxford, and University College London have developed AI Research Preference Models (RPMs) to optimize machine learning experiments. Instead of predicting absolute scores, RPMs rank unexecuted candidates and select the one most likely to succeed before GPU resources are spent.
The system utilizes frozen pretrained LLMs without fine-tuning, using Qwen3.6-27B as the backbone. The open-source scaffold AIRA-dojo and benchmark AIRS-Bench are also provided. AIRA-dojo employs an evolutionary tree search with Draft, Improve, and Debug operators, where the RPM selects the winner of a knockout tournament among 15 parallel candidates.
Two versions of the model are presented: Inference-only RPM, which acts as an LLM-as-a-judge over plans and code with an offline accuracy of 57.7% to 59.0%, and Agentic RPM, which uses a sandbox with a single H200 GPU. The Agentic RPM runs small-scale pilot experiments capped at 30 with a 60-second threshold.
The setup included 20 public text and tabular tasks, with 24 hours on a single H200 per task. Results show a probability of improvement over No-RPM of 0.5923 and 0.5913. In terms of efficiency, Inference-only RPM reached the baseline's final score of 0.684 in 14.88 hours (1.61x speedup), while Agentic RPM did the so in 15.50 hours (1.55x speedup).
The approach achieved two new state-of-the-art (SOTA) results: 94.1% on WinoGrande, surpassing the previous agentic SOTA of 90.4% from AIRA2, and 95.7% on SVAMP, beating the previous human SOTA of 94.2%.
