Researchers tested AI agents on 50 unpublished theoretical physics problems. GPT-6-Astra successfully solved 49 of them, achieving a 98% accuracy rate. In several instances, the researchers initially misidentified the AI's correct answers as errors, only to discover mistakes in their own calculations after consulting with experts.
The test compared several models. Opus 5.5 solved 46 tasks, Gemini 3.8 Flash completed 34, DeepSeek-V4.1-Flash handled 27, and GLM 5.3 solved 21. The cost for GPT-6-Astra to process each task averaged less than $3. The only problem Astra failed to solve remained unsolved by all other models due to ambiguous conditions.
The models maintained similar accuracy levels even when internet access was disabled. This suggests that the AI has internalized a vast amount of scientific literature. However, the study notes that these models still struggle to identify significant research problems or determine when to pivot their focus, despite their proficiency in solving defined tasks.
