Scale Labs and the startup founded by a former Google DeepMind researcher have launched Humanity's Sixth Sense. This benchmark evaluates how AI models interpret implicit signals in visual data. The test includes 522 tasks, comprising 288 static images and 234 video clips, focusing on intuitive human judgments like spatial reasoning or social dynamics.
Testing involved 25 current multimodal models, which showed a significant performance gap compared to humans. While human participants achieved 93.1% accuracy, the top-performing model, GPT-6 Astra, reached only 53.6%. Other models like GPT-6.1 Sol and Claude Opus 5.5 scored 46.6% and 44.6% respectively, with the median accuracy across all tested models at 30.9%.
An analysis of over 8,500 incorrect responses revealed that 94% of errors stem from computer vision limitations, specifically the failure to detect visual markers or maintain context. Only 5% of errors were attributed to flawed reasoning. Increasing computational resources to 4,000 tokens per response did not improve performance, and in some cases, it led to lower accuracy.
The researchers concluded that scaling logical inference is insufficient to overcome fundamental architectural constraints in computer vision. Techniques such as cropping frames to simulate focus provided only minimal improvements, suggesting that current approaches to training multimodal models require significant adjustments.
