Artificial Analysis has updated its Intelligence Index, resulting in a significant shift in model rankings. The Kimi K3.1 Pro model, which initially debuted in fourth place, has dropped to ninth position. It now trails behind eight closed-source models in the updated version 5 of the index.
The performance gap between open-source models and the top of the leaderboard has widened to 12 points, compared to the previous 3-point difference. Additionally, the Grok 4.7 model experienced a 12-point decline in its score following the methodology changes.
For the first time, 60 percent of the index is based on private test sets. This measure aims to combat benchmaxxing, where models are trained specifically on known test questions. Researcher Nathan Lambert notes that gains in reinforcement learning environments often mimic benchmarks, requiring infrastructure not available to all labs.
The shift to private test sets introduces challenges regarding transparency, as external observers cannot verify errors in prompts. Artificial Analysis acknowledged this issue and revised several ambiguous or flawed questions within its AA-Omniscience knowledge test to ensure accuracy in the new index.
