Researchers Owen Queen and James Zou from Stanford University analyzed 50 claims regarding large language models against 20 different systems. The study revealed that 44 percent of these findings have already expired, while 34 percent have significantly weakened.
The average lifespan for findings with a determined expiration date is 15.7 months. After one year, 16 percent of claims had expired, a figure that rose to 39 percent after two years. Only 16 percent of the original findings remain valid, and 6 percent have strengthened.
Findings concerning model limitations, such as reasoning errors, expire the fastest. Conversely, claims regarding model preferences, sycophancy, and behavioral tendencies tend to persist longer. The researchers noted that fabrication of scientific citations dropped from 55 percent in GPT-3.5 to 0 percent in GPT-6.1 Sol, though other models like Qwen3.7-Flash still showed high rates.
Performance varies significantly even among models released within the same timeframe. For models released between July and September 2026, the number of expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. The study concludes that empirical findings should be treated as observations of specific models rather than universal truths.
