In July 2025, METR published a study involving 16 open-source developers. Participants were split into groups with and without AI access to complete 246 tasks. The results showed that using AI tools slowed developers down by 19%, while participants subjectively believed they were 20% faster.
A follow-up study in February 2026 involved 57 developers. The organization noted a decline in experimental validity due to developers refusing to work without AI tools, even for payment. Participants increasingly avoided tasks where AI offered significant advantages, leading to biased data collection.
In May 2026, METR surveyed 349 technical workers. Respondents estimated that AI increased their work value by 1.4 to 2 times. However, the researchers noted significant selection bias and discrepancies between subjective reports and actual performance metrics observed in previous experiments.
METR identified that the gap between perceived and actual productivity reached 40 percentage points. The organization concluded that traditional randomized controlled trials are becoming harder to conduct as developers grow more dependent on AI and increasingly reject manual workflows.
