Jacob Coxon, a researcher from Anthropic's AI pre-training team, resigned due to concerns regarding recursive self-improvement (RSI). He warned that superhuman systems could soon acquire real power and resources, potentially endangering humanity.
Evan Hubinger, Anthropic's alignment science lead, confirmed these fears, stating he believes there is a greater than 10% chance that AI could kill all humans within the next decade.
While no lab has implemented RSI, Anthropic and OpenAI are automating model development. In April, Anthropic revealed that Claude completed an AI research experiment on mitigating LLM risks with minimal human input.
The resignation has sparked political reactions. Senator Bernie Sanders announced plans to introduce legislation to ban superintelligence and pause AI development, while Representative Lori Trahan called for urgent Congressional action.
Recent breakthroughs highlight the speed of progress. OpenAI's unreleased model solved a complex problem regarding Navier-Stokes equations, and Anthropic's Claude developed a computer-verifiable mathematical proof in 11 days.
The industry is seeing a trend of caution. OpenAI Chief Scientist Jakub Pachocki recently urged a slowdown in development, and a group of researchers signed an open letter in July warning about self-improving frontier models.