Anthropic's alignment research lead, Evan Habinger, believes there is a more than 10% chance that AI could destroy all of humanity within the next decade. He admitted that the company currently lacks a solution for the superintelligence alignment problem and has not yet found a clear path forward, despite their best efforts.
A former employee, Jacob Coxon, resigned from the company, stating that AI labs like Anthropic and OpenAI are gambling with human lives. Coxon warned that systems surpassing human capabilities could soon emerge, capable of hacking any system and seizing power and resources overnight.
The concerns are centered on recursive self-improvement, where AI systems optimize themselves without human intervention. Anthropic previously noted in June that fully recursive self-iteration could increase the risk of humans losing control over AI systems.
Coxon views recent incidents, such as an OpenAI model's abnormal behavior and intrusion into the Hugging Face platform in July, as warning signs. He argues that while such events might encourage collaboration between US labs, a global AI race remains inevitable unless costly measures, such as temporary bans on increasing model capabilities, are implemented.
