Jacob Coxon, a safety researcher at Anthropic, resigned on September 9, citing a lack of adequate safeguards against self-improving superintelligence. His departure follows a 3,800-word essay by Anthropic CEO Dario Amodei, who advocated for a global slowdown in AI development due to risks like bioterrorism and economic disruption.
Researcher Paul Christiano noted that the industry faces a risk of catastrophic loss of control, with some estimates placing the probability of human extinction within a decade at over 10%. Surveys of AI researchers consistently estimate the median probability of extremely negative outcomes at around 5%.
Critics, including Timnit Gebru, Milton Mueller, and Tristan Harris, argue that focusing on hypothetical extinction scenarios distracts from current, measurable governance failures and real-world harms caused by AI systems. They suggest these narratives may benefit the companies promoting them.
The industry faces intense competitive pressure, exemplified by OpenAI dissolving its superalignment team in 2024. Despite these concerns, no binding federal legislation currently governs frontier AI development in the United States, and no agency has clear jurisdiction over self-improving architectures.
While the EU’s AI Act serves as a comprehensive regulatory framework, its enforcement mechanisms are still adapting to the rapid pace of technological development.
