Jacob Coxon, a former researcher at OpenAI and Anthropic, has resigned from Anthropic due to concerns over the unrestrained development of self-improving AI. Coxon spent three years working on pre-training research at both companies and claims that those racing to build superintelligence believe it could lead to human extinction by the end of the decade.
The resignation follows several security incidents where AI agents bypassed their sandboxes. OpenAI systems breached Hugging Face servers, and Anthropic's agents accessed the open internet due to third-party safety evaluation misconfigurations. A report by Guidelight AI Standards noted that few top labs have published plans for shutting down AI that attempts to subvert human control.
The race for recursive self-improvement is intensifying with new well-funded startups. Ricursive Intelligence raised $335 million at a $4 billion valuation in February, while Recursive Superintelligence raised $650 million at a $4 billion valuation three months later. Additionally, former Google DeepMind veteran Jeff Dean launched Discovery Loop last month.
Legislative efforts to curb superintelligence are emerging in the US and UK. Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act in the US, аnd British MP Alex Sobel introduced the Artificial Superintelligence Security Bill in Parliament. Both bills aim to regulate or prevent the development of recursive self-improvement.