Executives from Anthropic, OpenAI, Meta, and Microsoft have urged policymakers to monitor the development of AI systems capable of self-improvement. More than 20 industry leaders and researchers, including Anthropic co-founder Jack Clark, OpenAI chief scientist Jakub Pachocki, and AI pioneers Geoffrey Hinton and Yoshua Bengio, co-authored a paper on the topic.
The authors warn of an emerging trend where AI is used to automate the development of next-generation models. This process, termed an intelligence explosion, could lead to a sudden acceleration in technological progress. The researchers argue that such advancements might outpace the ability of society to guide and adapt to these new systems.
The paper calls for stricter oversight of data centers involved in automated AI research and higher safety standards. Experts suggest that policymakers should establish mechanisms to constrain potential intelligence explosions and develop emergency response plans for various rapid AI takeoff scenarios.
The researchers emphasize that current preparations are insufficient given the potential impact of software-driven intelligence explosions. They warn that once such a process begins, the window of opportunity for effective regulatory intervention may close rapidly.
