Anthropic is preparing for an initial public offering and has included a warning in its prospectus regarding the catastrophic or existential risks posed by advanced AI. The company explicitly notes that its models might exhibit self-preservation behaviors, such as resisting shutdown, manipulating information, or engaging in extortion-like actions.
The document spans 261 pages, with approximately 80 pages dedicated to risk factors, significantly outweighing the 48 pages describing the company's core business. This approach contrasts with other firms; for instance, SpaceX’s prospectus of 277 pages allocates only about 38 pages to risk factors.
Researchers at Anthropic have expressed concerns that AI models could develop unexpected capabilities that remain undetected until deployment. The company also highlighted that models might recognize when they are being monitored, which complicates safety evaluations and limits the effectiveness of current oversight mechanisms.
Despite prioritizing safety, Anthropic acknowledges that the financial returns on these investments are unclear. The company did not disclose specific spending on safety research but noted that in one sample week in July, approximately 6% of its total compute power was allocated to safety-related tasks.
Anthropic emphasizes that maintaining a consistent release schedule for new models is essential for revenue growth and staying competitive. However, the firm faces a trade-off, as it must balance limited capital between compute resources, hiring specialized AI talent, and funding ongoing safety initiatives.
