Anthropic Urges AI Industry to Develop Safeguards as Self-Improving Systems Near Reality
Artificial intelligence company Anthropic has warned that AI technology is advancing at a pace that could soon allow systems to improve themselves without direct human involvement, raising concerns about safety, oversight, and long-term control.
In a recent blog post authored by Marina Favaro, head of The Anthropic Institute, and Anthropic co-founder Jack Clark, the company cautioned that the industry may be approaching a stage known as “full recursive self-improvement,” where AI systems can build and enhance future versions of themselves.
According to the authors, such capabilities could unlock major breakthroughs in areas such as healthcare and scientific research but could also introduce significant risks if adequate safeguards are not in place.
“Full recursive self-improvement also might increase the risks of humans losing control over AI systems,” they wrote. “If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important.”
The company argued that the AI sector is closer to achieving self-improving systems than many experts previously anticipated and called on developers to consider slowing or temporarily pausing frontier AI development to allow researchers time to better understand the technology’s potential societal consequences.
Anthropic also urged the industry to create mechanisms that would allow humans to intervene if advanced AI systems begin behaving in unintended ways.
Speaking during an interview with CNN’s Anderson Cooper, Clark compared the current state of AI development to driving a car equipped only with an accelerator.
“When I look down at the car we’re driving, all I have is a gas pedal. I don’t have a brake pedal, and surely at some point in the future we might want that option,” he said.
When asked whether his concerns mirrored science-fiction scenarios in which AI turns against humanity, Clark acknowledged that such possibilities are taken seriously within the industry.
“Yeah, we read the science fiction and watch science fiction here as well, so it’s not lost on us,” he said. “How do you maintain control over fleets of scientists that are much, much larger and much faster than ones you’ve had before?”
Clark added that one of the most significant challenges facing AI development is ensuring that systems remain understandable, verifiable, and trustworthy as they become more sophisticated.
The warning comes as Anthropic prepares for a potential stock market debut that could generate billions of dollars to fund additional AI infrastructure, including data centres and computing resources. The company joins a growing list of major AI firms seeking investment to accelerate development in an increasingly competitive industry.
Despite the intense rivalry among leading AI developers, Clark argued that cooperation on safety standards remains possible. Drawing parallels with international efforts to manage nuclear weapons during the Cold War, he said companies and governments should work together to establish safeguards before the technology becomes more powerful.
“We’ve done this before. In the height of the Cold War, under highly tense situations between rivalrous countries, they found ways to stabilize aspects of the nuclear arms race,” Clark said. “All of this has been done before in other domains, and it may need to be something we do in the domain of AI.”
The comments add to an ongoing global debate over how best to balance rapid AI innovation with the need for regulation, transparency, and long-term safety measures.
