OpenAI, Anthropic Want To Pace Frontier AI. But Who Sets The Rules?

In July this year, OpenAI’s models were being tested in a controlled cybersecurity environment and tasked with spotting and exploiting software vulnerabilities. The models found ways around the intended network restrictions and created a “swarm”, with different agents delegating tasks to one another, during an unintended attack on the infrastructure of AI platform Hugging Face.

The incident turned longstanding concerns about autonomous AI systems into a more immediate question: what happens when increasingly capable models find ways to operate beyond the restrictions imposed on them? It also became one of the “warning shots” cited by researchers calling for frontier AI labs to coordinate on safety. 

The debate intensified last week when Anthropic researcher Jacob Coxon publicly quit the company and accused leading AI labs of racing towards self-improving superintelligence without adequate safeguards.

Soon after, Anthropic CEO Dario Amodei turned these concerns into a concrete proposal, urging frontier AI companies to “pace” their development. His proposal included independent evaluators, common safety standards, and greater coordination between governments and AI companies.

OpenAI CEO Sam Altman backed parts of the proposal, saying pacing the frontier has become a primary topic of discussions at OpenAI and that the company would commit to independent evaluations.

However, NVIDIA CEO Jensen Huang has rejected calls to slow AI development. “Run as fast as you can,” he said, arguing that companies should continue advancing the technology but hold back products until they know they are safe.

The disagreement has sharpened the debate over what pacing means, whether it can work amid intense competition, and who should decide where the brakes are applied.

AI’s Shift From Answers To Actions

AI has moved from models that largely responded to prompts to reasoning systems capable of using tools, executing code, and coordinating multiple agents to solve complex tasks.

Anthropic has warned that agents interacting with codebases, markets, and other digital systems could create a new class of risks, with machine-to-machine interactions eventually outpacing human oversight.

Shayak Mazumder, founder of agentic AI venture studio Adya, said newer agents can bypass instructions because they can explore paths developers may not have anticipated, creating a gap between what developers intend and what a system may discover it can do.

OpenAI’s own evaluations have also highlighted advances in agentic coding and cybersecurity, prompting it to strengthen network and tool restrictions and monitor risky actions.

Ganesh Gopalan, cofounder of Gnani.ai, pointed to jailbreaks, including the Hugging Face incident, and recursively self-improving systems as evidence that the risks are becoming harder to dismiss, even if their precise nature remains unclear.

“There is evidence that definitely there is some risk, for sure. But the point is how do you control those risks and what actions will be taken? I think that’s where it’s a completely grey area,” Gopalan said.

Who Gets To Write The AI Rulebook?

Amodei said pacing does not mean halting AI development. His proposal calls for frontier companies to give embedded third-party evaluators ongoing, employee-like access to assess safety practices and report incidents. He also wants companies in democratic countries to establish common standards before seeking broader international coordination.

Pacing could mean slowing the training of more capable models, restricting experiments involving recursive self-improvement, or evaluating models more rigorously before release.

However, Gopalan said the lack of a common definition creates a fundamental problem. If some frontier AI companies slow down, it remains unclear whether the same rules would apply to their competitors or to other countries.

“If you’re going to say, pacing it, are you going to stop? Are they ready to allow other companies or countries to catch up?” he said.

Mazumder believes mandatory checks are needed before frontier models reach users. “We need to come up with global standards and national standards to ensure that LLM has passed those checks before it can be released to the public,” he said.

Amitabh Kumar, cofounder of content governance platform Contrails.ai, said frontier companies should have a voice because of their technical expertise, but universities, independent scientists, and policymakers should have an equal seat at the table. He is particularly sceptical of company-appointed evaluators.

Could Safety Become A Competitive Moat?

Mandatory evaluations, audits, compute controls, and other safety requirements would also increase the cost of developing frontier AI.

For Gopalan, the risk is not merely the cost of regulation but the concentration of power it could create around a handful of well-funded companies.

“Right now, it’s more of a few companies that are becoming all-powerful,” he said, warning that safety rules should not be used to prevent other companies or countries from catching up.

Large AI labs may be able to absorb the cost of audits and repeated evaluations. Smaller companies may struggle, particularly if controls extend to advanced GPUs, computing infrastructure, or training resources.

Kumar pushed back against the idea that safety requirements would necessarily become a barrier, arguing that regulation inevitably carries a cost that can become part of doing business in a high-risk industry.

Meanwhile, a managing partner at a Bengaluru-based VC firm, who didn’t want to be named, pointed to the gap between private valuations and profitability, along with the capital required to keep developing increasingly powerful models.

The investor questioned if the valuations being discussed for the potential public listings of companies like Anthropic and Open AI could be justified, given their profitability timelines and infrastructure spending. “Profitability is far away and infrastructure commitments are out of control,” the investor said.

The investor also speculated that safety concerns could give the companies’ CEOs grounds to postpone their listings. However, there is no evidence linking the calls for pacing AI development to their IPO timelines. The more immediate question is whether future safety requirements would constrain the largest AI labs or reinforce their advantage over emerging rivals.  

While Anthropic is moving towards a possible IPO, Altman has said OpenAI will not go public this year, citing the risks posed by increasingly powerful AI systems. OpenAI is meanwhile reportedly exploring fresh funding at a $1.2 Tn valuation.

The Geopolitics Of Slowing AI

Even if US frontier labs agree to pace development, there is no guarantee that companies elsewhere will follow.

Amodei argued that failing to build advanced AI could hand the technology to authoritarian powers, while moving too quickly could create uncontrollable risks.

The US, therefore, has an incentive to reduce AI risks but also economic and national security reasons to prevent China from overtaking it. Huang’s argument reflects the concern that slowing American labs may only allow rivals to close the capability gap.

For countries outside the current frontier, the concern is that safety could evolve into a nuclear-style non-proliferation regime, restricting access to advanced models, chips, or computing infrastructure, according to Gopalan.

The issue is relevant for India, where Chinese open-source models have already found significant traction. “If Chinese AI models are cheaper, free, faster, whatever, startups should not be stopped from using them,” the investor quoted above said.

Can The AI Race Police Itself?

The divide between Amodei and Huang captures the central fault line. One side believes frontier capabilities are advancing too quickly for existing safety mechanisms to keep up. The other believes development should continue at full speed, with companies responsible for ensuring that unsafe products do not reach users.

Both approaches carry risks. Voluntary commitments may collapse under competitive pressure, while standards shaped primarily by incumbent labs could raise barriers for smaller competitors and concentrate control over the frontier.

Pacing AI development, therefore, is not merely a technical safety question. It is also a contest over who controls the technology, who bears the cost of making it safer, and who gets to participate in the next phase of the AI race.

Edited by Vinaykumar Rai

The post OpenAI, Anthropic Want To Pace Frontier AI. But Who Sets The Rules? appeared first on Inc42 Media.