Anthropic proposes court-backed power to halt AI models in catastrophic-risk cases

Anthropic has circulated draft legislation to US lawmakers that would let the federal government seek a court order to suspend the release or use of an advanced AI model in narrowly defined, catastrophic-risk scenarios. The proposal remains a draft, not a bill formally introduced in Congress.

Under the reported thresholds, the government would need to show an imminent risk of more than 50 people being killed or seriously injured, or more than $1 billion in damage. The risks covered would include an AI system helping create or release a biological weapon, enabling a major cyberattack, or escaping controls imposed by its developer. A judge would have to approve any suspension; officials would not have unilateral authority to halt a model under the proposal.

The draft would also establish a role for independent safety evaluators. They would register with the Commerce Department and assess frontier developers’ testing and safeguards. Evaluators could be paid by the companies they review, but their compensation could not depend on the outcome; if they found a qualifying imminent risk, they would have to notify the department within 72 hours of completing the assessment, according to a report on the draft.

Anthropic says the proposal would require frontier developers to test their most powerful models, disclose findings publicly, submit to independent evaluation and maintain robust security programs. The draft’s approach goes beyond transparency: it would create a legal route to block or deter deployment once specified risk thresholds were met.

The proposal comes as US AI policy has relied largely on voluntary industry pledges and executive orders, while Congress has yet to pass a federal AI law. It would override some state rules, including California’s, CGTN reported. The proposal also marks a shift from Anthropic’s June position that states should retain full authority in the absence of a strong federal law.

The debate over government oversight is unfolding alongside other questions about who gets to shape AI systems. Anthropic has also convened law professors and democracy experts to discuss who should set Claude’s values, though NBC News reporting summarized by AI Weekly says the company retains final say over how outside feedback is applied. Separately, an IPWatchdog opinion argues that Congress should establish a federal right protecting artists’ voices and likenesses from unauthorized digital replicas; that piece describes a different legislative question from Anthropic’s proposed model-safety regime.

The draft would apply to the largest developers, according to the account of its provisions: the framework described in the reporting targets models trained using more than 10²⁵ floating-point operations and companies above specified revenue or research-and-development spending thresholds. The draft has not been introduced, and there is no guarantee Congress will adopt it. Charlie Bullock of the Institute for Law & AI called the proposal reasonable but said it was weaker than the House’s FRONTIER Act, CGTN reported.