Anthropic pitches court-backed power to halt AI models in catastrophic-risk cases

Anthropic has confirmed it is circulating draft legislation among U.S. lawmakers that would let the federal government seek to block the release or use of powerful AI models in tightly defined catastrophic-risk scenarios. The proposal remains a draft; it has not been formally introduced in Congress.

Under the reported framework, a court could be asked to intervene if there were an imminent risk of more than 50 people being killed or seriously injured, or of more than $1 billion in damage. The covered threats would include an AI system assisting with the creation or release of a biological weapon, enabling a serious cyberattack, or escaping controls set by its developer. The Commerce Department would have to seek a federal judge’s approval; the proposal would not give agencies unilateral authority to shut down a model.

The draft would also establish a formal role for independent safety evaluators, who would register with the Commerce Department and periodically assess frontier developers’ testing and safeguards. Companies could pay for the evaluations, but not on terms tied to whether the findings were favorable. An evaluator that found a qualifying imminent risk would have 72 hours after completing its assessment to notify the department, according to the reported draft.

Anthropic says the measure would require frontier developers to test their most powerful models, publicly disclose findings, submit to independent evaluation and maintain robust security programs. The proposed rules would apply to the largest developers, with the company’s framework setting thresholds based on training compute and company revenue or research spending. The draft would also override some state laws, including rules in California, according to CGTN’s account.

The proposal would move U.S. policy beyond the voluntary industry pledges and executive orders that have largely shaped the federal approach, as Congress has yet to pass a national AI law. It also marks a change in emphasis from Anthropic’s June position that states should retain authority in the absence of strong federal legislation. The company argues that transparency alone may not be enough once a model crosses a defined threshold for severe harm.

The legislative pitch lands amid broader disputes over how responsibility for AI-related harm should be assigned. A Reuters report on the legal debate described negligence as one possible basis for holding developers accountable, while legal experts interviewed by Reuters said strict liability or changes to the burden of proof could also be considered. Those questions are distinct from Anthropic’s draft, which focuses on preventive government intervention rather than liability after harm occurs.

Early assessments have been mixed. Charlie Bullock of the Institute for Law & AI called the proposal reasonable and said it went further than other frontier labs, but described it as weaker than the House’s competing FRONTIER Act, according to CGTN and an AFP report republished by New Vision. There is no guarantee Congress will take up Anthropic’s language.