Anthropic circulates draft AI safety bill that could block models risking 50-plus deaths

Anthropic has been circulating a draft AI safety bill among US lawmakers that would make it possible to block the release of an AI model when it poses a serious and immediate danger to life or property. The proposal, first reported by Politico and covered by TechCrunch, is a rare example of an AI lab drafting the federal legislation it wants applied to its own industry.

Concrete numbers for catastrophic risk

The draft sets out specific conditions under which a model could be barred from distribution. According to Politico, the bill targets cases where releasing a model carries a "foreseeable, imminent risk" that more than 50 people could die, or that more than $1 billion in property damage could result. That risk must stem from one of three sources: the creation or release of a biological weapon, the model evading the controls of the company that built it, or a cyberattack.

As TechCrunch points out, those figures echo California's Senate Bill 53, recently passed into law, which uses the same numbers to define "catastrophic risk."

Evaluators funded by the labs they assess

Under the proposal, models would be examined by independent evaluators registered with the US Commerce Department. Those evaluators could be paid by the AI companies themselves, provided the payment is not contingent on the outcome of the assessment. TechCrunch notes that it is not clear how this arrangement would guarantee that funding cannot sway an evaluation.

An alternative model has been floated elsewhere in the industry: DeepMind chairman Demis Hassabis suggested some months ago that AI regulation could mimic the Financial Industry Regulatory Authority, the self-governing body that oversees US securities firms.

A 72-hour escalation, but no unilateral agency power

The bill lays out a timeline for acting on a finding of danger. Once evaluators conclude that a model poses an imminent catastrophic risk, they would have 72 hours to notify the Commerce Department, which would then ask a judge in Washington, D.C., to halt the model's release. Notably, per Politico, no federal agency would be permitted to block a model on its own authority — only a court could order a halt.

A lab writing its own rules

Anthropic confirmed the existence of the draft to Politico, with a spokesperson saying the bill reflects the proposals in the company's Advanced AI Framework, which it has been promoting to federal and state lawmakers since June. The company did not respond to TechCrunch's request for comment.

The proposal arrives as the industry faces growing scrutiny over security lapses, particularly a string of episodes in which AI programs apparently disobeyed their human operators and hacked websites on the open internet. Yet the major labs have largely resisted mandatory oversight, favoring voluntary schemes in which third-party observers are embedded inside AI companies to monitor models for signs of risk or abuse. Anthropic's draft goes somewhat further than most industry positions — Nvidia chief executive Jensen Huang recently argued that AI needs no new laws at all.

Why it matters

Whatever its fate in Congress, the draft shows how one of the leading AI labs wants to be regulated: hard numeric thresholds, lab-funded but formally independent evaluators, and a court order as the only route to blocking a model. Defining catastrophic risk as 50 deaths or $1 billion in damage gives legislators a concrete starting point that most policy debates have lacked.

At the same time, the details matter as much as the headline numbers. Letting companies pay their own evaluators and stripping agencies of independent enforcement power are industry-friendly features that critics are likely to challenge. If lawmakers adopt the framework, it could become the baseline for US AI regulation; if they amend it, the fight over who pays for, and who controls, safety evaluations will be the one to watch.