METR report: AI itself is driving Anthropic's doubled pace of model improvement

Doubling pace driven by AI itself

Safety research organization METR spent 11 weeks inside Anthropic and concluded that the company's models are improving roughly twice as quickly as they were a year ago — with AI itself, rather than added computing power, responsible for most of the speedup. That is the central finding of a report published Thursday, covered in detail by The Verge. The Verge notes the probe was separate from a program announced last month that grants safety researchers employee-like access to Anthropic systems.

METR's benchmark analysis suggests Anthropic compressed roughly two years of historical progress into the last twelve months. The researchers said they are confident the acceleration did not come from scaling up training compute, the traditional engine of AI progress. Instead, as of October, most of the gain is attributable to AI, and that share keeps growing.

Twelvefold productivity claims

According to The Verge, informal Slack polls and staff interviews supported the benchmark picture. Employees using Anthropic's most capable internal model reported being roughly twelve times as productive as they would have been without AI assistance — about double what the same staff reported a few months earlier while using a model called Mythos 5.

Most of the acceleration currently comes from AI assisting human researchers rather than working on its own. But METR warned that fully autonomous contributions appear set to grow soon, and it found no clear signs of diminishing returns or bottlenecks that would prevent even faster progress.

Part of the story involves models nobody outside Anthropic has used. METR says the lab finished training three unreleased models in the past six months — labeled Model 2, Model 3 and Model 4, completed between July and October — along with an unreleased model dating back to 2022.

Redactions cut the most useful evidence

METR submitted a roughly 100-page assessment to Anthropic, but under the terms of the engagement Anthropic could redact the public version, and it did so extensively. METR said a majority of the evidence it found most informative was removed, that its information pipeline was narrow, and that a small number of its data requests were denied.

That matters because the withheld material could support multiple interpretations, meaning the public has to trust METR's judgment far more than the organization considers healthy for questions of this weight. Anthropic defended the cuts on competitive grounds, arguing the details could help rivals. METR noted that most of the redacted material was high-level and said it could not judge whether the commercial rationale was sound.

Anthropic cofounder Sam McCandlish publicly acknowledged the trade-off, admitting the redactions make METR's work harder to verify but arguing that disclosing exactly what is driving the speedup would hand competitors a roadmap to move faster. He said Anthropic will publish its own report on the pace of AI progress, along with new safety measures, within weeks.

Tension with Anthropic's own warnings

The findings have drawn criticism from safety researchers who see a contradiction at the heart of Anthropic's position: the company has been among the loudest voices for caution — CEO Dario Amodei has publicly urged labs to slow down — even as it accelerates development faster than many anticipated, while preparing for an initial public offering.

For some observers, the data suggests that even aggressive timelines for AI progress may be too conservative, and that recursive self-improvement, in which AI systems build increasingly capable successors with limited human involvement, may be closer than assumed. METR's report covers only Anthropic; The Verge notes that OpenAI said in September it is making strong progress toward an AI researcher that could advance deep learning and alignment work by March 2028.

Why it matters

This is some of the clearest public evidence that AI has become a primary input into its own development. If the pace of frontier improvement now depends on AI assistance rather than compute or headcount, the timelines informing regulation, safety research and investment may be calibrated to a slower world than the one arriving. METR sees no bottleneck that would prevent further acceleration, and it expects autonomous contributions to grow.

Equally significant is what the episode reveals about oversight. The most informative evidence was redacted, so the public must accept METR's conclusions largely on trust. Whether safety practices and independent scrutiny can keep up with AI-driven development is now the open question — and this report shows how hard it is to even check.