Anthropic's Claude Code survey shows usage and PRs up while delivery metrics stay flat

What the survey showed

Anthropic asked 132 of its own engineers about their use of Claude Code, and the responses look like a success story. According to figures cited in a dev.to analysis, merged pull requests per day rose 67 percent, and the share of engineers using the tool every day climbed from 28 to 59 percent. The engineers themselves put their productivity gains somewhere between 20 and 50 percent.

Then someone checked the delivery dashboard. At the organization level, the delivery metrics showed no corresponding movement.

That gap is the heart of the story: a tool can be in constant use, well liked by the people using it, and still leave no trace on the numbers a business actually runs on.

A gap that extends well beyond one company

The dev.to piece treats the Anthropic data point as an entry into a wider measurement problem. McKinsey research cited there found that only 30 percent of leaders could account for where the time AI freed up actually went; the remaining 70 percent could not. Asked from the other direction, Gartner's 2025 survey put the share of leaders reporting significant value from AI tools at 22 percent — a figure dev.to notes sits where McKinsey, Deloitte and ServiceNow each landed when measuring the question their own way.

Usage and token spend are inputs, not returns

Two numbers routinely stand in for an answer to the ROI question, and neither qualifies. Adoption shows whether anyone is using the tool — a useful leading indicator, but high adoption with no measured outcome is activity, not value. Token spend is a cost figure: it belongs on the cost side of the calculation, and tracking it closely says nothing about whether the resulting work was worth having. Both are easy to pull from a dashboard, which is exactly why they get reported.

Two ways ROI numbers get inflated

The first inflation is soft hours presented as hard dollars. The standard business case multiplies hours saved per person by that person's hourly cost and reports the total as savings. The hours are often real; the dollars usually are not, because the budget did not change. Nobody was let go, no contractor was dropped, no line item fell. Per the dev.to argument, freed hours only become real money when the time is redeployed to value-generating work, a planned hire is avoided, or the same headcount produces more of something the company sells. The suggested fix is simple: label every claimed dollar as hard or soft, and report the two separately. The total gets smaller and much harder to dispute.

The second inflation converts calendar time into labor time. A feature that once took three weeks and now takes one gets dollarized at an engineer's rate, but the original three weeks likely included queue time, review waits and unfinished dependencies. Cycle time and labor time are different things, and pricing the calendar span at an hourly rate invents work nobody performed.

The piece also warns against trusting self-reported speed at all. METR, in a controlled study that timed the same developers with and without AI assistance, measured a 19 percent slowdown against a self-reported 20 percent gain. The developers were confident they were faster; the clock disagreed.

Building a number that survives scrutiny

With the inflations stripped out, two mechanisms remain, each with its own conversion path. Acceleration — doing existing work in less time — converts through hours saved multiplied by hourly cost, then subtracts rework, since output a person has to redo never saved the time it appeared to. Avoided work — a task that stops happening entirely, like a support ticket resolved by a knowledge base — converts through the full cost of one interaction rather than a wage rate. Dev.to offers illustrative arithmetic, explicitly not client data: support running 20,000 a month over 2,500 tickets makes each avoided ticket worth 8 dollars, with overhead and tooling already baked in.

The denominator needs the complete cost base: build cost amortized over the tool's life, token spend, infrastructure, maintenance and human review of output. The one discipline required is avoiding double-counting, so the same saved dollar or the same token cost is not counted twice.

Why it matters

The Anthropic data point carries extra weight because it is the vendor's own engineering organization reporting it. Heavy usage, high satisfaction and self-reported gains of up to 50 percent coexisting with flat delivery metrics is the clearest recent demonstration that sentiment and adoption are not evidence of return. Combined with METR's finding that self-assessed speed can run opposite to measured speed, the lesson for anyone building an AI business case is to instrument outcomes rather than ask opinions: decide up front whether value arrives as acceleration or avoided work, apply the matching conversion, subtract rework and full costs, and treat any hours-saved-times-rate figure that never touched a budget line as soft.