METR's follow-up to its AI coding slowdown study faltered when developers refused to work without AI

A research group that measured whether AI makes developers faster has found that measuring it again is getting hard. According to a post on dev.to, METR's attempt to repeat its well-known 2025 coding experiment produced results the lab itself considers unusable, largely because developers no longer want to work without AI tools.

The original finding

In early 2025, METR recruited 16 experienced open-source maintainers, paid them $150 an hour, and had them resolve 246 real issues in large repositories they had contributed to for years. Whether each task could use AI was decided at random. When AI was permitted, tasks took 19% longer, with a confidence interval between +2% and +39%. Participants had predicted a 24% speedup before the study began and, after it ended, still believed AI had made them roughly 20% faster.

The follow-up that lost its control group

The second experiment began in August 2025 at a larger scale: 57 developers, 143 repositories and more than 800 tasks. METR published its findings in February 2026 and concluded that its own dataset could not reliably estimate the current productivity effect of AI coding tools.

Recruitment was the central problem. METR reported a sharp rise in developers declining to take part specifically because they did not want to work without AI. Among those who did join, between 30% and 50% said they were withholding tasks they preferred not to do unassisted. METR believes both effects push its estimates toward understating AI's benefit.

Two further issues added noise. Pay dropped from $150 to $50 an hour, which METR suspects introduced an additional selection effect, and developers who run several AI agents at once made per-task timing difficult to interpret, since it is unclear how to clock a task delegated to a background agent.

Numbers that point the other way, weakly

The follow-up did produce estimates. Returning developers showed a speedup of -18% with a confidence interval from -38% to +9%; newly recruited developers showed -4% with an interval from -15% to +9%. On METR's scale, a negative figure means tasks took less time with AI, so both results lean in the opposite direction from the 2025 finding. Both intervals, however, cross zero. METR's own reading is that developers are probably faster with current tools than they were in early 2025, but that this round provides only very weak evidence of how much.

Self-reported speed carries a known error

METR also ran a survey from February to April 2026 covering 349 technical workers, 87 of them software engineers. Respondents reported a median 3x speed gain and a 1.4x to 2x median gain in the value of their work, and the median respondent said they would give up 29% of a month's salary to keep AI tools, a figure METR itself flags as weak evidence. The lab cautions that survey answers of this kind are not necessarily grounded in reality, noting that participants in the 2025 trial overestimated AI's effect on their task time by about 40 percentage points on average.

Why it matters

The dev.to post argues that the failed replication matters more than any single estimate. A randomized trial needs participants willing to work without the treatment being tested, and that willingness is drying up as AI assistants become embedded in everyday development workflows. The parallel-agent workflows developers now cite as proof of speed actively undermine stopwatch-style measurement, since task boundaries blur when several agents run in the background. What remains as evidence is largely self-reported speed, the one measurement that has been checked against objective timing and found badly off. The post's author suggests teams instead track outcomes that survive review, such as merged, dropped and reverted pull requests, and notes that as of mid-October 2026 no newer METR trial supersedes either estimate, leaving the industry's central productivity question harder to answer with each tooling cycle.