METR trial: experienced developers were 19% slower with AI tools yet felt 20% faster

What the trial measured

According to a write-up on dev.to, a 2025 randomized controlled trial run by METR — an organization that studies how AI systems behave in real conditions rather than on benchmarks — asked sixteen experienced open-source developers to fix genuine issues in their own mature repositories. For each task, participants were randomly assigned to either work with AI tools or without them, and completion times were recorded by the clock rather than self-reported.

The population matters here. These were not beginners working through exercises; they were maintainers with years of accumulated context about the exact systems they were modifying.

A 39-point gap between belief and stopwatch

Before starting, participants forecast that AI assistance would trim roughly 24% from their completion times. After finishing, they still judged themselves about 20% faster on the assisted tasks. The measured times pointed the other way: those tasks took roughly 19% longer than the unassisted ones. The dev.to article sums this up as a gap of 39 percentage points separating what skilled professionals thought had happened from what the timer actually captured.

Familiarity made the slowdown worse

The slowdown was not spread evenly. It hit hardest on the codebases developers knew best — the opposite of the intuitive assumption that deep expertise and AI assistance should compound into bigger gains.

The explanation offered in the write-up: long experience builds a fast, private mental model of a system — which areas are risky to modify, which interfaces are unreliable, where problems tend to cluster. Consulting that model costs essentially nothing. An AI assistant does not share it, so every suggestion has to be checked against it, and that verification time never registers when a developer judges how the tool felt in the moment.

The article draws an analogy to a trap every developer meets early: in binary floating point, adding 0.1 and 0.2 yields 0.30000000000000004. Nothing is malfunctioning; human intuition about decimal arithmetic is simply miscalibrated against how machines store numbers. The author argues the METR result has the same shape. An internal sense of one's own speed is a model — usually good enough, occasionally and confidently wrong, and only correctable by checking it against ground truth rather than trusting the feeling.

What actually helped

The developers who avoided the slowdown were not the ones who rejected AI tools outright, the write-up notes. They were the ones who measured — timing themselves on their own tasks, repeatedly, and treating the question of whether a tool helps as something to establish empirically rather than by impression.

A second habit that held up: writing down why the code looks the way it does, before that reasoning lives only in one person's head. A decade of experience is largely an undocumented model of a system. AI tools can turn out convincing code quickly, but they are not a substitute for that model — and mistaking them for one is, in the article's framing, how a veteran loses time while being convinced they saved it.

Why it matters

Much of the current enthusiasm around AI coding assistants rests on how developers say the tools make them feel. This trial — small, at sixteen participants, and focused on one specific scenario, experts modifying codebases they know intimately — suggests those impressions can be wrong by nearly forty percentage points, and in the wrong direction.

The result does not automatically generalize. It covered seasoned maintainers working on familiar, mature code, and it may not transfer to greenfield projects, unfamiliar codebases, or less experienced developers. But it shifts the burden of proof: claims about productivity gains from AI tools deserve validation with measured task times on real work, not impressions gathered right after a session that felt productive.

The remedy the write-up proposes is unglamorous and cheap: run the comparison yourself, timing several of your actual tasks with and without the tool before drawing conclusions. If the sensation of speed and the clock disagree, trust the clock.