Anthropic trial: developers learning a new library with AI scored 17 points lower

What the trial measured

A randomized trial associated with Anthropic found that developers who had an AI assistant available while learning an unfamiliar library did substantially worse on a follow-up test than those who worked without one. According to a dev.to write-up of the study, the AI-assisted group averaged 50% on the quiz while the hand-coding group averaged 67%, a gap of 17 percentage points. The speed difference was roughly two minutes and not statistically significant.

The underlying paper, "How AI Impacts Skill Formation" by Judy Hanwen Shen and Alex Tamkin, appeared on arXiv on January 28, 2026. Anthropic's accompanying post described the 52 participants as mostly junior software engineers. All of them had written Python weekly for more than a year, had some prior exposure to AI coding tools, and had never used Trio, a Python library for async concurrency.

Each participant got a short warm-up and then 35 minutes to build two Trio features: a timer that prints every second while other work runs, and a record fetcher that handles missing records. Half could consult an AI assistant during the task; the other half could not. Everyone then took a 14-question quiz worth 27 points, covering seven Trio concepts through debugging, code-reading and conceptual questions.

Where the gap came from

The dev.to piece reports the paper's headline statistics: the quiz difference was 4.15 of 27 points, with an effect size of Cohen's d = 0.738 and p = 0.010. The time difference came in at p = 0.391. The largest portion of the gap sat in debugging questions, and the smallest in code reading.

The error counts are the detail the dev.to author highlights. The median hand coder hit three errors during the task; the median AI user hit one. Of the twelve participants who finished both tasks without any errors, only two had coded by hand. In other words, the assisted group had an easier session and came away with less learning from it, and the paper's authors suggest the mistakes themselves were doing much of the teaching.

Interaction patterns mattered

The paper also grouped participants by how they used the assistant, with approximate figures given in Anthropic's post. The four people who repeatedly asked the AI to fix and verify their code were the slowest group, at about 31 minutes, and scored lowest of all, around 24%. The four who delegated everything from the start were fastest, at roughly 19.5 minutes, but scored only about 39%. Two other mid-reliance patterns landed in the mid-30s.

At the other end, the seven participants who asked only conceptual questions and fixed their own errors finished in about 22 minutes and scored around 65%. A small hybrid group scored about 68%, and the two people who generated code and then kept asking questions until they understood it averaged around 86%, though a subgroup that small should be read as a hint rather than evidence.

The caveats

The dev.to write-up is careful about the limits. The sample was 52 people, 26 per arm, and the interaction-pattern subgroups ranged from two to seven people; Anthropic states plainly that the pattern analysis does not establish a causal link between how people used the assistant and what they learned. The quiz came immediately after a single one-hour task, while the paper notes that real skill forms over months to years, so nobody yet knows whether the gap shrinks or widens over time.

The assistant itself was GPT-4o in a chat panel that could see the participant's code, not an agentic coding tool. Anthropic expects a larger effect from agents such as Claude Code, but that was not tested here. And despite the junior framing, the paper's balance table puts 29 of the 52 participants at seven or more years of coding experience; what they shared was being new to Trio. Anthropic also points to earlier observational work showing time savings of up to 80% on tasks people already knew how to do, which suggests the risk is specific to learning unfamiliar things rather than to AI assistance in general.

A staged approach for juniors

The dev.to author's practical proposal is to stage delegation per area of the codebase rather than by tenure. First, a junior writes the initial working version of anything new without code generation, with the agent restricted to conceptual questions about errors or design. Second, generated code becomes allowed, but only alongside follow-up questions until every line can be explained, echoing the top-scoring pattern in the study. Third, full delegation comes last, and debugging is the final thing to hand over, since it was both the weakest skill among heavy AI users and the widest part of the quiz gap.

Why it matters

Many teams now hand coding agents to new hires on day one. This trial suggests the cost of that convenience is not speed but retention, particularly the debugging skill that struggling through errors normally builds. With a small sample and an immediate quiz it is not the final word, but it points to a cheap policy: let juniors ask the agent questions while they keep finding the bugs themselves.