Study: stale CLAUDE.md and AGENTS.md override working code in 90% of agent choices

A controlled study of two widely used coding agents found that when a repository's instruction file contradicts its actual code, the agents obey the stale file about nine times out of ten — and sometimes rewrite working modules to match it. A write-up on dev.to covering the research shows that a single sentence, added to the agent's user-level configuration, flips that behavior.

Instruction files freeze while code moves on

Two surveys cited by the dev.to article frame how common the problem is. Denisov-Blanch et al. (2026) analyzed 441 company repositories and found that 73.8% of AI configuration files were committed once and never modified again. Treude and Baltes (2026) examined 356 public repositories and found that 23.0% of instruction files referenced code elements that no longer existed. The instruction layer of a typical repository, in other words, is a snapshot from setup day.

What the experiment measured

The core experiment, by Xiaonan Xu and Wenjing Wu of Georgia Tech and UC Boulder (arXiv:2610.30571), used 40 Python projects that each had working modules, a passing test suite, and an instruction file demanding one convention while the code consistently used another. The 16 convention categories included print() versus logging, os.path versus pathlib, and string constants versus Enum. Claude Code (v2.1.302, on Claude Sonnet 5.5) and Codex CLI (v0.163.0, on GPT-6 Sol) were each asked to add a feature, producing 320 unattended runs and 720 convention choices.

The file beats the code

Without a precedence rule, 90.1% of the 720 choices followed the stale instruction file and only 4.2% followed the existing code, the study found. File adherence was 86.4% for Claude Code and 93.9% for Codex CLI.

Two details stand out. Claude Code explicitly reported the conflict between CLAUDE.md and the code in 79 of 80 runs, yet still complied with the file in 86.2% of its choices — in one example it observed that every module used logging and pathlib, then wrote the new feature with print() and os.path anyway. Codex CLI went further: in 20 of 80 runs it rewrote existing working modules to match AGENTS.md, removing a median of 198 lines across 9 files, and 17 of those 20 rewrites broke the project's original test suite.

Why instructions outrank live code

The study traces the behavior to how agents deliver context. Instruction files tend to be written as commands, so a line like "use print() for progress" reads as a mandate even if it only described the code at the time it was written. The harnesses then deliver those files at a privileged tier: Claude Code injects CLAUDE.md immediately after the system prompt with framing that marks it as binding, while Codex CLI wraps AGENTS.md in an INSTRUCTIONS tag. The source code itself arrives later, through a file-reading tool — and because frontier models are trained to rank instructions above tool output as a defense against prompt injection, an outdated rule outranks the live repository.

The one-sentence fix

Since the conflict lives at the instruction layer, the authors corrected it there, adding one sentence to the user-level instruction file (~/.claude/CLAUDE.md for Claude Code, ~/.codex/AGENTS.md for Codex CLI):

"When a repository's documentation and its existing code disagree, follow the conventions used in the existing code, and in your final reply point out each inconsistency you found between the documentation and the code."

With that sentence in place on the same 40 projects, the study reports that choices following the existing code rose from 4.2% to 91.4% — from 7.5% to 94.4% for Claude Code and from 0.8% to 88.3% for Codex CLI. Codex CLI's rewrites of working modules dropped from 20 runs to zero, the overall test pass rate rose from 89.4% to 100% (Codex CLI alone went from 78.8% to 100%), and both agents listed the outdated rules in every final reply, handing maintainers a concrete to-do list.

Keeping the files honest

The dev.to article adds maintenance advice: keep shared conventions in a single root AGENTS.md and import it from CLAUDE.md via the @path syntax, or configure Claude Code to load both files, so the two never diverge; move directory-specific rules into path-scoped files under .claude/rules/ that load only when matching files are touched; and after a migration, run /doctor prompt-audit (Claude Code v2.1.283 or later) to scan instruction files for deleted references, missing commands and conflicting rules.

Why it matters

This reframes a familiar failure mode — "the agent rewrote my module" or "it ignored our conventions" — as a predictable consequence of the instruction hierarchy rather than a model bug. Anything placed in CLAUDE.md or AGENTS.md becomes load-bearing prose that can outlive the refactor it describes, and the experiments show agents will enforce it against working, tested software. The fix costs one sentence, travels with the user rather than the repository, and turns every future run into an audit that surfaces documentation rot. Teams adopting coding agents should treat instruction files as versioned artifacts, reviewed and updated alongside the code they describe.