Parsimony ranks AI coding models by how much extra code they add

A benchmark for code volume, not just correctness

Parsimony, a new open-source benchmark introduced through a Show HN post on Hacker News, sets out to answer a question most evaluations ignore: which AI coding agent changes the least code? According to the project's site, established benchmarks focus on solve rate and cost, and nothing in those scores penalizes a model for writing far more code than a task requires — even though excess code makes a codebase harder to work with over the long term.

Parsimony's answer is to measure the size of the change itself.

Counting syntax units instead of lines

The benchmark does not run anything. Instead, it takes the patches that existing benchmarks already publish, applies each one to the task's code, and parses both versions of every changed file. It then counts "coding units": syntax elements such as statements, calls, comparisons, names and literals.

Counting units rather than lines is deliberate, because a model cannot improve its score by cramming more code onto fewer lines. Deleted code is subtracted from the total, so the headline figure is net units — added minus removed.

The leaderboard ranks models by mean net units added on the tasks they solved, lowest first, and that average includes failed attempts. Solve status comes from SWE-bench Verified's published results; Parsimony does not rerun the tests. Tests, documentation and generated files are left out of the measurement, and only Python is currently covered.

What the numbers show, and what they don't

The project is upfront about its limits. Unit counts measure size, not readability or design, so a smaller diff is not automatically a better one. The site's own example makes the point: a function that scans a list for a negative value using a flag-setting loop, and one that simply returns any(value < 0 for value in values), produce the same result with very different unit counts.

More consequentially, the data reveals a tension between brevity and capability. According to the project, models that write more code also tend to solve more tasks, with Spearman correlations of 0.33 to 0.60 on leaderboards containing 26 or more models. Every model on the board runs through the same agent, mini-SWE-agent, but the project notes that the agent scaffolding still influences how much code gets written — so results reflect the model-plus-agent pairing as much as the model alone.

Bring your own data

Parsimony also invites others to extend the approach. Anyone with code-size numbers from another benchmark can download an example JSON file, fill in their results and open it in a browser to generate the same chart and table. The file, the site notes, never leaves the user's browser.

Why it matters

As AI agents take on a growing share of routine programming, the volume of machine-written code becomes a maintainability problem, not just a correctness one. Large diffs cost review time, add surface area for bugs, and reshape codebases in ways teams have to live with long after a ticket closes. By making change size a first-class, comparable metric alongside solve rate and cost, Parsimony offers a lens that mainstream benchmarks lack.

Its honest caveats are part of the value: if brevity trades against solve rate, and unit counts say nothing about design quality, then "which model writes the least code" becomes a genuinely useful argument to have — with numbers attached.