Martian's Code Review Bench scores 14 AI code review tools across 16,017 pull requests

A benchmark with no seats to sell

Rankings of AI code review tools usually come with a built-in conflict of interest. As a dev.to write-up points out, DeepSource's comparison puts CodeRabbit first on a site that sells code-quality tooling, CodeAnt's own roundup declares CodeAnt the best overall pick, and other results on the page trace back to an agency with services to pitch. The write-up's proposed alternative now exists: Code Review Bench, run by Martian, which the post describes as an AI research lab rather than a review-tool vendor.

How the scoring works

According to the post, the benchmark gathered real open source pull requests on GitHub where AI review bots had taken part, then reconstructed each review timeline from the bot's suggestion through the developer's response to the code change that actually landed. A suggestion only counts as a hit when a developer acted on it. Every tool receives a precision, a recall and an F1 score, and the whole setup is reproducible: the code-review-benchmark repository is MIT-licensed on GitHub, and the input set is described rather than hidden behind a login. When a number ships without a method, the write-up argues, it is a claim rather than a result.

What the leaderboard shows

The live leaderboard covers 16,017 pull requests and 14 tools. On overall F1, the top of the table is tightly packed:

• Cubic Dev AI: 65.7 (precision 72.9, recall 59.8)
• GitHub Copilot: 63.9 (precision 68.8, recall 59.6)
• Claude: 62.5 (precision 70.4, recall 56.2)
• Greptile: 61.9, with the highest precision of the group at 80.3 but recall of 50.4
• CodeRabbit: 60.8 (precision 66.8, recall 55.8)
• Qodo Code Review: 58.9
• CodeAnt AI: 50.0 (precision 72.3, recall 38.2)

The dev.to author's advice is to read the precision and recall columns before the F1, because they reveal how each tool trades thoroughness against noise. Greptile's comments get acted on most often, but it misses the most issues among the leaders. CodeAnt sits at the opposite extreme in this sample: its comments are precise, yet a recall of 38.2 means most of what a reviewer would flag goes unreported. Copilot and CodeRabbit land near the middle on both axes.

Sample sizes are part of the method and part of the comparison. The post reports Copilot scored across 767 pull requests, CodeRabbit across 2,109 and Qodo across 2,208. A tool sitting high on a thin sample reads as a warning sign, not a headline; the author suggests preferring a position that survives its sample size.

A wider field than any vendor will show you

The ranked field spans platform-native tools like Copilot and Cursor, standalone reviewers like CodeRabbit, Greptile and Qodo, and coding agents such as Claude and Devin, whose review output was scored too. Precisions run from 80.3 down to 63 across the table, and the gap between the top and bottom F1 scores is roughly 15 points. No single tool dominates, which the write-up calls the honest shape of this market — and one that no vendor's marketing page, which shows only its own row, will ever display.

Why it matters

The benchmark's value lies less in crowning a winner than in shifting who controls the evidence. When a vendor ranks itself first in its own list, that is not evidence; when an unrelated party publishes an open, reproducible scoreboard covering 14 tools, it at least gives buyers a checkable starting point. The write-up is equally clear about the limits. The sample consists of open source pull requests weighted by whether developers adopted suggestions, which is not your closed codebase, your compliance rules or your review culture — and dimensions like platform fit, per-seat cost versus volume and self-hosting never appear in an F1 score. The practical move it recommends: use the leaderboard to shortlist two or three tools whose precision and recall suit your noise tolerance, run them for two weeks on a defined slice of your own pull requests, and count the suggestions your reviewers would have made. That count, not the public table, is the benchmark that actually describes your code.