Cheap LLM code reviewers hold up on routine bugs but collapse on authorization flaws

A vendor-run comparison of AI code review models has produced a sharply practical result: a budget model nearly matched a frontier model on everyday correctness bugs but missed the majority of security flaws, with the worst performance on code that handles authentication and permissions.

According to a dev.to write-up by Cole Halton, the evaluation covered 50 public benchmark pull requests from five open-source codebases: Cal.com, Sentry, Discourse, Keycloak and Grafana. A second dev.to analysis by Tessa Insley attributes the comparison to Entelligence, dated September 14, 2026, and names the two models as GPT-5.6 Luna and GPT-6 Astra.

What the numbers show

On cost, the cheap model looks unbeatable. Luna verified 69 bugs for $0.20 in total spend, while Astra verified 92 for $5.66, which works out to roughly $0.003 per verified bug versus $0.061. Insley quotes per-review costs of $0.0041 for Luna against $0.113 for Astra.

On routine defects the gap almost disappears. Halton reports Luna found 39 verified data and logic bugs to Astra's 47, and 10 concurrency bugs to 13. Insley adds that in Sentry, Discourse and Grafana, Luna finished within two verified bugs of Astra, with the gap widening at Cal.com, 21 versus 30.

Security is where the pattern breaks. Luna caught 9 of the 24 security bugs in the benchmark; Astra caught 19. On Keycloak, an identity and access management server, Luna produced 6 verified findings against Astra's 14, and only about half of its findings survived verification, compared with 93 percent for Astra.

The bugs that got away

Two Keycloak bugs Astra caught and Luna missed illustrate the failure mode. In one, federated recovery codes were never marked as used, so a code could be reused indefinitely. In the other, a newly added global view permission silently overrode denials configured on individual clients.

Neither bug is visible on any single line. Spotting them requires reasoning about the permission model as a whole after the change. Both Halton and Insley argue that a small model scanning for local defects is structurally blind to this class of bug.

Noise and instability

Precision adds a second cost. About one in four of Luna's findings was wrong, 24 of 93, versus 4 of 96 for Astra, or 74 percent precision against 96 percent. Halton's warning is that false positives erode trust: reviewers who watch a quarter of AI comments turn out to be noise begin skipping them, at which point the genuine catches stop landing too.

Insley highlights a repeatability check the vendor ran, rerunning two PRs per codebase a second and third time. On those ten PRs, Luna logged 15 verified bugs on the first pass, but only 7 recurred in both repeats and 12 in at least one; Astra's figures were 15, 10 and 14. The sample is small, but it shows a model can find a bug on one run and miss it on the next, variance that single-run benchmark scores conceal.

Caveats on the benchmark

Both writers flag the same weaknesses. Astra served as one of the two judges verifying findings, so it may have graded its own output favorably. And every benchmark PR predates both models' training cutoffs: Insley notes that the injected defects are new, but the surrounding code is public and old, with commit dates back to 2013, meaning the models may have effectively seen correct versions of these files. The absolute magnitudes deserve skepticism; the shape of the result is what holds up.

Why it matters

For teams adopting AI reviewers, the takeaway is routing, not replacement. A cheap model is a reasonable triage layer for everyday changes, and it caps spend dramatically. But code touching authorization, authentication or permission logic needs a stronger reviewer, and its output needs close human attention regardless of which model produced it. The benchmark also makes the case for demanding more from vendors: splits by bug class and codebase, repeatability data, and disclosure of whether benchmark code predates training data. An aggregate score hides exactly the failure mode, a missed permission bypass, that costs the most downstream.