Paper frames two gaps that make AI self-review of generated code structurally weak

The two-gap framework

A dev.to post published on September 15, 2026 walks through a new arXiv paper, "Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering" (arXiv 2609.12039), by Krentsel, Agarwal, Cemri, Zaharia and Stoica. As relayed by dev.to, the paper's core idea is that any evaluator of AI-written code — a test suite, a reviewing model, a human — works from two approximations, and the distance each one travels from the truth sets the ceiling on the entire evaluation.

The first is the requirement gap: the distance between what stakeholders genuinely want and what the written requirements say. The second is the model gap: the distance between the assumed deployment environment and the environment the code will actually run in. According to the dev.to summary, hallucination widens both gaps by inventing requirements or environment assumptions, while reward hacking exploits whatever either gap leaves unspecified.

Why self-review is structurally weak

The problem for self-review follows directly from this framing. Both gaps are measured using the same requirements and the same environment model that the implementation was built against. When the model that wrote the code turns around and evaluates it, it checks the output against the very approximations it just used, so its blind spots about stakeholder intent and production conditions pass straight into the review.

In practice, per dev.to, that means a same-model reviewer flags problems the writer already half-suspected were shaky while approving anything that shares its incorrect assumptions. The result reads like verification but behaves more like a single model's consistency check: coverage that looks reassuring while missing precisely the errors the model is predisposed to miss.

What actually narrows the gaps

The dev.to write-up highlights two moves from the paper that do reduce risk.

Cross-model review: a second, independent model re-derives the requirements and environment assumptions on its own, bringing different training data, different priors and different blind spots. This does not close the requirement gap — the requirements remain approximations — but it breaks the specific failure mode in which writer and reviewer converge on the same wrong picture of the world.

Execution: the paper's title carries the argument — reality is the only final verifier, and every pre-deployment evaluation is a proxy. A test that actually runs in an environment close to production forces the evaluator to commit to observable behavior rather than a plausible-sounding story, which is what surfaces behavioral errors that static passes miss.

Where human attention fits

The paper also offers a resource-allocation view, according to dev.to: human judgment is the scarce input for the requirement gap, and faithful but expensive evaluation is the bottleneck for the model gap. Because no team can apply dense human review to the volume of code agents now generate, the suggested pipeline routes cheap AI-generated diffs to a different model for a first independent pass, uses execution-based checks to catch behavioral failures, and reserves human review for the diffs that survive both stages — the point where an hour of attention buys the most.

The dev.to author adds a caveat: same-model review is not useless. It works fine as a linter that catches a model's obvious mistakes. It simply should not be mistaken for verification.

Why it matters

Agentic coding workflows are producing code faster than teams can manually inspect it, and letting a model check its own work is a tempting default. This paper names the exact failure mode behind that instinct: shared approximations between writer and reviewer produce agreement without independent evidence. Teams designing review pipelines now have a vocabulary — the requirement gap and the model gap — for deciding where independent models, real execution and scarce human judgment each belong, and why stacking more copies of the same model into the loop adds review steps without adding new information.