SWE-Gate benchmark finds 34% of test-passing agent patches fail code review rules

A second gate for agent code

SWE-Gate, a newly published benchmark, argues that test passage is the wrong finish line for evaluating coding agents. According to a write-up on dev.to, the benchmark measures two outcomes separately: whether a patch passes a repository's functional tests, and whether it satisfies the review rules that would otherwise block a merge. Across its experiments, 644 agent-generated patches cleared the functional gate, and 221 of those — roughly 34% — failed the review gate.

The point is not simply that agents are weak; it is that standard evaluation hides a failure mode. Benchmarks in the SWE-bench lineage score an agent on resolving a GitHub issue with a patch that passes the existing test suite. Production code, however, must also survive human review, and the constraints enforced there — style and formatting rules, architectural conventions, security boundaries, performance expectations, maintainability norms — are not encoded in test suites. They live in review guidelines, team habits and the judgement of senior engineers, so agents optimised for tests get no signal about them.

How the benchmark is built

SWE-Gate assembles 303 repository-level repair tasks drawn from 75 Python repositories. Each task ships with a functional test suite reflecting the original issue's acceptance criteria, a separate constraint suite derived from real pull request comments, a non-compliant patch that passes the tests while breaking the rules, and a gold patch that clears both gates.

The constraints were mined from review comments on merged pull requests. The authors kept only comments that requested changes rather than casual discussion, then turned the enforceable requests — adding type hints to function signatures, or replacing hand-rolled regex with an existing validation helper — into tests that run once the functional suite has passed.

That ordering is deliberate. An agent's score is a pair of values, functional correctness and constraint compliance, and the second is only measured after the first succeeds. This keeps "the agent could not solve the problem" distinct from "the agent solved it in a way no reviewer would accept" — different failure modes that call for different interventions.

What the experiments showed

The evaluation ran four LLM backends — GPT-4, Claude and two open models — inside a shared agent scaffold that supplied repository context, the issue description and test feedback in a retry loop.

Of the recorded attempts, 359 patches (35.8%) failed the functional tests, usually through logic errors, incomplete understanding of the issue or hallucinated APIs. Another 221 attempts (22.0%) passed the tests but broke review constraints. Only 423 (42.2%) cleared both gates. The 221 constraint failures are the ones a test-only pipeline would never catch: they would sail through automated CI and then come back from a reviewer.

Ordinary violations, not edge cases

The constraint failures tracked by SWE-Gate are unglamorous. Missing type annotations accounted for 18% of them, copy-pasted logic that should have been a shared helper 15%, business logic placed at the wrong abstraction layer 12%, hardcoded magic values 11% and missing docstrings 9%. These are routine points of friction in code review rather than rare edge cases — which is exactly why a benchmark that ignores them flatters agent performance.

Reworking the agent feedback loop

The findings argue for changing how agents are orchestrated. The familiar single loop — generate a patch, run tests, retry — becomes a nested one: retry with test output until the functional suite passes, then retry with constraint feedback until the review gate passes.

The second loop is harder to feed. Functional feedback is concrete, an assertion mismatch with expected and actual values; constraint feedback is interpretive, requiring the agent to understand why a repository bans, say, database calls from controller code. The write-up suggests agents need repository guidelines as context, explanations of why a violation matters rather than bare failure messages, and examples of the preferred pattern. That is a heavier context load than functional feedback, and it depends on retrieving the right guidelines at the right moment.

Multi-stage evaluation also means more state to track: which patches passed which gate, which constraints the agent has violated before, and which rules apply to the current patch. The authors model this as a state machine with distinct states for functional failure, constraint failure and completion.

Why it matters

SWE-Gate puts a number on something practitioners already suspected: passing tests is not the same as shipping code. A third of the test-passing patches in its experiments would not survive review, which means benchmark scores built on test passage alone overstate how much agent output is actually mergeable. For teams building or deploying coding agents, the practical moves are to evaluate both gates separately, to encode review standards as testable constraints, and to treat constraint feedback as a first-class signal in agent loops rather than an afterthought.