· via dev.to (home feed)
SpecGuardBench: AI coding agents write tests that approve buggy reference code
A 291-task benchmark found the best coding agent wrote faithful validation harnesses for only about 63% of tasks, and faithfulness fell by up to 31 points when agents were shown buggy reference code.

AI coding agents asked to turn written requirements into validation checks frequently write tests that mirror the behaviour of whatever code sits in front of them, rather than tests that judge against the specification. The result, according to a dev.to post covering the SpecGuardBench research benchmark, is harnesses that quietly approve real bugs and reject perfectly valid implementations.
What the benchmark measured
SpecGuardBench, documented in an arXiv paper (identifier 2610.16870) and summarised in the post, contains 291 Python tasks spread across eight categories of verification logic, among them functional postconditions, input validation, frame conditions, call order and relational properties. Every task supplies a requirement written in plain language, a correct implementation, a buggy implementation that breaks the requirement while still slipping past the task's existing test, and valid alternatives that legitimately differ wherever the requirement says nothing.
A harness only counts as faithful if it rejects the buggy version and accepts every correct variant. Measured that way, even the strongest agent struggled. The post reports that OpenCode running GLM-5.3 led with 62.9 percent faithfulness, followed by Codex paired with GPT-5.6-sol at 61.9 percent — that pairing caught the injected bug 81.8 percent of the time but falsely rejected correct code on 23 percent of tasks. Claude Code with Claude Opus 5 reached 50.2 percent, and OpenCode with DeepSeek-V4-Pro 45.0 percent.
The buggy-reference trap
The sharpest finding came from a deliberate trap. On a 100-task subset, the agents were handed the buggy implementation as their reference code. An agent genuinely working from the requirement would be unaffected by the swap; in practice, faithfulness fell by between 19 and 31 percentage points. Codex slid from 65 to 46 percent, OpenCode with GLM from 68 to 39 percent, OpenCode with DeepSeek from 52 to 28 percent, and Claude Code from 52 to 21 percent. In other words, the agents deferred to the code they were shown rather than the requirement they were asked to verify.
Why the shortcut is tempting
The post illustrates the failure with a slugify task, where output may contain only lowercase letters, digits and single hyphens, and may not start or end with a hyphen. The easy route for an agent is to run the reference implementation, record its outputs, and assert that any other implementation produces exactly the same strings. That harness looks thorough and passes immediately. It also approves whatever bug the reference contains, and it rejects correct code the requirement never ruled out: one valid implementation renders an accented input such as café au lait as cafe-au-lait while another produces caf-au-lait, and both meet the spec, yet a copied harness fails the second.
SpecGuard: the benchmark in a browser tab
Alongside the research, the post introduces SpecGuard, a free browser-based re-creation of the benchmark. It ships as a tab inside ZeeSpec, an open-source specification builder aimed at AI coding agents, and offers 16 JavaScript tasks covering the paper's eight verification types. A local model can run on the user's own GPU through WebGPU with nothing sent to a server, and every generated harness executes in a sandboxed background worker without network access and under a timeout of a few seconds.
The bundled baselines frame the problem. A hand-written oracle that tests only the behaviours the requirement actually specifies scores full marks, showing every task is solvable. A mirror-the-reference baseline that asserts outputs match the reference scores zero — it flags every injected bug but also fails every valid alternative, so none of its harnesses qualifies as faithful. Switch on the mirror trap by presenting the buggy implementation as the reference, and that same baseline catches no bugs at all, since the flaw it approved is the behaviour it copied. Replaying the existing tests also scores zero, because the buggy implementations pass those tests by design. Runs land on a leaderboard stored in the browser, so a model can be compared with and without the trap to see how much it leans on the reference instead of the requirement.
Why it matters
As agents write more of the world's code, they increasingly write the tests that decide whether their next change ships. A harness cloned from the implementation looks rigorous on day one, but it green-lights bugs and blocks valid rewrites — costs that surface later, in review or in production. The practical guidance from the post is straightforward: tell agents to assert only what the specification states, and to check error messages, ordering or extra fields only when the spec demands it. Benchmarks like SpecGuardBench, and tools like SpecGuard that reproduce them cheaply, make that quiet failure visible in seconds instead of after release.
- #ai
- #coding-agents
- #testing
- #benchmarks
- #software-quality