AI agents pass tests on 74% of CVE tasks but produce secure code in only 18%

"The tests pass" is how most teams accept an AI coding agent's work. A newly published benchmark suggests that is a weak signal for security-sensitive code. On November 4, 2026, eight researchers (Yu et al.) posted a preprint called A.S.E 2.0 to arXiv, and according to a dev.to write-up of the paper, it measures how the repository context an LLM sees changes the security of the code it writes. The results are unflattering for anyone relying on green CI as a safety check.

How the benchmark works

A.S.E 2.0 contains 145 tasks built from real CVEs across 51 open-source projects. Each task runs in Docker and is graded twice: once by functional tests, and once by a validated proof-of-concept exploit. An output counts as secure only if the tests pass and the exploit fails. The dev.to post notes that the paper is a preprint and has not yet been peer reviewed.

Passing tests is not proof of safety

Claude Code running Sonnet 4.5 passed the functional tests on 74.0 percent of tasks, but only 18.4 percent of its output was secure. On 55.6 percent of tasks, every test passed while a working exploit still succeeded. The pattern was consistent across setups: whether the harness was Claude Code, SWE-agent on Sonnet 4.5 or GPT-5.4, or plain retrieval with no agent loop at all, 73.2 to 77.3 percent of test-passing output remained exploitable.

Agents search for functionality, not safeguards

The agent-based setups read far more of the repository than any retrieval pipeline, yet they encountered the relevant safeguard — an existing validator or access policy — in only 27 to 36 percent of runs. Their search behaviour explains why: 71 percent of agent queries named functional concepts such as a file format, while only 9 percent named security concepts.

Context mattered when it was the right context. Runs that both saw and used an existing safeguard reached a 34.5 percent secure rate, versus 12.7 percent for runs that never saw it. Seeing was not enough on its own, though — 41 percent of runs that observed a safeguard failed to use it. Supplying a reference context of annotated safeguards lifted secure output to 24.8 percent, against 12.6 percent with the target file alone, while a generic "write secure code" reminder barely helped at 13.9 percent.

More context and bigger loops did not help

Feeding the model more of the repository was close to useless for safety. A truncated 128K-token repository raised the functional pass rate from 56.9 to 60.9 percent while cutting the secure rate from 12.6 to 12.1 percent. Inserting a single insecure example raised correct-but-exploitable output by 6.3 percentage points, even when the safeguard was present.

The agent loop also bought working code rather than safe code. Claude Code cost $1.41 per task and read 204K tokens, for an 18.4 percent secure rate; the retrieval setup without the loop cost $0.09 per task and read 7.2K tokens, for 17.9 percent.

What the write-up recommends

For teams letting agents touch security-sensitive code, the dev.to post advises naming the safeguard explicitly in the prompt (for example, calling out a specific validation function and permission check), listing validators, auth checks and safe APIs with their file paths in the agent instruction file, and removing or labelling insecure examples in the codebase the agent reads.

For reviewers of agent pull requests, it suggests treating passing tests as a functional result only, verifying that the safeguard is actually called and called in the right place — a permission check that runs too late is a real failure mode in the paper — and not leaning on static analysis alone, which disagreed with the exploit oracle on 36.8 percent of 28,984 functionally correct generations.

Caveats

The tasks come from known CVEs, so they are security-sensitive by construction and may overstate typical risk. The paper tested Sonnet 4.5 and GPT-5.4, not newer models, and a successful exploit demonstrates a vulnerability but cannot rule one out.

Why it matters

Most organisations have quietly adopted "CI is green" as the acceptance criterion for AI-generated code. This benchmark says that on security-sensitive paths, that signal is close to meaningless: most test-passing output remained exploitable across every configuration tested. The bottleneck is not model capability or context size but retrieval — agents do not spontaneously go looking for the safeguards that already exist in a codebase. That points to cheap, concrete fixes: explicit prompts naming security functions, curated context, and human review that checks the safeguard is invoked, not just the tests.