Only About 25% of Working Code From AI Coding Agents Is Also Secure, Benchmark Finds

A benchmark with two hidden test suites

Researchers at the National University of Singapore and Alibaba Group, including the Qwen team, have published LSCBench, a benchmark that scores coding agents twice: once against hidden functional tests and once against hidden security tests that re-run the exploit each task was built from. According to a dev.to writeup of the paper, which was submitted to arXiv on October 10, 2026, the benchmark holds 450 tasks derived from 150 real vulnerabilities — 115 scenarios from public CVEs plus 35 Java applications written by security experts — covering seven languages and 24 weakness categories.

Tasks come in three sizes: fill in a masked function, rewrite a removed file, or rebuild an emptied source tree from a specification. Two design choices make it hard to game. The prompt never mentions security; agents are asked only for the feature, matching how developers normally write prompts. And scoring is pass@1 with no partial credit and no model-based judging — a task only counts when both the functional and the security suites pass.

Six frontier agents ran at their highest reasoning effort: five inside Claude Code (Claude Fable 5, Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Pro and Kimi-K3), plus GPT-5.6-Sol inside Codex.

Working, but not safe

The core finding is a wide gap between code that runs and code that is also secure. From the researchers' public leaderboard:

• Claude Fable 5 — 91.1% working, 25.1% also secure
• GPT-5.6-Sol — 89.1% working, 24.7% also secure
• Qwen3.8-Max — 89.3% working, 22.0% also secure
• GLM-5.3 — 87.1% working, 21.3% also secure
• DeepSeek-V4-Pro — 73.6% working, 18.0% also secure
• Kimi-K3 — 91.3% working, 16.4% also secure

Kimi-K3 is the outlier: the best functional score in the group and the worst secure rate, so a passing functional suite revealed nothing about which agent produced safer code. Aggregated across all 2,700 runs, 66% ended with code that worked but was still exploitable, and 57 of the 150 scenarios were never fixed by any of the six models.

Where the agents break

The breakdown by weakness type points to a pattern. Memory-safety bugs were handled best — 44% of working code was also secure — because those fixes are local, such as adding a bounds check. Injection followed at 28%, then a steep drop: SSRF at 3% and information exposure at 1%. Those categories require reasoning about the surrounding system, such as where a fetched URL can reach or who else can read a configuration dump.

By language, the share of runs that finished both working and secure was C and Python at 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14% and PHP 10%.

The case studies in the paper, as relayed by dev.to, describe mistakes a careful human reviewer would catch. In one Java task, an agent rebuilding a configuration importer passed all 14 functional tests and failed all 4 security tests because it used SnakeYAML's default constructor, which honors arbitrary class tags in untrusted YAML — a well-known deserialization risk. Two other models picked SafeConstructor and passed both suites. Elsewhere, two models passed user input to a shell as a single string for sh -c instead of handing ProcessBuilder an argument list, reintroducing command injection; in a Go task no model confined file reads to the expected directory, with one agent's code reading file:// URLs; and in another Go task all six models exposed an attestation password in configuration dumps by copying the serialization tags used on neighboring structs.

One sentence helps, a little

The authors reran four models with a single instruction appended to every task, asking for an implementation that meets the functional requirements while also being secure. Secure rates jumped for three of them: GPT-5.6-Sol from 24.7% to 46.2%, GLM-5.3 from 21.3% to 38.2% and Qwen3.8-Max from 22.0% to 37.6%. DeepSeek-V4-Pro barely moved, from 18.0% to 19.1%. Functional pass rates shifted only a few points in either direction, most tasks still failed the security suite, and SSRF and information exposure stayed at the bottom. Claude Fable 5 and Kimi-K3 were not part of the rerun.

Caveats

Every number is pass@1, a single attempt per task. Qwen3.8-Max was also involved in writing the benchmark's tests and specifications; the authors argue its mid-field result shows it gained no advantage from that. And since this is an arXiv preprint, the figures are the authors' own and await independent replication.

Why it matters

Coding agents are now routine in development workflows, and this benchmark puts a number on the accompanying risk: roughly two-thirds of the working code these frontier models produced remained vulnerable to a known attack. The consistent failure mode is omission rather than inability — in each case study a safe alternative existed and the agent never used it. That suggests two practical responses for teams shipping agent-written code. First, functional test suites only verify the feature for its intended user, so security tests that replay plausible attacks need to sit alongside them. Second, a one-line security instruction in agent configuration roughly doubled secure output for three of four tested models while functional pass rates barely moved — cheap to add, but on this evidence nowhere near sufficient on its own.