Trail of Bits releases patching-bugs and regression-hunt plugins for AI agents

Trail of Bits has released two plugins, patching-bugs and regression-hunt, that let engineering teams hand patch development and regression detection over to AI agents. The announcement, published on the security firm's blog and surfaced on Hacker News's front page, frames the tools as a way to make agent-assisted security patching repeatable and reviewable.

What the plugins do

According to Trail of Bits, patching-bugs reproduces a reported bug, develops a fix and then tests that fix. Its companion, regression-hunt, examines whether a change breaks behavior that used to work and that the project supports. Engineers stay in control: they direct the agents' work and review the output, while maintainers can define the security assumptions and patch requirements the agents must respect, keeping the work inside the project's own rules.

Both plugins grew out of Patch the Planet, a joint initiative with OpenAI that has co-authored hundreds of security patches for widely used open-source projects, including Scapy. Trail of Bits says the design draws on an analysis of merged patches plus thousands of its engineers' Codex agent sessions.

A lesson learned from its own patches

Part of the motivation came from the company's own missteps. A September Trail of Bits post described regressions introduced by some Patch the Planet patches. In the review that followed, the company examined every patch its engineers had written that maintainers merged upstream, and confirmed 21 regressions across more than 130 merged pull requests: 16 functional, one performance-related and four affecting builds or tests. None of them weakened a security protection, the company says.

Those 21 confirmed regressions surfaced among 58 candidate findings in a single run, roughly 36% precision, and because agents take different approaches on every pass, the pile of candidates to triage kept growing. Trail of Bits drew two conclusions: patch writing needed a standard procedure applied identically each time, and regression checking needed written rules for what actually counts as a regression.

Design built on live proof

The plugins follow four design principles, per Trail of Bits: brief, opinionated guidance; modular steps that can run alone or together; strict input and output schemas, so each step checks the previous step's raw evidence instead of trusting its summary; and a live proof of concept for every claim, meaning a built, running instance of the project exercised through the interfaces a real user would touch. If a step cannot produce a live PoC, its result is marked inconclusive.

regression-hunt traces the changed code through its callers to its real interfaces to enumerate candidate regressions, reproduces each candidate on a live deployment, replays the proof with a separate agent holding no prior context, and finally adjudicates: the agent must rule out false positives, incomplete fixes, latent bugs and pre-existing bugs before anything is reported. When a threat model is available, it is passed into the context of every step so the agents must consult it before reaching a conclusion.

patching-bugs adds what Trail of Bits calls a "three-arm proof": the original code must fail the reproduction, the patched code must pass it, and a full revert of the patch must fail again, tying the outcome to the patch itself rather than to environmental noise. It also steers agents toward repairing the violated security invariant rather than the surface symptom, and it requires a regression-hunt pass on every patch it produces.

Fewer false positives, with a tradeoff

Evaluated against the 21 confirmed regressions, regression-hunt produced 25% fewer false-positive findings than a baseline Codex agent simply prompted to "review the PR for regressions," with a 95% confidence interval of 14% to 38%. The strict evidence bar carried a cost: some known regressions stayed classified as borderline, meaning the plugin flagged them but could not gather enough evidence to confirm them. Trail of Bits says it will investigate those cases next and try to confirm more real regressions while giving maintainers clearer signals when evidence stays inconclusive.

Why it matters

Bug disclosures now arrive at a pace that manual patching struggles to match, and agent-written patches are becoming routine. But an agent that fixes one bug can quietly break something else, and Trail of Bits's tally of 21 regressions in 130-plus merged patches shows the risk is not theoretical. By standardizing both how patches are written and how regressions are defined and proven, with live proofs of concept, schema-checked steps and maintainers' threat models baked in, the plugins offer a template for making AI-assisted patching auditable. The 25% reduction in false positives is modest and the borderline tradeoff is unresolved, but the structure itself, including an explicit burden of proof on the agent, points toward patch pipelines that maintainers can actually trust.