Audit found flaws in 30% of Terminal-Bench, shifting agent accuracy by over 10 points

A post on dev.to about Terminal-Bench makes a case that coding-agent benchmarks decay on their own: not because tasks change, but because models keep discovering solutions that are genuinely correct while automated verifiers reject them. The proposed remedy is to run benchmark evaluation like a continuously maintained codebase rather than a fixed dataset.

The motivating numbers come from Terminal-Bench 2.0, which shipped after extensive human and LM-assisted review. A later audit still found problems in roughly 30% of the benchmark, and according to the post, fixing them shifted measured accuracy by more than 10 percentage points and moved leaderboard positions by as many as three ranks. The author's argument is that this reflects no carelessness in the original release — a benchmark simply behaves like software, with all the upkeep that implies.

How benchmarks go stale

The post breaks the decay into three compounding layers. First, verifiers go stale: an agent may produce a correct solution arranged differently than expected — a checker that assumes one file layout will fail equivalent code written another way. Second, infrastructure drifts: harnesses change, dependencies update, and timeout assumptions stop matching reality, so a task designed around 60-second execution may now need 120 because the agent explores more paths. Third, specifications are ambiguous: implicit assumptions only surface when models probe edge cases, as with a profile-creation task that never says whether duplicate usernames should error or be overwritten — both behaviors are defensible, but the verifier accepts only one.

These layers compound. When a stale verifier produces false negatives, teams start tuning agents to satisfy the checker rather than solve the task, and the score ends up measuring compliance with the verifier instead of capability.

A repair pipeline modeled on CI

Terminal-Bench's answer is a workflow that mirrors software CI/CD. Tasks, verifiers, agent traces and oracle checks live in a Git repository with tagged releases, so historical comparisons reference commit hashes rather than a floating "latest" pointer. Users file reports when an agent produces a valid solution that fails verification, attaching the trace, the failing test and a human explanation of why the solution is right. Before any repair merges, an automated oracle check replays the updated verifier against stored agent outputs and flags any case where a verdict flips, sending it to human review. A maintainer then approves, merges and cuts a new version, with the old one kept available.

Concretely, the pipeline has five stages: run the agent and capture stdout, stderr, filesystem changes and API calls; run the verifier; store the trace in object storage under a content-addressed key; triage reported issues against the task spec; and validate repairs against historical traces for regressions. Because evaluation is deterministic given a trace and a benchmark version, verifiers can be rerun against old traces without spending agent quota — which is what makes it possible to know which past results a verifier change would have affected.

Resource budgets change the question

The post also reframes time and resource limits as part of the specification. Doubling the wall-clock limit from 60 to 120 seconds closed a 20-point accuracy gap on GPT-4o and shrank a 30-point gap on GPT-4o mini to 5 points, per the post. That is not a bug: the benchmark measures what an agent can do within a budget, not with unlimited time, so changing the timeout changes the question. Token budgets, API rate limits and memory ceilings behave similarly, and the post argues they must be versioned alongside the tasks — otherwise a leaderboard entry from six months ago may have run under different limits and be incomparable.

Failure modes of the continuous approach

Continuous maintenance brings its own risks: version proliferation (rerun every agent on each new version, or maintain separate leaderboards), brittle oracle checks that can reject valid repairs when the historical trace corpus is unrepresentative, maintainer queues growing faster than reviews, and leaderboard churn that tempts teams to chase rankings rather than capability. The suggested mitigation is batching repairs into quarterly releases and keeping a stable frozen version for official comparisons while the continuous pipeline runs in the background.

Monitoring and isolation

Debugging staleness requires metrics at three layers: per-task pass rate, execution time and token consumption, where a sudden pass-rate drop points at the verifier and runtime spikes suggest infrastructure drift; verifier-level counts of false positives and false negatives; and agent-level behavior such as repeated retries, which can signal valid outputs being rejected. The post describes a dashboard that opens a review issue whenever a task's pass rate falls more than 10 points between versions. Because the harness executes untrusted agent code, it runs in ephemeral containers without network access, with filesystem writes confined to a sandboxed directory and CPU and memory capped.

Why it matters

The Terminal-Bench figures show that benchmark validity erodes without anyone touching the tasks. For anyone evaluating models, the practical takeaways are to pin benchmark versions, record resource budgets alongside results, and archive traces so past runs can be re-verified when verifiers are repaired. For teams building agents, a failed benchmark task may indict the verifier rather than the agent. More broadly, a leaderboard score is only as trustworthy as the maintenance process behind it.