Real-SWE benchmark: longer agent runtimes barely help, and the harness hides the model

Specific Labs has released Real-SWE, a benchmark that evaluates coding agents against licensed, private enterprise codebases — billing systems, tax logic and multi-service integrations — rather than the public repositories most software-engineering benchmarks rely on. According to a pair of analyses published on dev.to, two findings stand out from the leaderboard: giving an agent more wall-clock time barely changes whether it solves a task, and the CLI harness a model runs in can disguise which model, and how capable a one, is actually doing the work.

More minutes, roughly the same outcomes

The duration data is strikingly flat. The dev.to analysis reports that 71.4 percent of rollouts finishing in under ten minutes failed, while 73.4 percent of rollouts that ran ten minutes or longer also failed. Overall pass rates sit in a narrow 27–29 percent band regardless of runtime, so stretching a rollout from a few minutes to a much longer session shifts the result by roughly two percentage points — a difference the analysis characterises as noise.

The suggested explanation: quick, shallow tasks get solved fast, which creates an impression of high throughput on short rollouts. The work buried in real payroll, tax and integration code instead runs into limits that are structural rather than computational. The agent is not exhausting its compute budget; it is missing understanding or context, or the harness never hands it a usable entry point. Extra rounds of retries fix none of those things.

Same harness, very different scores

The leaderboard also argues for reading harness names carefully. Real-SWE lists GPT-6 Astra on Codex CLI at 33.8 percent resolution and GPT-5.6 Sol on Codex CLI at 16.2 percent — one vendor's CLI, one benchmark, and more than a 2x gap between the two numbers. The same spread appears under Claude Code, where Fable 5.1 reaches 38.8 percent and GLM 5.3 lands at 28.8 percent, ten points apart behind a single harness name.

Per the dev.to analysis, Real-SWE presents every result as a model-plus-harness combination rather than a model in isolation, an approach the author considers more honest than most leaderboards, since vendors rarely publish a low score achieved under their own tooling. The practical implication is that a number attributed to "Codex CLI" or "Claude Code" describes the pair, not the product. Swapping the model inside a fixed harness is the single biggest lever on the outcome, and it is largely invisible in how these tools are marketed.

Even the leader fails most enterprise tasks

The top configuration on the board, Fable 5.1 paired with Claude Code, resolves 38.8 percent of tasks, with GPT-6 Astra on Codex CLI close behind at 33.8 percent. Even the best-performing pair therefore fails roughly six out of ten private enterprise tasks — the kind of gap that polished vendor demos tend to leave out.

Why it matters

For teams shopping for coding agents, knowing that a team uses a particular CLI says almost nothing about the skill of the agent they get; the model running inside it dominates the result. When a vendor shows you a pass rate, identify both components behind the number — the model and the harness, including its conventions, context handling, tool loop and judge — and ask which slice of tasks the figure came from. A model that looks strong because it clears fast, shallow work may be hiding exactly the harder set you need solved. The dev.to posts also warn that leaderboards publishing bare model names without pinning the scaffold invite meaningless cross-comparisons, and that changing the scaffold can swing a score more than cranking up reasoning effort. If a vendor will not say which model produced a benchmark number, treat that omission as a warning sign rather than a detail.