Real-SWE benchmark: Fable 5.1 leads at 38.8%, Gemini 3.8 Flash wins on cost

What the benchmark measured

Specific Labs published Real-SWE in September 2026, a benchmark that evaluates frontier coding models on private production codebases rather than public repositories, using tasks licensed from real companies across billing, tax, customer migration and infrastructure work. As a write-up on dev.to reports, the run covered eight model-and-harness combinations, ten tasks and 640 rollouts, scoring resolution as pass@1 averaged over eight independent runs per task with 95% confidence intervals. Tasks executed in isolated sandboxes in Harbor format, with verifiers injected at grading time and drawn from or inspired by each codebase's existing test suites. Reference solutions edited a median of 11 files, against 6 in FrontierCode and DeepSWE, with a median instruction length of 1,742 characters.

The leaderboard

Fable 5.1 running inside Claude Code tops the table at 38.8% resolution, at an estimated $6.96 per rollout — the most expensive setup measured. GPT-6 Astra via Codex CLI follows at 33.8% for $4.67, and Gemini 3.8 Flash via Gemini CLI is the value pick at 31.2% for $2.50. Further down: GLM 5.3 in Claude Code at 28.8% ($5.12), Grok 4.6 in Grok Build and Muse Spark 1.3 in Muse Code tied at 23.8% ($3.44 and $2.74 respectively), Kimi K3 in Kimi Code at 18.8% ($3.90), and GPT-5.6 Sol in Codex CLI at 16.2% ($2.65).

Two caveats apply. Usage data was incomplete for Grok 4.6 and Kimi K3, so their real costs may be higher, though LavX News independently corroborates the overall cost spread. And the top three confidence intervals overlap — roughly 32–45% for Fable, 27–40% for Astra and 25–38% for Gemini — so first versus third place should be read as directionally consistent rather than settled.

Cost follows verbosity, not price lists

According to Specific Labs, Fable 5.1's higher bill comes from writing far more, not from steeper list pricing. On the tax jurisdiction task it produced a mean of 78,000 output tokens per rollout, with measured figures of 86,000 and 88,000 on two other tasks, while GPT-6 Astra stayed within a band of 13,000 to 32,000 tokens. The dev.to piece draws the practical conclusion: the premium buys a few extra points of first-try success with a lot of extra tokens, and if a human reviews every agent patch anyway, a cheaper setup with a second attempt may arrive at the same outcome for less money.

Harnesses are part of the score

Real-SWE explicitly scores a model together with its harness, not a model in isolation. The clearest illustration sits inside a single CLI: Codex CLI reached 33.8% with GPT-6 Astra but only 16.2% with GPT-5.6 Sol, while Claude Code delivered 38.8% with Fable 5.1 against 28.8% with GLM 5.3. Teams comparing agent shells rather than models are therefore comparing a different variable than they might assume.

Where the agents fail

The tasks are hard, and averages hide most of the story. Per Winzheng, six of the ten tasks scored below 15% resolution; the multi-region sweep task reached 67.2% while the tax jurisdiction task reached 3.1%, and the analytics stream reducer produced no successful rollouts at all among the tested combinations.

The failure taxonomy matters more than raw difficulty. Missed requirements is the most common failure category across the field: 67.2% of Grok 4.6's failed runs (41 of 61), 53.8% of Kimi K3's (35 of 65), 38.6% for GLM 5.3 and 36.7% for Fable 5.1. These agents are not mostly failing on compilation — they are failing to do everything the ticket asked. Giving them more time does not fix it either: Superpower Daily found that rollouts under ten minutes failed 71.4% of the time against 73.4% for runs of ten minutes or more, a near-identical rate.

Why it matters

For tool selection, the benchmark reduces to a constraint trade-off. Where review time is the bottleneck and the work touches billing or migrations, Fable 5.1 in Claude Code offers the best first-try success rate and the highest token bill. Where agents run at volume across a large backlog, Gemini 3.8 Flash delivers most of the capability for roughly a third of the spend. The most actionable finding, though, is independent of model choice: since missed requirements dominate failures, writing tickets with explicit, enumerated acceptance criteria attacks the dominant failure mode directly — and longer runtimes demonstrably do not.