· via Hacker News – Front Page (native)
Coding agent benchmark of 27,280 runs finds 64% of inference spend on failed runs
RUNTA's FrontierHarness Eval v1.1 ran 27,280 coding agent trials and found 64% of inference spend went to failed runs, while the cheapest harness changed with every model tested.

What was measured
RUNTA, working with Baseten, has released FrontierHarness Eval v1.1, a large-scale benchmark of coding agents that consumed 83 billion tokens across 27,280 agent runs. The study crossed 22 harness configurations — 18 harnesses, with the DeepSeek Harness counted in four profiles and OpenCode in two versions — against five models on 62 terminal tasks, with four attempts per combination. That is 76 times the scale of v1.0, which tested nine harnesses on a single model, Kimi K3. The five models were GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash 1017, Muse and Kimi K3, all served by Baseten.
Each harness ran with its default settings at maximum reasoning effort under the Harbor evaluation framework, and every trial got a fresh, disposable runtime so no trial could see an earlier one's files. Task time limits ran from 15 minutes to eight hours. Audits of the trial logs flagged 216 trials for reward hacking, 163 of them online lookups, and those trials were scored zero.
The Hacker News headline for the report carries its sharpest number: 64% of the inference spend across the 27,280 runs went to runs that failed.
The model sets the bill, the harness moves it
According to RUNTA's analysis of variance on average cost per task, the model accounts for 92.8% of the spread, the harness 2.3%, and the harness-model interaction 4.9% — unsurprising given that the models' list prices differ by 20 times or more. But hold the model fixed and the harness still matters: average cost per task varied by a factor of 1.8 to 5.0 depending on the pairing. Shuffling trials between harnesses 300 times produced chance ratios of only 1.2 to 1.8, and every real ratio exceeded all shuffled ones (p = 0.003), so the spread comes from the harnesses, not rerun noise. The widest gap was with DeepSeek, where Cline CLI cost 5.0 times as much as OpenCode v1 for 2.0 more points of pass rate.
Rankings collapse when you swap models
The cheapest harness for one model is frequently middling for another. The median harness moved 11 of 22 places in cost rank between models; Codex CLI went from 1st on cost per task with GLM-5.3 to 16th with Kimi K3, and the Creator profile of the DeepSeek Harness fell from 1st with Muse to 21st with GLM Flash. Split-half checks show this is signal, not noise: rank correlations were 0.71 within a model but only 0.26 across models.
RUNTA's best-value pick per model (lowest cost per pass, with a tie-breaker allowing up to 30% more cost for at least 5 more points) was Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek, Oh My Pi with Muse and Alma with Kimi K3. A bootstrap redrawing the 62 tasks 4,000 times kept Codex CLI first with GLM-5.3 in 99.6% of draws; the other four winners were far less settled, at 53%, 47%, 42% and 25%.
Paying more does not buy passes
Across all models, only four harnesses sat on the efficiency frontier, with no other harness beating them on both cost and pass rate: OpenCode v2 (lowest cost per task, $0.561), Pi, Codex CLI and ZCode (highest pass rate, 61.9%). Every other harness cost up to 1.6 times as much while passing between 50.6% and 60.2% of tasks. The harness with the highest pass rate ranked between 5th and 20th of 22 on cost per task, and with four of the five models the most expensive harness per pass actually passed less often than the cheapest. Correlations between cost per task and pass rate ranged from -0.39 to +0.22. On cost per pass across all models, Codex CLI was cheapest at $0.937 and Antigravity CLI the priciest at $1.63.
RUNTA also checked whether simpler harnesses are cheaper. Prompt size, tool count and cache hit rate were not the main cost drivers; the number of steps in the agent trajectory was.
Why it matters
For teams running coding agents, the actionable finding is to benchmark the pair, not just the model. Switching models while keeping the same harness can quietly multiply inference bills, because a harness that is cheap with one model can be several times more expensive with another. Splurging on a heavier harness is not a shortcut either, since the study found no consistent link between spending and pass rate. And with roughly two-thirds of inference dollars going to failed runs at these pass rates, cost per pass — not cost per task — is the number that reflects what a solved problem actually costs. The caveats are real: one lab setup, default settings, 62 tasks, five models, and several winning pairings that the bootstrap shows are statistically close.
- #benchmarks
- #coding-agents
- #llm
- #cost-optimization
- #evaluation