FrontierHarness Eval: same model, nine harnesses, 17x spread in cost per passed task

A benchmark for harness economics

FrontierHarness Eval, a new benchmark project that reached the Hacker News front page on 2 September, asks a question most model evaluations skip: when the model stays the same, what does the choice of agent harness do to your bill? The project ran nine harnesses — Codex v0.148.0, DeepSeek Harness v0.1.0-rc.8, Claude Code v2.1.237, Piv v0.84.2, Oh My Piv 17.4.0, Kimi Code v0.37.2, Exo Harness v0.1.0, OpenCode v1.18.19 and Hermes v0.20.4 — on software engineering and terminal-based tasks, comparing pass rate, median cost per successful task, median cost per task, median cache hit rate per successful task, and median time per successful task.

To keep runs comparable, everything executes on Runta agent runtimes. The harness and the task environment are prepared once as a golden checkpoint, and every run starts from a fresh restore with identical vCPU, memory, disk size, disk contents and memory state. That leaves the harness itself — its prompts, agent loop, tool usage and context management — as the main variable.

A seventeen-fold spread on identical work

According to the FrontierHarness Eval site, median cost per successful task varies 17x across the nine harnesses even though the model is held constant. The project's own commentary also shows that pass rate and cost can move in opposite directions: Claude Code passes 19 tasks but reaches $18.34 in cost per task, an example of strong results bought at a high price rather than an outright efficiency win.

The implication is that the harness layer alone can swing total spend by roughly an order of magnitude on workloads where the underlying model never changes.

Caveats the authors flag

The benchmark is candid about its limits, listing three caveats on the site.

OpenCode's headline cost figures exclude failures and rest on only 15 passes; once failed attempts are counted, the figure becomes $3.24 per task. Definitions of cost per pass therefore differ between rows unless you read the fine print.

Cache hit rate is not a proxy for cost. A cached 300-turn failure can still burn more than a short run that misses the cache entirely, so an impressive hit rate does not make a harness cheap.

Scope is narrow. FrontierHarness v1.0 focuses on software engineering contexts and terminal-based tasks, and the authors say it may not generalize to other areas of knowledge work.

One more thing worth weighing: the site invites developers to test their own harness on Runta and offers $100 in credits to get started, so the project is closely tied to the platform it measures on.

Why it matters

Most teams agonize over which model to use and treat the harness as a neutral shell around it. This benchmark argues the opposite: the harness is a first-order cost and quality decision. A 17x spread means the same model, on the same tasks, can be either a bargain or a budget drain depending on the scaffolding wrapped around it. For anyone budgeting AI-assisted engineering, cost per successful task — not raw pass rate, not cache statistics — is the number worth tracking. And for benchmark readers in general, the OpenCode caveat is a reminder that excluding failed attempts can flatter a harness's economics considerably, so methodology notes deserve as much attention as the leaderboard itself.