Nine coding harnesses on one laptop: prompt size, not token speed, decides usability

A developer has run nine coding harnesses against a model served entirely from a laptop, and the resulting measurements explain why raw throughput benchmarks say so little about how these tools actually feel to use. The write-up, hosted on Notion and featured on Hacker News's front page, focuses on the gap between what a tool like llama-bench promises and what happens when a harness built for cloud APIs meets localhost.

What was tested

The author put mini-swe-agent, pi, cline, codex, dsh, goose, crush, opencode and chad through 24 tasks built from eight Exercism exercises, each in auto-approve mode with an identical one-sentence prompt. The hardware was an M4 MacBook Pro with 24GB of RAM running macOS 26.6.2, serving a 3-bit quantization of Qwen 3.8 27B through llama.cpp build 10470. Every harness shared the same llama-server instance, a proxy enforcing Qwen's recommended sampling settings (temperature 1.0, top_k 20, top_p 0.95, min_p 0.05), and the same 32,768-token unified cache spread across four slots. All figures come from llama-server's own accounting rather than harness self-reports, except the chad rows using its in-process MLX engine, which were taken from chad's prefill trace.

Where the time goes

The analysis starts with laptop physics. A machine in this class reads a prompt at roughly 90 tokens per second and generates around 10, so every 1,000 tokens of prompt cost about 11 seconds of silence before the model writes anything. The system prompt plus tool schemas ranged from 1,171 tokens for mini-swe-agent to 18,046 for opencode, with pi at 2,008. On a data-center GPU prefilling at 10,000-plus tokens per second, the difference between those extremes is 0.2 versus 1.8 seconds. On the laptop, the author measured 22 seconds versus 226.

Context is the second cost. With a reasonable model on a reasonable laptop, the author estimates about 32,000 tokens of working room once weights are loaded. pi leaves 94% of that budget for the actual task; opencode, having spent 18,046 tokens before the conversation begins, leaves 44%.

Side requests are the third. Harnesses assume a remote server and fire extra calls for things like session titles and summaries. Locally the laptop is both client and server, so those calls queue behind or collide with the agent's turn. Over the 24 tasks, opencode issued 33 such requests, crush 51 and dsh 24, nearly all overlapping an agent turn. The model was effectively busy for 125% of wall-clock time under opencode and 114% under crush, meaning two requests competing for one GPU.

How the harnesses fared

Measured wait before the first token: mini-swe-agent 12.2 seconds, pi 21.6, chad on llama.cpp 25.6, cline 64.1, codex 87.8, dsh 94.4, goose 110.3, crush 199.8 and opencode 225.7. Chad's in-process MLX engine cut that to roughly 4.7 seconds, and its dflash2 variant reached 17.4 experienced tokens per second, a figure that counts generated tokens over total wall clock including prefill and tool time, versus roughly 5.7 to 8.1 for the rest.

On the pass gate, 24 deliberately trivial Python tasks that the author says should not be read as a ranking, chad on llama.cpp completed 24/24, goose 22/24, pi and codex 19/24, dsh and crush 18/24, cline 17/24, opencode 15/24 and mini-swe-agent 11/24 with 14 timeouts at the 1,200-second cap. Cache reuse sat between 94% and 100% for current versions; the author notes goose only earned its spot there at version 1.50.0, because earlier builds re-rendered a minute-resolution timestamp into the first user message each turn, dropping cache reuse to 78%.

The author sorts the field into three groups: lean tools that work locally out of the box (pi, mini-swe-agent, chad, with mini-swe-agent's timeout count as the caveat); heavier tools whose prefixes are byte-stable so the pain is front-loaded (dsh, cline, codex, goose); and two, crush and opencode, where nothing appears on screen for three to four minutes.

Caveats worth noting

The author discloses a direct stake: they have been tinkering with chad, a harness tuned specifically for Qwen 3.8 27B on Apple silicon, which is exactly the configuration tested. Experienced throughput also varied by up to 50% between nights, so fine differences among the lean tools are not meaningful. The piece is explicitly framed as measurement rather than criticism: opencode's large prompt exists because it helps frontier models behind an API, and crush's 26 tool schemas are harmless with 200k context and near-instant prefill. Those choices only break down when prefill stops being free.

Why it matters

Throughput benchmarks measure the engine, not the vehicle built around it. As local models become genuinely usable on consumer hardware, the harness becomes as decisive as the model: prompt size, tool schema count, cache stability and background requests now determine whether a local setup feels responsive or unusable. The pass rates underline the point, since the heaviest harnesses completed fewer tasks rather than more, while a configuration designed for the local case finished all 24 within budget. For harness developers, the write-up doubles as a checklist of assumptions that quietly stop holding the moment the data center is swapped for localhost.