AI agent harnesses vary 2.7x in prompt-cache efficiency, Neuralwatt data shows

Why the harness controls the cache

Neuralwatt, an inference cloud provider that bills on measured energy, has published an analysis of its own traffic showing that the client software wrapping a model — the agent harness — largely determines how expensive each request turns out to be. Writing on the Neuralwatt blog, co-founder and CEO Chad Gibson reports that uncached input ranged from 1.6% to 11.1% across the busiest clients between September 8 and October 7, 2026. For the same conversation, that spread means about 2.7 times the prefill computation.

Prompt caching depends on an exact prefix. When a coding agent sends a 100k-token conversation with a little text appended, the server can reuse its key-value cache for everything already processed and compute only the new part. Neuralwatt says almost all of its traffic is agentic and heavily input-dominated, so this mechanism does most of the work.

If the harness edits anything it previously sent — even a timestamp near the start of the prompt — the server must recompute from the first difference onward. The harness, not the model or the GPU, decides whether that happens.

The measured clients included open-source coding agents such as OpenCode, Pi and Oh My Pi (omp), alongside editor integrations, the OpenAI SDKs and custom apps; OpenCode alone accounts for roughly one in four Neuralwatt customers. Because workload also affects hit rates — long sessions naturally score better, and about half of the lead held by a client Neuralwatt labels deepseek-harness came from users running very long sessions — Gibson points to cache-break frequency as the fairer comparison: about 1 turn in 100 for deepseek-harness versus about 3 in 100 for OpenCode and Pi. OpenCode 2.x logs 2.2% uncached input against 3.0% for 1.x, but Neuralwatt attributes most of that to model mix; on the same model the two cache about equally, even though 2.x fixes most 1.x cache breaks in the lab, a discrepancy the company says it cannot yet explain.

The energy and latency cost

Comparing same-sized requests of about 50k prompt tokens during the week of October 1–7, 2026, Neuralwatt found that a mostly uncached request — one with under 10% of its prompt cached — consumed roughly twice the energy of a fully cached one, and about 2.5 times as much on GLM-5.3. Redundant prefill also contends with generation on the same GPUs, slowing responses for everyone else on that server.

Depth makes misses far worse. Tracking sessions turn by turn on its three busiest models for October 6–13, Neuralwatt found that beyond 128k tokens of context, about 3 turns per 1,000 re-read more than half of their prompt. Those few turns account for 75% of all re-read tokens at that depth and average about 230k re-read tokens each, costing as much prefill as roughly 600 ordinary turns, which add a median of only about 370 new tokens.

What actually breaks it

In lab tests, Neuralwatt ran OpenCode 1.18.34, Pi 1.0.4, omp 18.6.3 and Kilo Code 7.8.3, plus OpenCode 2.0.29, through a recording proxy against DeepSeek V4.1 Flash and GLM-5.3, using a scripted session that covered planning, sub-agents, restarts, MCP connections and compaction. Of 539 requests that followed an earlier one in the same conversation, 88% were pure appends; the remaining 65 broke the cache, and every break traced to the harness rewriting text it had already sent. Idle time, restarts and sub-agents caused none.

Two culprits stood out. OpenCode's plan-mode reminder is attached to a user message and then removed on the following turn, forcing each turn to re-read the previous one — 330 to 2,340 tokens per turn in these tests; OpenCode 2.x moves the reminder into its own message. omp replaces earlier reads of a re-read file with a superseded marker; in one run that edit landed 62% of the way into the prompt and forced a re-read of 5,846 tokens out of 15,520. Neuralwatt cautions that production numbers may also include cache evictions under memory pressure, which it cannot distinguish from harness edits from the server side.

Why it matters

Efficiency debates usually focus on model architecture or accelerator hardware, but this data shows the client is a first-class lever. Every cache break Neuralwatt could reproduce in the lab came from harness behavior, which means teams building or selecting agent harnesses can roughly halve the energy of requests that currently miss the cache, and avoid long visible stalls deep in extended sessions, simply by never mutating prompt history and only appending. Since cost and latency scale with uncached tokens, the choice of harness is effectively a pricing decision.