Four harness bugs explain why an agent burned 3.1x Codex CLI's tokens

A post-mortem published on dev.to by a member of the team benchmarking the Octomind agent explains how their client ended up consuming 3.1 times the tokens of Codex CLI on its worst tasks, and what a manual read of 150 execution traces revealed about where the waste actually came from.

What was measured

According to the dev.to post, the team ran Octomind 0.70.1 head-to-head against Codex CLI 0.167.0 on the GOLD benchmark: 65 tasks and 150 paired runs, with both clients using the same model, gpt-6.1-sol. Pairing means each task went to both clients inside the same time window rather than on separate days.

The averages looked healthy, but six tasks ran hot, using up to 3.1x the tokens and 2.3x the wall-clock time of Codex CLI. After the fixes prompted by the trace analysis, per-task ratios settled at 0.75x tokens, 0.87x cost and 1.08x time. A full round consumed 76.6 million tokens against Codex CLI's 122.4 million, with reasoning tokens at 96K per round versus 196K for Codex.

The author adds a methodological caveat: only 38 of the 65 tasks were solved by both clients, and that overlap is the only fair basis for comparing token efficiency. A run that gives up early looks cheap for the wrong reason, and one that thrashes for minutes before failing looks expensive for reasons unrelated to efficiency.

One extra call is a multiplier, not a line item

The framing insight of the analysis is arithmetic rather than a bug. Every call in an agent loop against a stateless completions API re-sends the entire conversation. A spurious call early in a long task is therefore not a one-off expense: the accumulated prompt is billed again on that call and on every call after it, because the extra turn becomes part of the history.

Four problems the traces exposed

The first was a replay bug. A code path that reconstructed conversations and handed them back to the model inserted assistant turns the model had never produced. The model then had to reconcile a transcript in a style it never used, which pushed reasoning tokens up on every subsequent call. The author found it by diffing what was sent against what the trace recorded as received.

The second was a harness instruction asking the model to emit a work-in-progress status message with no tool call attached. In a tool-calling loop, an assistant message without a tool call is how a turn ends, so the instruction collided with the loop's termination rules and left the model grinding on work with no clean exit. A grep across 241 Codex CLI messages found zero instances of the pattern, simply because Codex never receives that instruction.

The third was a confidence gate set at 0.8 that was supposed to short-circuit work the model was already sure of. It never fired once in 40 runs, meaning the code path was dead and every run the author had reasoned about as gated was not. The advice given: log the values passing near any threshold, and delete or move a gate whose histogram is empty on one side.

The fourth was timeout handling. When calls hit a 300-second timeout, tokens already streamed were still billed, and roughly 14,000 of them across the runs were whitespace. A related hang remains open: one run sat on a silent stream for 336 seconds, longer than the timeout meant to guard it, with no diagnosis yet.

Where the agent still loses

Runtime is still 1.08x, and the token savings did not fix it, because wall clock is dominated by round trips and provider latency rather than token volume. One task, Pydantic, regressed to 1.57x tokens after the fixes; the author suspects removing extra calls reshaped the history and pushed that task down a longer path, but explicitly labels this a guess.

A benchmarking trap

Before the paired runs, the two clients were compared a week apart, showing averages of 11.9 seconds per model call for Octomind versus 7.2 for Codex CLI. The author attributes that gap to shifting provider load and routing rather than the clients, and says the team came close to spending a week optimizing latency that belonged to someone else's infrastructure.

Why it matters

For teams building or evaluating LLM agents, the post is a concrete case that cost overruns often live in harness design rather than the model. Instruction wording can silently collide with loop termination rules, dead gates create the illusion of architecture, streamed tokens are billed even when stalls produce nothing, and averages hide the tail where the bugs sit. Just as importantly, comparing agent runs from different time windows measures the provider, not the client; paired, same-window runs are the reliable way to benchmark.