Empirical study of 176 harness settings shows coding agent design choices hinge on model strength

A new empirical paper dissects the scaffolding around coding agents and measures which components actually earn their place. Posted to arXiv on 17 September 2026 and featured on Hacker News's front page, the study asks how harness design — the planning, tooling and context handling wrapped around a model — shapes long-horizon software-engineering performance. Its starting point is a gap in the literature: existing evaluations tend to treat harnesses as monolithic systems, so a benchmark score cannot be traced back to any individual design decision.

A fixed loop with three variable parts

According to the paper, the authors built a lightweight coding harness whose execution loop is held constant while exactly three components vary: planning, the action space and context management. Four models were evaluated on SWE-Bench Verified and Terminal-Bench 2.1, producing 176 matched settings that span five context-management strategies, four context-window budgets and targeted ablations of planning and action space. Because the surrounding machinery does not change between runs, differences in outcome can be attributed to the component under test rather than to incidental plumbing.

Context management works mostly as overflow insurance

The study finds that the value of context management grows as the context-window budget shrinks, and that most of that benefit comes from preventing context-overflow failures rather than from making the agent better at the underlying task. With a generous window, the choice of strategy matters far less.

Among the five strategies tested, the most efficient applied deterministic, rule-based trimming first — stripping content before the model ever sees it — and LLM-based summarization second. A seemingly attractive extension, making the trimmed content recoverable so the agent could fetch it back later, added machinery that models rarely used and produced no accuracy gain. For builders weighing whether to implement retrieval of elided material, the data here suggests the complexity does not pay for itself.

Planning and tooling depend on model strength

Planning played two distinct roles depending on the model. For weaker models it functioned as a support that lifted accuracy directly. For stronger models its main effect was reducing cost, with little movement in accuracy.

The action-space results follow the same logic. Predefined tools improved performance for models with weaker command-line skills, but models already fluent in bash ran effectively on a bash-only interface at a much lower cost, with the gap widest on command-line-centric tasks — consistent with the Terminal-Bench half of the evaluation.

What the trajectories reveal

To explain these effects, the authors analyzed agent trajectories rather than just final scores. Context management lengthens execution trajectories without substantially changing how the agent behaves: runs simply keep going instead of dying on overflow. Planning changes the point at which trajectories stop. The action space changes the granularity at which code gets written. The result is a mechanical account of the benchmark numbers, not only a statistical one.

Why it matters

Harness design for coding agents has largely been guided by convention and anecdotes from individual teams. This paper replaces some of that folklore with matched measurements, and its conclusions translate into practical guidance:

• Match context-management effort to your context budget. The tighter the window, the more the strategy matters, and simple staged trimming plus summarization beat heavier machinery on efficiency.
• Skip recoverable elision unless you have evidence your models use it; in this study they rarely did, with no accuracy to show for it.
• Treat planning and predefined tools as model-dependent choices: useful scaffolding for weaker models, cost levers or unnecessary overhead for stronger ones.
• A bash-only interface is a legitimate, cheaper option when the underlying model has strong shell skills, particularly for terminal-centric work.

Beyond the specific findings, the fixed-loop, component-swapping setup offers a modular framework for evaluating future harness components — effectively a way to benchmark harnesses themselves rather than only the models running inside them.