Harness design alone lifted SWE-bench bug fixing from 43 to 72 tasks

One model, two harnesses, 29 more bugs fixed

A dev.to article on harness engineering argues that the question practitioners ask most — which model to pick — matters less every quarter. Its central evidence is a paper posted to arXiv in August, identified in the piece as "Same Model, Different Harness": one model, with weights, tasks and context window held constant, was run over 169 bug-fixing tasks from SWE-bench Verified. Only the agent system wrapped around the model changed, and solved tasks rose from 43 to 72.

Frontier models now sit close enough together, the author writes, that the surrounding software decides most of what teams care about: task cost, completion, and whether the output can be trusted.

What counts as a harness

The article borrows a compact definition from Birgitta Böckeler of Thoughtworks, writing on Martin Fowler's site: an agent equals the model plus the harness. The model is rented capacity; the harness is everything built around it — the loop that keeps the agent working, the tools it can call, what enters its context, what it may do, and how its work is checked. Claude Code and Codex CLI are harnesses in this view.

The discipline layered up over time. Prompt engineering handled the words; context engineering handled what travelled with them, such as retrieved documents, memory and summaries of earlier turns; harness engineering adds what a model needs to act rather than talk. A next layer, loop engineering, is already forming: outer loops that re-run an agent on a schedule or event, each ending on a condition a machine can check.

A thirty-line pseudocode loop in the piece shows the imbalance. One line calls the model; every other line encodes a human decision — when to compact, how much of a 4,000-line test log the model sees, whether policy allows a forced git push.

Where the gains came from

Long tasks eventually hit the context limit, and what the harness does at that moment decides whether the agent finishes. According to the article, the paper's improved harness shortened older tool results in stages as the window filled and stepped in when the agent repeated a failing command. Nothing else moved.

There is a caveat: with a 262K-token window, the gap between harnesses nearly disappears. That helps less than it sounds, the author notes, because production systems bill for every token.

Common techniques include compaction (summarising old turns), truncation (trimming old tool output while keeping recent output whole), memory files such as CLAUDE.md or AGENTS.md loaded at session start, and sub-agents that take a side task into a fresh context and return only the answer. The newest is a full context reset, which exists because Anthropic's team building long-running applications found compaction insufficient: as windows filled, models showed what the team calls context anxiety, finishing early in anticipation of the limit. A clean reset with a structured handoff file worked better than a summary the model knew was running out of room.

Guardrails and verification

An agent that runs commands can also delete things, so every harness picks a point on a spectrum. Cline's default waits for approval on every action; Claude Code's auto mode and Cursor's Auto-review route decisions through a second model; Codex CLI defaults to an OS-level sandbox, workspace-only, with networking off; Pi skips sandboxing and prompts entirely, which its author calls full YOLO mode, recommending a container. None is wrong, the article says — the right choice depends on what a mistake would cost.

Verification is what makes agent output trustworthy without reading every line. Böckeler splits it into guides, which steer the agent before it acts (instructions, conventions, examples), and sensors, which check afterwards (tests, linters, type checkers, review agents). The need is real: Anthropic found that agents asked to evaluate their own work tend to praise it confidently even when a human can see it is mediocre. Its remedy used three agents — a planner writing the spec, a generator building, and a separate evaluator testing the running application with Playwright against criteria agreed in advance. The solo agent took 20 minutes and $9; the three-agent setup took six hours and $200 and delivered a far better result.

The shell, meanwhile, remains the workhorse. Exposing one shell tool hands the model every program on the machine, backed by decades of documentation, and the article cites Doug McIlroy's 1978 rule that a program's output should be expected to feed another, as yet unknown, program — a fair description of a language model.

Why it matters

The 43-to-72 jump quantifies what anecdotes have suggested: significant performance gains are available without swapping models, by engineering context management, policy and verification. It also makes the trade-offs explicit — Anthropic's comparison of $9 in 20 minutes against $200 over six hours shows verification quality is bought with time and money, making harness design an economic decision as much as a technical one. And because agents cannot reliably judge their own work, anyone deploying them inherits the job of building the checks that can.