One green run means little: 77% agent pass rate drops to 53% over five runs

A measured 24-point reliability gap

Testing an AI agent once per case can overstate how it behaves in production, and a recent IBM Research paper puts a number on the problem. According to a dev.to post discussing the work, the paper — Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course, by Evelyn Duesterwald and colleagues — ran a ReAct agent built on GPT-4.1 through the AppWorld benchmark, executing every task five times. The average pass rate per run came out at 77%, yet only 53% of tasks succeeded in all five attempts. The researchers call the 24-point difference the consistency gap, and it appeared on a controlled benchmark with no environmental noise to blame.

The post's author adds a production anecdote of the same failure mode: an invoice-triage agent that passed 22 test cases three days in a row later filed one PDF under two different vendors when it was uploaded twice ten minutes apart, with no prompt, model or data changes in between.

Why one green run is not evidence

Temperature looks like the obvious cause and is the wrong one, the post argues. Even set to zero, variation remains from provider-side factors such as request batching, hardware differences and model updates shipped behind a stable name. More importantly, the agent loop amplifies small deviations: a marginally different tool call at step three changes the observation at step four, and by step eight the agent is on a different trajectory entirely.

The paper's approach

The researchers' contribution is a fix rather than the diagnosis. A Consistency Analyzer compares the five recorded trajectories for a task and pinpoints the step where they diverge. A Guideline Generator then writes a short, targeted instruction for that step and stores it as episodic memory, injected when the agent meets a similar task. On AppWorld this lifted the all-five pass rate by 16 points on the original tasks and by 13 points on similar but unseen tasks. The authors present this as narrowing rather than eliminating the gap, and the post's author finds the unseen-task figure the more convincing one, since guidelines can be overfitted to a fixed task list.

Practical changes that need no framework

The post describes three changes the author made without adopting the paper's machinery:

• Every eval case runs N times, and the report shows both the per-run rate and the all-N rate, alongside a counter of tool-call paths per case. That counter exposes where divergent runs split: the author's flaky invoice cases all diverged at a vendor lookup that sometimes returned two matches, which one line in the system prompt fixed.
• The all-N rate becomes the headline number reported to clients, with the per-run rate demoted to a footnote, on the argument that the all-N figure is what predicts support tickets.
• Flaky cases block release. A case passing four of five runs is treated as a bug with a known reproduction: the unstable step either becomes deterministic through a rule, lookup or hard filter, or the path gets a human approval gate.

Caveats worth noting

Five repetitions may be too few for high-volume automation, the author cautions; an agent running 400 times a day with a 2% flip rate still produces a steady stream of wrong answers, so N should be chosen from call volume rather than from the paper. Injected guidelines also carry a cost, since accumulated rules add context weight and can degrade cases that were never flaky, a trade the post says the paper does not quantify. The published numbers cover GPT-4.1 only; the author's informal five-repeat run with Claude Sonnet 5 produced 91% per-run and 79% all-five pass rates — a smaller gap with the same shape, though from a small dataset over a single evening. A related arXiv paper, AgentAudit, approaches the problem from another angle, evaluating full traces across planning, tool selection, execution and memory to attribute failures to a specific stage.

Why it matters

A system that fails on a known 23% of inputs can be routed around; one that handles identical input differently from day to day cannot, because there is nothing stable to route on. Single-run evaluations therefore systematically overstate agent reliability, and the gap between per-run and consistent pass rates is arguably the more decision-relevant metric. A five-repeat harness over existing test cases is cheap to build, and as the post shows, the tool-call divergence patterns it surfaces often point directly at fixable steps.