Identical SWE-bench agent runs still vary by up to 6 points, preprint finds

A preprint study has measured how much an agent benchmark score can move when nothing changes except the run itself. According to a dev.to write-up, the paper — "On Randomness in Agentic Evals" by Bjarni Haukur Bjarnason, André Silva and Martin Monperrus, posted to arXiv in February 2026 and revised in March — found that identical configurations on SWE-bench Verified produce single-run scores spread over 2.2 to 6.0 percentage points.

How the measurement was done

Most benchmark results come from running a configuration once. This study instead ran three models — Qwen3-32B, DeepSWE-preview and Devstral-2 — across two temperatures and two agent scaffolds, nano-agent and R2E-Gym, ten times each on the 500 tasks of SWE-bench Verified. The dev.to summary puts the total at 60,000 trajectories, about 25.6 billion tokens and 1.88 million tool calls.

The headline spread measures repeatability, not capability: it is the gap between the best and worst single run under a fixed setup. One example from the paper's table: DeepSWE-preview on nano-agent at temperature 1.0 averaged 31.4% across ten runs with a standard deviation of 1.0, but individual runs landed anywhere between 28.8% and 32.4%. Two teams that each ran the setup once could honestly report numbers 3.6 points apart while measuring the same system.

Where the instability comes from

The write-up breaks the variance into three parts. First, sampling: above temperature zero, every token is drawn from a distribution, and one different token reshapes all the context that follows. Second, the scaffold, which shifts both the level and the spread — DeepSWE-preview scored 31.4% ±1.0 on nano-agent but 34.4% ±1.5 on R2E-Gym, while Devstral-2 went from 63.5% ±1.1 on nano-agent at temperature 0.2 down to 34.9% ±1.5 on R2E-Gym. Third, the runtime, which matters even at temperature zero.

Temperature zero does not remove the noise

Greedy decoding is often assumed to be reproducible. The paper argues it is not: floating-point addition is not associative, and differences in parallel reduction order, hardware kernels and batching can nudge the logits slightly, with autoregressive decoding amplifying the drift. Standard deviations shrink at temperature zero but never reach zero — the write-up cites figures such as 16.4% ±1.2 for Qwen3-32B on nano-agent and 22.3% ±1.8 for Qwen3-32B on R2E-Gym, with 1.8 points the largest temperature-zero deviation in the table. Lower temperature does not reliably tighten results either: Qwen3-32B on nano-agent showed a standard deviation of 0.7 points at temperature 0.6 and 1.2 points at temperature 0.

Runs diverge at the start

Divergence happens near the beginning of a run. For DeepSWE-preview on nano-agent at temperature 1.0, the median position of the first differing token between two runs was token 5, roughly half a percent into the trajectory; at temperature zero it moved to token 56. The paper concludes the first divergence almost always falls within the first 1% of a trajectory, making everything after it a consequence of one early token. In one cited case — task django__django-9296 with Qwen3-32B at temperature zero — two runs stayed identical for 93 tokens, then split when one searched for Django's Paginator class while the other read the source; one produced a misplaced patch and the other passed.

How many runs make a result trustworthy

The paper's power calculation, which assumes approximately normal single-run scores (Shapiro-Wilk tests passed for 11 of 12 configurations), uses the observed standard deviations of 0.7, 1.5 and 1.8 points. Detecting a 2-point improvement at p < 0.05 takes roughly 2, 9 or 13 runs depending on the variance; a 1-point improvement takes about 8, 36 or 51. A 10-point gap shows up in a single run. The write-up also separates real effects from noise by size relative to the spread: an 11-point temperature difference for DeepSWE-preview is treated as genuine, while a 0.3-point difference for Devstral-2 is not.

Limits of the study

The dev.to piece is explicit about what the work cannot show. It is an unreviewed preprint covering one benchmark, three models and two scaffolds; no code or data artifact is linked; there is no indication of when the runs were collected; and temperature-zero variance will depend on the specific inference stack. Variance from too few tasks is also a separate problem from variance from too few runs, and only the second is addressed here.

Why it matters

Agent benchmarks increasingly drive model selection and purchasing decisions, and this paper quantifies how fragile a single number is. If two runs of the same system can differ by several points, a claimed 3-point improvement means little without a run count and a variance estimate behind it. Anyone evaluating coding agents should treat a leaderboard score as one sample from a distribution — and budget for the repeated runs needed to tell a real gain from noise.