A 0.005% weight difference moved benchmark scores by 1.4 points, prompting a retraction

A rounding-level weight change produced a 1.4-point gap

A small developer lab has publicly retracted a model-improvement claim after finding that its evaluation pipeline could not separate real gains from numerical noise. In a post on dev.to, vidraft_lab described comparing two files of a 4B-class model on 400 multiple-choice reasoning questions, graded by generating answers at temperature 0 with a cap of 8,192 tokens. The files differed in only about 230,000 of roughly 4.66 billion weights — around 0.005% — and the largest relative difference was about 1/25th of a single bf16 rounding step, an artifact of the tools that produced the files rather than an intentional change.

Despite that, the two files scored 0.7744 and 0.7889, a gap of about 1.4 percentage points that the team calculated at 3.9 standard deviations. Running the identical file three times yielded scores within 0.17 points of each other, which had suggested stability. But files differing only at rounding level spread across roughly 1.2 points. Repeating one file, the authors conclude, understates the variation between near-identical files by about seven times.

Why near-identical weights can diverge

According to the post, two effects overlap. Inference servers are generally not batch-invariant: the authors cite a September 2025 write-up from Thinking Machines Lab in which one prompt run 1,000 times at temperature 0 on Qwen3-235B produced 80 distinct outputs, the most common appearing only 78 times, because kernel outputs depend on batch size and batch size changes with server load. Thinking Machines Lab reported that batch-invariant kernels removed the divergence, at the cost of latency.

Long generations then amplify the divergence. If a last-bit logit difference flips an early token, the rest of the reasoning chain branches. The team observed truncation rates of 16.75%, 16.25% and 18.00% across runs of the same setup. The authors note this is an explanation grounded in prior work and observation, not a causal decomposition they measured themselves.

The baseline trap

The instability had a concrete cost. One candidate had been reported as 1.06 points above the original file. But every candidate had been produced by a tool that left the rounding artifact described above, while the baseline had not passed through that tool. When the team rebuilt the baseline with the same tool, it scored 0.7889 and the candidate's edge became -0.39 points; no candidate showed improvement, and the report was withdrawn. The rule the team draws — build baselines with the same tool used to build candidates — also applies to model merges, checkpoint selection and quantization comparisons, where quantization is essentially rounding.

Truncation, not ability

A second problem surfaced late. Across five models, generative scores tracked truncation rates almost exactly, and answers cut off at the token cap count as wrong. Model D, truncated on 47.2% of questions, scored 0.948 on the questions it finished and 0.249 on the truncated ones. Re-measured with deterministic log-probability grading on 1,198 questions — one forward pass per question, no generation, no truncation — model D came out 2.34 points ahead of the team's candidate (exact McNemar p=0.0078), reversing a 15-point deficit in generative scoring. The candidate's real gain, the authors write, was a habit of answering more briefly, which is useful under tight output limits but is not added knowledge or reasoning. The comparison between the candidate and model A, meanwhile, was statistically indistinguishable (p=0.5966).

The fix: deterministic scoring

The team now compares near-identical models deterministically, checking which option letter has the highest log-probability after the prompt, without any generated chain of thought. On 498 questions, repeated runs matched to six decimal places with zero changed predictions, and grading took 62 seconds versus 400 seconds for 400 generative questions. It is a companion tool, not a replacement: dropping chain-of-thought lowers scores sharply (0.437 versus 0.775 on the same model), so the two scales cannot be compared directly.

Why it matters

Leaderboard margins of one or two points are routinely read as capability differences. This report — self-measured, not re-scored by any external party, based on small samples, with the models anonymized as A through E — suggests such margins can instead reflect file tooling, server batching and token limits. The practical takeaways for anyone comparing similar models: report truncation rates alongside scores, estimate noise from rounding-different replicas rather than repeated runs of a single file, build baselines with the same pipeline as candidates, and remember that a few hundred paired questions cannot resolve one-point differences at all.