Why temperature=0 LLM calls still drift: floating-point math and GPU batching

A dev.to post describes a familiar kind of flake: a CI test asserting on an LLM's output string failed roughly once a week despite a fixed prompt, a pinned model snapshot, temperature set to 0 and a seeded request. After ruling out code changes, the author looped the identical request 500 times and diffed the results. Twelve responses differed, and one classified a refund ticket as 'billing' rather than 'fraud' — a real behavioural change from a byte-identical input.

The explanation, according to the post, is that temperature=0 never guaranteed determinism in the first place.

Greedy sampling is deterministic; the arithmetic is not

Setting temperature to 0 collapses the sampler to argmax: always take the highest-scoring token. Given identical logits, that step is fully deterministic. The problem is that you never get identical logits from a shared inference server.

Floating-point addition is not associative, so (a + b) + c and a + (b + c) can differ in their last bits. On a GPU, reductions — summing across hidden dimensions, softmax denominators, normalisation layers — get split across blocks, and how they are split depends on the tensor shapes the kernel receives. A different shape means a different reduction order, different rounding, and slightly different logits.

The batch you cannot see

This is where batching comes in. The post invokes the concept of batch invariance: a single request producing the same output no matter what else is batched alongside it. Most production inference kernels lack this property, and the author frames it as a performance decision rather than a bug. A request landing in a batch of 48 at 3pm gets a different tiling and split-K strategy in the matmul than the same request in a batch of 3 at 4am, so the last-bit rounding of the logits differs.

Each individual run is deterministic given its batch — nothing is randomised — but you neither control nor observe the batch. The post lists further factors on real endpoints: mixture-of-experts routing with capacity limits, where expert assignment depends on competing tokens; prefix caching, which moves the compute boundary; speculative decoding, whose verification path is numerically distinct; and heterogeneous GPU fleets behind one endpoint. The seed parameter on major APIs, the author notes, is documented as best-effort because it pins the sampler's RNG, not the arithmetic.

Near-ties amplify a wobble

Most token positions are not close calls; when the gap between the top two candidates is large, a 1e-7 perturbation cannot flip it. But at near-ties — 'however' versus 'but', 'fraud' versus 'billing' on an ambiguous ticket — the gap is smaller than the numeric noise and argmax effectively coin-flips. Because decoding is autoregressive, one flipped token changes the input to every subsequent token, so a single-bit wobble can rewrite the second half of a paragraph. This is also why flakes cluster on the hardest, most ambiguous inputs, the post argues: those are exactly where near-ties live.

Fix the tests, not the math

On a hosted API, bitwise reproducibility is out of reach, so the post recommends making assertions robust instead:

• Parse the output and assert on the fields that matter rather than the surrounding prose.
• Pin the dated model snapshot, not the alias, since aliases roll silently and cause a separate, often-misdiagnosed drift.
• Shrink the output space: a classifier emitting a single enum token has far fewer near-tie positions than one that buries the label in a paragraph.
• Measure the flip rate by running the eval set multiple times and treating flapping items as low-confidence rather than pass/fail.
• Use K-of-N voting for decisions that matter, routing disagreements on something like a refund to a human instead of retrying.
• Log the response fingerprint so 'backend changed' can be distinguished from 'we hit a near-tie'.

True bitwise determinism, according to the author, requires running locally: batch size 1, fixed library and driver versions, deterministic kernel flags, and one GPU model — and even then it is reproducible only against that box. Batch-invariant kernels are being built into serving stacks but cost throughput, so assume you do not have them unless a vendor says so explicitly.

The post also warns against a tempting anti-pattern the author has watched a team deploy: a retry loop that reruns the model until it produces the expected string. That is not determinism, the author writes, but rejection sampling with extra steps and a larger bill. The honest framing is that an LLM call is a statistical dependency, not a pure function.

Why it matters

Teams increasingly put LLM-backed behaviour into CI, and the instinct is to treat a model call like an ordinary function returning identical output for identical input. As this post shows, that assumption fails for reasons rooted in floating-point arithmetic and GPU scheduling — not model 'creativity' — and the failure surfaces as weekly flakes on exactly the ambiguous inputs where correctness matters most. Understanding the real cause prevents wasted effort such as retry loops or superstition around temperature settings, and points to the durable fix: assert on parsed structure and invariants, measure nondeterminism explicitly, and build human escalation paths for genuinely uncertain cases.