Identical LLM cost benchmarks swung 27% on an untouched server, dev.to post shows

Four checks, four different answers

A developer writing on dev.to reports running the same cost measurement four times in a row against a local LLM server and getting materially different numbers each time, including a 27% jump between consecutive runs. The setup was deliberately plain: a MacBook running Ollama with the llama3.2:3b model, the same eight built-in prompts, temperature 0, a fixed cap of 64 output tokens per request, and a GPU rate of $1.50 per hour used purely as an assumption in the arithmetic. All four checks finished within roughly 70 seconds of each other.

Cost per million output tokens came out at $8.77, $8.86, $11.26 and $10.02 across the four runs. The third run was 27.1% more expensive than the second; the fourth was then 11% cheaper than the third. Nothing about the server, the model or the settings had changed in between.

Same tokens, different clock

According to the dev.to post, every request block in every run produced exactly the same token count, so the price movement came entirely from wall-clock time. The three blocks in the first run finished in roughly 5.2 to 5.5 seconds each; in the third run the equivalent blocks took 6.6 to 7.1 seconds. Around 30% more elapsed time for the same output translated into roughly 27% more dollars per token, because the cost model is just the assumed hourly rate multiplied by time and divided by tokens.

The post attributes the timing drift to the ordinary behaviour of a real machine: competing processes, thermal state and clock speed, memory pressure, OS scheduling, and on shared cloud nodes the activity of other tenants. The author did not instrument the laptop, so the specific cause is unknown — which is presented as the point. A single before/after pair cannot tell background load apart from a genuine change.

Why the error bars did not catch it

Each check reported a 95% confidence interval computed from the blocks inside that one run. The intervals for the first run ($8.19 to $9.35) and the third ($10.31 to $12.21) do not overlap, which under the usual rule of thumb would look like a real regression. The post argues this misreads what the interval measures: it describes variation over roughly 20 seconds inside a single run, and every block in a run shares the same machine conditions. If the whole system slows down for a minute, all three blocks slow down together and the interval stays tight around the wrong number.

Measuring the noise you are actually exposed to

The proposed fix is to measure run-to-run variance directly by repeating the unchanged configuration at least three times. The author treats the first three runs ($8.77, $8.86, $11.26) as that sample, which gives a relative standard deviation of 14.6% on 2 degrees of freedom. Combining that with the Student's t multiplier of 4.303 and a square-root-of-two factor for comparing two independent checks yields a noise bound of about 89% — far larger than either observed swing, so the honest verdict on the 27% jump is "no winner" rather than regression. Under the same logic, the run 2 versus run 3 comparison could not be judged at all, because there were not yet enough repeats to estimate noise.

Two levers tighten the bound: more repeats, since the t multiplier falls quickly with degrees of freedom (4.30 at 2 df, 2.57 at 5, 2.23 at 10), and a quieter, longer measurement taken at production concurrency. For decisions that will reach production, the post recommends interleaving baseline and candidate runs — baseline, candidate, baseline, candidate — so that drift hits both sides equally. The author's current routine: run the unchanged config three times in the same day, change one thing, and trust a delta only when it exceeds the calibrated noise bound and the confidence intervals do not overlap.

Caveats

The experiment ran on a laptop with a 3B model, short workloads and an assumed GPU rate, so the size of the swing is specific to that machine on that morning; the author is explicit that other servers will have their own noise that has to be measured rather than borrowed. The measurement tool, an open-source CLI called Throttle, repeated identical prompts across runs, which the post says rules out prompt-cache effects here since the cache was warm in all four runs; from version 0.4.1 the tool tags requests so a prefix cache cannot make later checks look artificially cheap.

Why it matters

Anyone budgeting or optimising LLM inference makes decisions from before/after numbers: a new serving flag, a quantised model, a different stack. This post shows that on ordinary hardware those comparisons can be swamped by measurement noise, and that conventional within-run confidence intervals invite false confidence — an overlap-free 27% regression appeared out of nowhere. Calibrating a run-to-run noise floor before trusting any delta is cheap, and skipping it risks rolling back good changes or announcing savings that do not exist.