Reported Ollama prefill rate of 13,826 tok/s was really 43 due to cache math

A writeup on dev.to documents a reproducible flaw in how local model throughput gets measured through Ollama: the same prompt, sent twice to the same daemon about thirty seconds apart, produced a reported prefill speed of 2,272 tokens per second on the first request and 13,826 on the second — when the honest rate for that second request was roughly 43 tokens per second.

According to the post, which used Ollama 0.34.0 running qwen2.5:7b, the gap comes from two API fields whose units no longer line up. prompt_eval_count reports the full prompt length — 318 tokens in the test — while prompt_eval_duration times only the tokens the daemon actually computed. On the warm request, 317 of those 318 tokens were served from the KV cache, so 23 milliseconds of compute ended up divided into a whole prompt's worth of tokens.

The inflation scales with your cache

The error is not a fixed overhead. As the author works out, the inflation factor is exactly the ratio of total prompt tokens to uncached tokens, which is the cache hit ratio. The better the cache performs, the more the number lies: a cold prompt reports truthfully, while a fully warm one is off by roughly the length of the system prompt. Any dashboard plotting prefill tokens per second across a conversation is effectively charting its own cache hit rate and labelling it throughput.

The correction is a single subtraction: subtract the cached count from the total before dividing by the elapsed time. The post notes that Ollama's internal Metrics.Summary() already computes the rate this way, so the daemon is not contradicting itself — it simply exposes two fields that invite a wrong combination, and the obvious one is what most existing code already used.

Two edge cases are flagged. Daemons older than 0.33.3 omit prompt_eval_cached_count entirely, and defaulting a missing field to zero silently converts "unknown" into a confident claim that nothing was cached. And when every token is cached, there is no prefill rate at all — the author argues for reporting nothing rather than zero or infinity.

Three endpoints, three names

The cached-token figure surfaces under a different name on each API. The native /api/chat endpoint calls it prompt_eval_cached_count, the OpenAI-compatible endpoint reports it as usage.prompt_tokens_details.cached_tokens, and the Anthropic-compatible endpoint exposes usage.cache_read_input_tokens. All three returned 317 for the same warm prompt in the author's tests.

The Anthropic-compatible surface carries an extra trap: it returned input_tokens: 1 for a 318-token prompt, because that field there means total minus cache reads. Anyone summing it across turns to estimate load will find warm turns contribute almost nothing unless they add the two fields together.

Prompt layout becomes measurable

With cache counts readable, the author tested prompt layout using a 227-token body containing one changing timestamp. With the timestamp near the front, only 40 tokens were reused and prefill took 41.2 ms; with it at the back, 211 tokens were reused and prefill took 16.3 ms — a 2.5x difference caused purely by placement. The 40 tokens still reused in the front case came from the chat template's stable preamble, not the user's content.

The practical rule is unglamorous: put stable text first and volatile values — dates, session IDs, per-turn retrieved chunks — last.

The author also admits an initial methodological error: priming the cache with one prompt layout and measuring a different one produced an inverted result. The sound approach is to send the same layout twice, mutate only the volatile value between sends, and measure the second request, since real traffic repeats a layout rather than alternating between two.

Why it matters

Tokens-per-second is the headline metric for local inference, and it feeds benchmark comparisons, capacity planning and dashboards. If a harness divides the total prompt token count by a duration that excludes cached work, every warm-cache measurement is inflated by the cache hit ratio — potentially by orders of magnitude — and comparisons between models or machines become meaningless. The numbers in the post come from one machine, so magnitudes will vary, but the direction of the error does not. The author, who maintains the Apache-2.0 local observability tool LLMxRay, urges anyone reporting prefill speed to spend thirty seconds checking which subtraction their code performs.