KV cache sizing: the arithmetic that decides if a local LLM fits in VRAM

The allocation that lands before the first token

Developers running local models on consumer GPUs often hit a familiar wall: the weights fit in VRAM, yet the model refuses to load, or generation crawls because the card is nearly out of memory. According to a guide published on dev.to, the culprit is almost never the model file. It is the KV cache — the per-token key and value tensors that every attention layer keeps so it does not have to recompute the past — which is reserved up front, scales linearly with context length, and on a 7B model at 32K tokens is measured in gigabytes.

One formula, and why GQA dominates it

For a standard full-attention transformer, the post gives the per-token cost as:

2 * n_layers * n_kv_heads * head_dim * bytes_per_element

The factor of two covers the K and V tensors. The term that surprises people, the author argues, is the number of KV heads, not the data type. Qwen2.5-7B-Instruct has 28 layers and 4 KV heads with a head dimension of 128, which works out to 28,672 cached values per token — 56 KiB at f16, or 1.75 GiB across a 32,768-token context before generation even starts. Mistral-7B-Instruct-v0.3 shares the head dimension but has 32 layers and 8 KV heads: 128 KiB per token and 4 GiB at 32K. Identical parameter counts, more than double the cache.

That is grouped-query attention in miniature. A 7B model with plain multi-head attention (32 KV heads) would need 512 KiB per token, or 16 GiB at 32K, which the post cites as the reason no one ships 7B MHA models for consumer GPUs. When comparing models, read the KV head count and layer count first; total parameters say almost nothing about context cost. Two caveats from the guide: some architectures set the head dimension explicitly in config.json rather than deriving it, and sliding-window or hybrid-attention models keep less than the full history, so the formula is an upper bound for them.

Quantized caches are not quite half

llama.cpp does not store one float per value in a quantized cache. It stores blocks of 32 values with a scale: q8_0 takes 34 bytes per block, q4_0 takes 18. That overhead is why the common advice that quantization halves the memory is close but wrong — q8_0 saves 47% of the cache, not 50%, and q4_0 saves 72%, not 75%. On a 4 GiB cache the gap between the claim and reality is roughly 120 MiB, enough to matter on an 8 GB card running a 14B model. At 32K, the Qwen example drops from 1.75 GiB (f16) to 952 MiB (q8_0) or 504 MiB (q4_0); Mistral drops from 4 GiB to 2.13 GiB or 1.13 GiB. These figures are pure cache — weights, the compute buffer and the CUDA context all sit on top.

The flags that actually control it

In llama.cpp, -c sets the context pool, and leaving it unset inherits the model's trained context — on a model trained for 128K tokens, an easy route to an out-of-memory crash. The -ctk and -ctv flags set the K and V cache types independently, and -fa on forces Flash Attention, which is required: quantizing the V cache without it fails at context creation with an explicit error, a behavior tracked in llama.cpp issue #10378. Because the cache is allocated at load time, the post recommends measuring VRAM with nvidia-smi before sending traffic rather than after.

Three quieter settings also shift the budget. --fit is on by default and silently shrinks context to fit memory, with a default floor of 4,096 tokens, so a request for 128K can quietly become 4K. --cache-ram, despite the name, is host RAM — 8,192 MiB by default — used to park prompt caches for idle slots. --no-kv-offload keeps the KV cache in system memory, trading latency for freed VRAM.

Ollama inherits the same physics with an extra trap: its default context is 4,096 tokens regardless of what the model was trained for, which is why a model can seem to lose track of later parts of a long document. The environment variable OLLAMA_CONTEXT_LENGTH raises it.

Why it matters

Context window is the budget item developers most often guess at, and guessing is what produces load-time OOMs and models that appear to forget long documents. The guide turns that guess into a short calculation: read config.json, apply the formula, pick a cache type, and verify the allocation before serving traffic. It also reframes model selection on 8–16 GB cards, where two 7B models with identical parameter counts can differ by more than 2x in context cost — often the difference between a usable 32K window and none at all.