DeepSeek's MLA compresses KV cache by 93% with latent attention

A technical deep-dive published on dev.to reconstructs how Multi-head Latent Attention (MLA), the attention mechanism used in DeepSeek-V2 and DeepSeek-V3, shrinks the key-value cache to 576 scalars per token — roughly a 93% reduction against a standard multi-head attention baseline — and walks through the math that makes it work during inference.

Decoding is a memory problem

According to the article, transformer inference splits into two regimes. Prefill processes the whole prompt at once and keeps GPU compute busy. Decode, which emits tokens one at a time, must re-read the cached keys and values of every preceding token for each new token, and its arithmetic intensity falls to roughly one FLOP per byte streamed. Decode is therefore limited by HBM bandwidth, not compute.

The KV cache grows with layer count, KV head count, head dimension, precision, batch size and sequence length. The article's per-token figures: a DeepSeek 67B baseline with standard multi-head attention would need about 3.84 MB per token at FP16, Llama 3 70B's 8:1 grouped-query attention about 320 KB, and DeepSeek's MLA about 135 KB — or 67.5 KB at FP8. Stretched to a 128,000-token context for a single stream, that becomes roughly 503 GB for the MHA baseline, 40.96 GB for the GQA model, and 17.28 GB for MLA (8.64 GB at FP8).

To make the cost concrete, the author models Llama 3 70B at batch size 8 with 128k contexts on an 80 GB H100 with 3.35 TB/s of HBM3 bandwidth. Streaming 327.68 GB of cache per generated token takes about 97.8 ms in transfer time alone, capping generation near 10.2 tokens per second while Tensor Cores sit mostly idle. Multi-query attention, the more aggressive fix of sharing one KV head across all query heads, reportedly costs 3.8% to 6.2% in scores on GSM8k and multi-document recall.

A latent vector instead of full keys and values

MLA projects each token's 5,120-dimensional hidden state down to a 512-dimensional compressed latent vector. Keys and values for all 128 heads are reconstructed from that latent via per-head up-projection matrices. Since the full content would otherwise occupy 32,768 scalars per token (128 heads at 128 dimensions each, for both keys and values), the latent represents a 64× compression of the content. Because every head uses its own slice of the up-projection, heads can still express independent attention patterns, avoiding the representational collapse that hurts MQA.

Splitting position from content

Rotary position embeddings create a snag. The article explains that the RoPE rotation matrix does not commute with the up-projection, so a runtime that cached only the latent would have to reconstruct and rotate every historical key at each decoding step — wiping out the bandwidth savings. DeepSeek's answer is decoupled RoPE: each key and query is split into a 128-dimension content part derived from the latent, plus a 64-dimension positional part shared across all heads. The attention score then becomes the sum of a content term and a positional term.

Absorption removes decompression entirely

Because the content keys are purely linear in the latent, the up-projection can be folded into the query vector once per decoding step, letting the attention dot product run directly against the cached 512-dimension latents. The value up-projection is similarly fused into the output projection. What remains in memory per token is the 512-dimension latent plus the 64-dimension shared RoPE key: 576 scalars, versus 8,192 for a 32-head MHA baseline, which is the source of the roughly 93% figure.

Why it matters

Decode speed and serving cost at long contexts are set almost entirely by how many bytes the KV cache moves per token. On the article's numbers, MLA at 128k needs 17.28 GB per stream instead of 40.96 GB for a GQA model — a difference that converts directly into larger batches, longer contexts or more concurrent users on the same hardware.

Caveats apply. This is a single explanatory article, and several of its comparisons are hypothetical baselines: Llama 2 70B is modeled as MHA only hypothetically, and the 93% figure is measured against a 32-head MHA cache rather than the GQA models most providers actually deploy, where MLA's relative saving is smaller but still large. The mechanism itself, however, is established practice — MLA shipped in DeepSeek-V2 and V3, and the combination of latent compression and matrix absorption is a big part of why their long-context serving economics differ from GQA-based peers.