Grouped Value Attention cuts transformer KV cache by ~45% with accuracy matching GQA

What was announced

Researchers have proposed Grouped Value Attention (GVA), an attention variant that reduces the memory transformers spend on their KV cache by roughly 45% while leaving benchmark accuracy essentially untouched. The result comes via a dev.to write-up published September 25, 2026, which summarizes two papers: one introducing GVA under the title "Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction", and one covering DeepSeek-V4.1-Flash's approach to cache compression.

The problem being solved

During autoregressive decoding, a transformer keeps a key and a value vector for every token it has processed. The cache therefore grows in step with sequence length and tends to become the dominant memory cost in long-context inference. Grouped-query attention (GQA), the widely adopted fix, trims the number of value heads but still stores a key for every token, so the underlying memory pressure remains.

How GVA works

GVA drops the key side of the cache almost entirely. It stores only grouped values and reconstructs each key on demand while decoding, so a complete key matrix never has to exist during generation. Removing that portion of the persistent state is what produces the near-halving of cache size.

The numbers

On a 350-million-parameter model, the write-up reports that GVA reduces persistent cache scalars by about 46% against a matched GQA baseline while holding average task accuracy within 0.01 points. The paper's authors describe the reduction as roughly 45–47%, and a 16-dimensional positional variant of GVA averages 44.35 accuracy across five tasks, versus 44.36 for GQA and 43.88 for MLA, the latent-attention scheme associated with DeepSeek-style models.

A second data point from DeepSeek

The same dev.to summary covers DeepSeek-V4.1-Flash, which pushes cache compression further. It reportedly shrinks the global HBM footprint to 890 bytes per token, about one quarter of its predecessor, and a deployment optimization the paper calls SWA Bounded Replay brings the persistent KV cache down to roughly one-eighth of DeepSeek-V4-Flash's. Despite these cuts, the newer model is claimed to outperform that baseline. The design also leans on FP4 caching and reuse across layers.

What remains unproven

Neither paper, as summarized, demonstrates end-to-end latency gains on commodity GPUs. GVA's custom decoding kernels are still being evaluated, so real-world throughput improvements remain open questions. DeepSeek's FP4 approach would benefit most from hardware with fast low-precision arithmetic, which adds a dependency on the deployment environment.

Why it matters

Memory for the KV cache is one of the largest line items in LLM serving: it caps batch size, limits context length and drives hardware cost. If GVA's roughly 45% reduction holds in production at unchanged quality, operators could serve the same workloads on far less memory, or push much longer contexts onto modest GPUs. The write-up also suggests re-running long-context benchmarks under tighter cache budgets, since methods like this could reset what counts as a feasible context length on consumer-grade hardware. The caveat is worth keeping in view: memory saved is not automatically latency saved, and until GVA's kernels ship and get benchmarked on real serving stacks, the practical payoff remains a promising hypothesis rather than a settled result.