QuantMLA's INT4/INT2 KV-cache quantization reports 5x long-context throughput over BF16

Fivefold throughput on long sequences

Serving transformers with long contexts — sequences running to several thousand tokens or more — tends to hit a wall that makes real-time deployment difficult. According to a dev.to writeup of the QuantMLA paper ("QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching"), the technique's dual-path INT4/INT2 KV cache delivered more than five times the BF16 baseline's total output throughput when the cache was run under sustained pressure on long sequences.

The source is not fully consistent with itself: its headline claims attention throughput doubles, while its body reports the fivefold whole-job figure. The two numbers appear to describe different measurements, and the larger one is specific to a workload where the BF16 cache is under heavy memory pressure.

Where BF16 caches fall over

The writeup explains the mechanics of the bottleneck. BF16 KV caches grow linearly with sequence length, and under heavy load they frequently trigger memory preemptions, where the serving system evicts work to free space. When the cache outgrows its allotted footprint, the system falls back to recomputing work it has already done. That behavior, the post argues, is what has capped the practical context window of deployed models, independent of how fast the attention kernels themselves run.

What the low-bit cache changes

QuantMLA quantizes the combined content-and-RoPE cache along two paths, and at a 128k-token window it compresses that cache by 3.59x. The paper reports that the configuration it calls C4R4 fit the workload with no preemption occurring, meaning the system never had to evict and redo computation under pressure.

Crucially, the authors are explicit that the fivefold gain is not the product of faster attention math. It comes from sidestepping the recomputation that a BF16 cache forces under the same load. Anyone evaluating the approach should read it as a memory-capacity result rather than a kernel-performance result.

The accuracy caveat

Low-bit caching is only useful if it preserves what the model actually needs. The writeup notes that the method depends on keeping RoPE-induced positional signals intact, and that quality may degrade subtly on tasks which lean heavily on fine-grained positional geometry. The source does not quantify that degradation, so teams running position-sensitive workloads would need to test before adopting it.

Practical takeaways

Two conclusions in the post stand out for practitioners. First, long-range reasoning benchmarks should be rerun against INT4-cached MLA models, since existing latency figures may not reflect what a quantized-cache deployment would actually deliver. Second, production services can realistically consider context windows of 8k tokens and above without breaking real-time service-level agreements — precisely the regime where BF16 caches have historically begun preempting and recomputing.

Why it matters

Memory, not raw compute, is often the binding constraint in long-context inference. If a cache that is 3.59x smaller genuinely eliminates preemption and recomputation, operators can widen context windows on existing hardware instead of provisioning more of it. The caveats deserve equal weight: the headline throughput figure is tied to one pressure-heavy workload, the source's own headline and body disagree on the magnitude of the gain, and positional fidelity under INT4/INT2 remains an open question for geometry-sensitive tasks.