LoopVQ stores looped transformer KV caches below one bit, roughly doubling throughput

Sub-bit compression for a cache that grows with every loop

LoopVQ is a quantization method that stores the key-value cache of looped transformer models at less than one bit per element, and according to a dev.to write-up of the research, it roughly doubles serving throughput on unchanged hardware.

The target is a specific architectural pain point. Looped transformers run the same layer repeatedly rather than stacking many distinct ones, and every pass through the loop writes its own keys and values. The cache therefore grows with the number of loop steps and ends up dominating memory use at serving time. According to dev.to, earlier vector-quantization schemes for KV caches never addressed the redundancy that appears along the loop axis, which meant most of the possible latency reduction went unclaimed.

How the method works

LoopVQ encodes multiple loop steps jointly instead of quantizing each one independently, then fine-tunes the quantization codebooks after the fact. Together, those steps squeeze each cache element below one bit while preserving the model's functional behavior. Two deployment details stand out: the implementation plugs into SGLang's serving stack, and it requires no retraining of the underlying transformer.

What the measurements show

At 0.5 bits per element, decode latency for long contexts falls to roughly half the BF16 figure at a 32K-token context, dev.to reports. The benefit grows with context length because cache reads dominate the decode pipeline.

Peak throughput improves even more than latency: up to 4.3× on the 1.4B Ouro model and 6.1× on the 2.6B Ouro model versus BF16 baselines. For typical batch sizes, dev.to translates that into roughly a two-fold increase in requests per second on identical hardware. The margin widens as generated sequences lengthen and full-precision batches shrink.

Quality holds up at 0.5 bits, with negligible loss on downstream reasoning and retrieval tasks and results that match full precision on chain-of-thought benchmarks.

The caveats

The evaluation covers the 1.4B and 2.6B Ouro models plus Huginn models, all of them looped architectures. The method leans on a codebook fine-tuning step applied after training, and dev.to cautions that porting it to other architectures may require architecture-specific calibration, so broader adoption will probably hinge on tooling that automates that work. Pushing down to 0.25 bits preserves most of the BF16 accuracy only when combined with TaSQ, a sign that ultra-low precision still carries a quality trade-off.

Why it matters

The KV cache is one of the biggest levers on inference economics: it caps how many requests fit in memory per GPU, which in turn caps throughput and cost per token. A cache-level change that doubles request capacity without touching model weights, retraining, or hardware is a direct cost cut for long-context serving, precisely the workload where cache pressure is worst.

There is also a benchmarking implication. If serving stacks keep reporting decode speed against raw BF16 latency, they understate what is achievable, and the write-up argues the field should treat a sub-bit KV cache configuration as the standard baseline for long-context decoding numbers.