deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

DepthGraft: hierarchical KV sharing cuts transformer memory by about a third

A dev.to write-up covers DepthGraft, which lets deep transformer layers reuse stride-aligned KV caches from earlier layers, cutting cache memory by roughly a third while benchmark scores improve.

DepthGraft: hierarchical KV sharing cuts transformer memory by about a third

A cache shared across depth

A write-up on dev.to spotlights DepthGraft, an architecture from a paper titled "DepthGraft: Structural Regularization through Hierarchical Cross-Layer KV Reconstruction" that changes where a transformer keeps its key-value cache. Rather than having every layer build and maintain its own KV entries during inference, later layers reuse the pairs computed by selected earlier layers whose positions align along a fixed stride. According to the dev.to article, this cuts KV cache size by about a third, a reduction large enough that deep models can fit comfortably on mid-range GPUs.

The sharing is hierarchical rather than flat. Deep-stage queries are reconstructed from stride-aligned sources sitting in the shallow-to-middle region of the network, which removes independent storage for much of the model's depth. Per the claims relayed in the article, the design also spreads gradients across those source layers during training and allows the prefill stage to finish early.

Why earlier sharing schemes fell short

Cross-layer connectivity is not new, the article notes, but previous attempts were treated mostly as architectural curiosities. Adjacent-layer sharing and global-anchor designs exist, yet they added matrix multiplications or extra memory hops without offering a systematic way to regularize the geometry of keys and values or to meaningfully shrink the cache. The practical outcome was that engineers came to treat heavy memory use as the unavoidable price of running deep models.

Reported numbers

The claimed compression does not trade away quality. On a 16-billion-parameter Mixture-of-Experts model, the article reports a 7.0 percent absolute improvement on MMLU-Pro, alongside gains of 6.6 percent on CMMLU and 6.4 percent on C3. Comparable improvements reportedly appear on dense models ranging from 0.5 billion to several billion parameters. The author reads this as evidence that hierarchical reuse improves how keys retrieve relevant content while giving values a richer representation, instead of degrading either.

What remains unproven

The article flags several open questions. Scaling past 16 billion parameters is untested, and the longest context evaluated was 128K tokens, so behavior on far longer sequences is unknown. The stride-aligned topology also assumes a regular transformer stack; models with mismatched or irregular layer depths may need custom engineering. Finally, although per-token overhead stays under one percent, latency-sensitive deployments are advised to profile the added reconstruction step.

Why it matters

The KV cache is one of the main reasons long-context inference demands expensive hardware. A method that removes roughly a third of it while improving benchmarks changes the arithmetic: within an existing memory budget, a service could run a larger model, hold a longer context, or serve more requests concurrently without buying new GPUs. The dev.to author goes as far as suggesting that benchmarks previously capped by memory, such as MMLU-Pro runs on a single A100, be rerun with DepthGraft enabled, and that shared caches could become the default choice in production. These are early claims drawn from a single write-up, so independent replication will matter, but if they hold, memory stops being the hard constraint on model depth and context length.

  • #transformers
  • #kv-cache
  • #inference
  • #memory-optimization
  • #llm

Related posts