Paper Proposes Direct KV-Cache Communication Between LLMs for Speed and Accuracy

A shortcut past text

Pipelines that combine several large language models almost always use text as the interface between them: one model writes out an answer, the next one reads it in. According to a paper on arXiv that recently reached the front page of Hacker News, that convention carries hidden costs. Flattening a model's internal understanding into a sequence of tokens strips away semantic detail, and producing those tokens one at a time adds delay. The paper, submitted by Tianyu Fu with a first version posted in October 2025 and a revision in March 2026, proposes letting models hand over their internal state directly instead.

What the KV-cache has to do with it

During inference, a transformer maintains a KV-cache: the stored key and value tensors computed for each token, which let the model attend back to earlier context without redoing that work. The authors first ran oracle experiments to check whether this cache could double as a communications channel. Their result: making a KV-cache carry richer semantics improved response quality without enlarging the cache. To the authors, that is evidence the cache is a viable medium for models to exchange meaning, not merely a speed optimization.

How Cache-to-Cache works

The proposed method, Cache-to-Cache (C2C), has three moving parts:

A small neural network maps the source model's KV-cache into the target model's representational space and merges it with the target model's own cache. Because not every layer profits equally from the injection, a learnable gating mechanism chooses which target layers actually receive the transferred cache. And because nothing is serialized into words, the target model can draw on the knowledge both models have developed without a written intermediate step.

The reported numbers

Measured against the individual models working alone, C2C delivers 6.4 to 14.2 percent higher average accuracy, the paper reports. Against pipelines where the models exchange answers as text, it gains roughly 3.1 to 5.4 percent. It also delivers an average 2.5x latency speedup, since no model has to spend time generating intermediate prose. The authors state that their code is publicly released, which should ease independent verification — worth noting, since all of these figures come from the paper's own evaluation rather than outside benchmarks.

Why it matters

Multi-model setups are already common in production: routing queries to specialist models, adding verification passes, and ensembling several systems are all standard practice. Nearly all of that coordination currently runs over text, which caps both how much information moves between models and how quickly the whole chain executes. If direct cache-level transfer holds up under wider evaluation, it offers a way to improve both dimensions at once — richer information exchange and shorter round trips.

The idea also reframes what communication between LLMs is for. Internal representations could become a first-class interchange format between systems, with human-readable text reserved for the humans at either end of the pipeline.

Open questions remain: how much training the projection and gating networks need for each pair of models, whether the gains persist across tasks and model scales, and how much architectural mismatch between source and target the method can absorb. The released code gives researchers a concrete starting point for testing those limits.