Memorilla compresses whole document collections into 16 memory tokens for frozen LLMs

What Memorilla is

A paper titled "Memorilla: Latent Semantic Memory for LLMs," which surfaced on the front page of Hacker News, proposes a memory system that lets a frozen large language model draw on an entire document collection without filling its context window. Instead of pasting retrieved passages or full histories into the prompt, Memorilla compresses the relevant knowledge into a small, fixed number of learned "memory tokens" that are placed directly in the model's input embedding space.

The problem it targets is familiar to anyone building agents or long-context applications. A model whose weights are not being updated can only use the information a memory system puts in front of it, and gathering the right fragments for each query while keeping the context short remains difficult for existing approaches.

How it works

According to the paper's abstract, Memorilla encodes source documents — user chats, book chapters — once into reusable embeddings. At query time, the system retrieves and combines query-specific information from those embeddings and writes it into the LLM's input embedding space as a fixed set of memory tokens; the reported experiments use just 16.

Because documents are embedded once and reused, the design avoids re-processing the corpus for every query. And because the memory lives in the embedding space rather than the token stream, the context length stays constant no matter how much source material has accumulated.

Benchmark results

Using its 16 tokens, Memorilla posts the best performance against baselines including Mem0 and MemoRAG on five of eight benchmarks spanning personalization, long-document question answering and knowledge-intensive question answering, according to the paper.

The standout result is on FactKG, a knowledge-intensive QA benchmark: Memorilla reaches 84.33% accuracy versus 80.43% for full-context inference, where all source documents are simply included in the prompt. It does so while representing the source documents with 108x fewer tokens. The flip side is that on the remaining three benchmarks, the baselines stayed ahead.

Agents and continual learning

The authors also frame Memorilla as a step toward continual learning in agentic systems, where the "documents" are an agent's past turns and their number grows over a trajectory. In a TextWorld environment, giving the agent Memorilla-based memory reduced its tendency to wander between rooms it had already explored, and improved its average success rate from 18.6% with full-history in-context learning to 24.2%.

Why it matters

Context length is one of the main cost and latency levers in modern LLM deployment. A system that matches or exceeds full-context accuracy with two orders of magnitude fewer tokens changes the economics of long-document and personalization workloads, where the default today is often to retrieve a large chunk of text and pay to process it again on every query.

The FactKG result also suggests that learned compression is not merely a cheaper approximation of reading everything: in at least one setting it beat reading everything, plausibly because the memory tokens filter out noise that a raw prompt carries. For agent builders, the TextWorld experiment is a small but pointed signal that raw histories can actively hurt planning, and that a compact, learned memory can help an agent stop repeating its own past mistakes.

The caveats are the usual ones for a research paper: benchmark wins on five of eight tasks, a single interactive environment, and results reported by the system's own authors. The arXiv listing shows the paper was submitted by Vignesh Kothapalli in mid-October, with revisions through early November. Still, the core direction — moving memory out of the token stream and into the embedding space — is a notable data point in the wider effort to make frozen models accumulate and reuse knowledge over long interactions.