From 64 memories to a million: essay derives linear layers, MLPs and attention as memory stores

How the essay frames the problem

A long-form technical essay by Alex Wang, "You could have invented transformers from memory," reached the front page of Hacker News on October 15, according to the site's RSS listing. Its motivating questions are simple: how do large language models remember so much, and how much can they hold?

Wang's approach is to treat the standard building blocks of modern networks — linear maps, MLPs, and self-attention — as competing designs for one object: an associative memory, a store that holds key-value pairs and hands back a stored value when queried with that key or something close to it. The organizing metric is what the post calls the "interference-to-signal ratio": how much noise the other stored memories inject when you try to retrieve one. A store's capacity is defined as the number of memories it can hold before that noise matches the strength of the desired recall.

A linear layer as an outer-product store

The baseline design simply accumulates memories, each pair contributing a rank-one update of value-times-key-transpose to a weight matrix. Querying with an input returns the sum of all stored values, each weighted by how closely its key resembles the query — retrieval purely by similarity, with no explicit index.

Wang notes a corollary: one gradient-descent step on an ordinary linear layer takes exactly this form, so training is literally writing memories, with the layer's input serving as the key. He cites work by Irie, Csordás and Schmidhuber that formalizes this view, casting a gradient-trained linear layer as attention over its past training inputs.

Interference sets the capacity

Recall fails not because storage fails, but because memories collide. Querying with one stored key returns the correct value plus a contribution from every other memory whose key points in a similar direction. Orthogonal keys would give perfect recall, but a key space of dimension d_k contains only d_k mutually orthogonal directions.

For random unit-length keys, each spurious similarity runs about ±1 over the square root of d_k, and because the cross-talk terms combine independently, their total grows with the square root of their count. The upshot, per the essay, is that noise relative to signal scales as the square root of M over d_k, giving the plain linear store a capacity of roughly d_k memories. With 64-dimensional keys, that is about 64 facts before the clutter drowns out the target. Interactive charts in the post let readers watch this breakdown memory by memory.

Two routes to a better store

Wang identifies two levers for improvement.

The first is to store more carefully. Rather than adding each memory in isolation, fit all of them jointly through least squares. The resulting solution equals the outer-product store with the keys' covariance inverted in front — undoing shared directions so heavily populated parts of the key space are not overweighted. This yields exact recall for up to d_k linearly independent keys and roughly doubles capacity, to about 2d_k memories.

The second lever is to recall more selectively. In a linear store, every memory contributes noise regardless of relevance, so using nonlinearities to suppress low-similarity keys cleans up retrieval. According to the post, this is the thread that leads to MLPs and ultimately to self-attention, which keeps memories in activations rather than weights and can therefore hold an unbounded number of them. With 64-dimensional keys, the essay's arithmetic runs from roughly 64 memories for the linear store, to about 2,700 for an MLP, to more than a million for attention.

Why it matters

For practitioners, the essay supplies what its author promises: heuristics for how much each layer type can memorize, derived from a single quantity. That converts architecture choices into storage estimates — and helps explain why parameter-bounded MLP blocks bottleneck memorization while attention over long contexts does not. It also ties training dynamics to inference-time retrieval: gradient descent writes outer-product memories, and attention performs the same lookup with far less interference. The pedagogical payoff is a derivation path that reaches attention from regression first principles, making a familiar mechanism feel less like an arbitrary invention and more like the natural endpoint of a memory design problem.