arXiv paper proposes attention-free Grassmann models within 15 percent of small Transformers

A challenge to attention's necessity

A paper posted to arXiv on December 22, 2025 by Chong Zhang under the title "Attention Is Not What You Need," and recently surfaced on Hacker News's front page, asks a blunt question: is explicit self-attention actually required for strong sequence modeling performance and reasoning? The authors' answer is no, or at least not necessarily.

Their argument starts by reframing what attention does. Standard multi-head attention, the paper says, is best understood as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise token interactions, and learning proceeds by constraining that lifted tensor through gradient descent. The mechanism is extremely expressive but mathematically opaque, because after many layers the model can no longer be described by a small family of explicit invariants.

What replaces the attention matrix

The proposed alternative is an attention-free architecture built on what the authors call Grassmann flows. Its core component, the Causal Grassmann layer, never forms the L by L attention matrix at all. According to the abstract, it does three things in sequence: it linearly reduces token states, encodes local pairs of tokens as two-dimensional subspaces on a Grassmann manifold via Plücker coordinates, and fuses those geometric features back into the hidden states through gated mixing.

On a Grassmann manifold, the mathematical space of linear subspaces of a fixed dimension, a token pair is represented not by a similarity score, as attention would compute, but by the orientation of the subspace it spans. Information then propagates through the network as controlled deformations of low-rank subspaces across multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space. The authors report that Grassmann mixing scales linearly with sequence length for fixed rank, a property that has also motivated other attention alternatives aimed at long contexts.

The evidence so far

Two benchmark results carry the argument.

On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. That is close, but still a deficit: the paper demonstrates competitiveness at small scale, not superiority.

On the SNLI natural language inference task, the outcome flips slightly. A Grassmann-Plücker classification head mounted on DistilBERT edges out an equivalent Transformer head, with best validation accuracy of 0.8550 versus 0.8545 and best test accuracy of 0.8538 versus 0.8511.

Both experiments are small by modern standards, and the paper makes no claim about production-scale language models. The contribution is closer to an existence proof: a geometric, attention-free design that lands in the same ballpark as Transformers on modest tasks.

Why it matters

Nearly every large language model deployed today is built on self-attention, and the cost of that mechanism shapes everything from context-window limits to training budgets. A credible attention-free design with linear scaling in sequence length and a cleaner mathematical structure matters for two reasons. It could reduce the compute cost of long-context modeling, and, as the authors argue, manifold-based designs may be more amenable to geometric and invariant-based interpretation than attention stacks whose behavior resists compact description.

This paper will not dethrone attention on its own. Its language-modeling results still trail Transformers, its models are tiny by current standards, and its datasets are small. But it joins a broader reassessment of what is essential in sequence modeling, alongside state-space models and linear-attention variants. The decisive test, whether Grassmann flows train stably and perform at billions of parameters, is not addressed here, and it is the question that will determine whether this line of work influences future LLM architectures or remains a theoretical curiosity.