Zero-parameter document cache overtakes a 1.43M-parameter transformer beyond 100-token documents

The setup

According to a write-up on dev.to, an experiment has pitted a small transformer against a counting mechanism with no parameters at all, and found that once documents grow past roughly 100 tokens, the free mechanism wins on top-1 next-token accuracy by an increasingly large margin.

The author compared six systems making predictions at identical positions in a continuous text cut into documents of length L, so that document length and coherence grow together. On one side sits a 1.43M-parameter transformer restricted to a fixed 64-token context window. On the other sits a count table built over the current document alone: zero parameters, no training, constructed in microseconds. A Witten-Bell-smoothed corpus model trained on 4M tokens serves as a baseline, and hybrids combine the components. The research repo is not public, so all figures come from the author's own harness.

The evaluation is deliberately out of distribution: a Gutenberg book tested against components trained on TinyStories. The author notes both trained components face the same shift, which keeps the comparison internally fair, and cites an earlier finding that the cache pays off most out of distribution anyway.

The crossover

At L=60 the transformer wins decisively: 0.152 top-1 against 0.103 for the cache, a difference the paired tests report as significant. But the transformer is flat across lengths — 0.152, 0.143 and 0.149 at L=60, 250 and 1000 — which the author attributes to structure rather than statistics, since a 64-token window literally cannot see a 1000-token document. The cache, meanwhile, goes 0.103, 0.172, 0.213, more than doubling.

Somewhere between 60 and 250 tokens, a count table over the current document overtakes 1.43 million trained parameters. At L=1000 it wins by 0.064 top-1, a 43% relative margin in the author's arithmetic. The paired tests flag the cache's wins at both L=250 and L=1000 as significant. The two systems also disagree on 20 to 25% of positions, against roughly 3% between two hybrids in an earlier experiment — evidence, the author argues, that these are genuinely different mechanisms rather than two routes to the same predictions.

The parameters stop paying

The ablation is the starkest part. Adding the 1.43M-parameter transformer on top of corpus plus cache buys +0.022 top-1 at L=60, +0.013 at L=250 and just +0.002 at L=1000, at which point the substitution is close to complete. The full stack of transformer, corpus model and cache is the best system at every length, reaching 0.230 at L=1000, but almost all of that comes from the two count tables.

The mixture weights arrive at the same conclusion without being told to: the cache's optimal share climbs from 0.25 to 0.70 to 0.85 as documents lengthen, so at L=1000 the best system is 85% document and 15% corpus.

What it does not show

The author is explicit about the limits. The experiment says nothing about models whose context window covers the document — a long-context model would read the text itself, and this setup cannot see that comparison at all. Counting also still fails to scale with corpus data; the axis that scales here is the amount of current document available, which is a different resource.

The L=1000 cell rests on only 74 documents and is the noisiest of the three. The out-of-distribution design also cuts both ways: an in-domain transformer would be stronger and the crossover would move right, though the project lacks a long-document in-domain corpus to measure that, which the author calls the clearest gap in this line of work.

Why it matters

The honest summary is not that counting beats neural networks — below about 100 tokens, the transformer wins by a significant margin. It is that two mechanisms can look like competitors while depending on entirely different resources: one spends parameters, the other spends document length. Measured at a single operating point, such a comparison tells you almost nothing; the author's own headline flips from a transformer win at L=60 to a cache win at L=1000 on the same day, with the same harness, and either result could have been published alone.

For anyone building or benchmarking small language models with fixed windows, it is a concrete demonstration that cheap, document-local statistics can absorb most of what a trained model appears to contribute as inputs grow. The broader lesson the author draws: before comparing two systems, identify the axis one can spend and the other cannot, then measure along it.