Next-Concept Modeling generates text as coarse-to-fine codes, up to 31x faster than diffusion

What the paper proposes

A paper from Stanford's Hazy Research group and Together AI, posted to arXiv on October 9, 2026, proposes a third route for text generation, sitting between autoregressive decoding and diffusion. According to a dev.to writeup of the work, the method — Next-Concept Modeling (NCM) — does not predict tokens at all. It generates a hierarchy of discrete codes ordered from coarse to fine, and at matched research scale of roughly 100 million parameters it reportedly matches or beats strong diffusion baselines while running up to 31 times faster.

How Next-Concept Modeling works

Autoregressive models emit one token per step, so a long answer requires hundreds of sequential computations that GPU parallelism cannot compress. Diffusion models refine every position at once, but each denoising pass reprocesses the whole sequence, and quality drops quickly when passes are cut.

NCM avoids both patterns with a two-stage design. Stage one trains a multiscale residual VQ-VAE that compresses text into a stack of code levels: the first is a short sequence capturing global structure, and each later level quantizes the error left behind by the levels before it, with a decoder mapping the full stack back to text. Stage two trains a transformer to produce these levels in order, predicting every code within a level in parallel, conditioned on the cumulative reconstruction of the coarser levels.

The speed advantage comes from two multipliers. Diffusion needs many passes over the full sequence; NCM needs only as many sequential steps as there are levels — a handful — and its earliest passes run on short, coarse sequences rather than the full text. The authors flag one text-specific difficulty: images downsample cleanly while sentences do not, and reconstruction must be nearly exact because a single wrong token can flip meaning or break a puzzle solution.

What the benchmarks show

All results are at ~100M parameters, against discrete-diffusion, flow-based and hybrid baselines rather than frontier models. Within that frame, the dev.to post reports the following. On LM1B, NCM achieves better generative perplexity than the strongest diffusion baseline at 21x faster wall-clock. On OpenWebText it is competitive on quality at 10x the speed. On XSum summarization it matches the baseline's ROUGE-L at 31x the speed, and exceeds that baseline's ROUGE-L by roughly 7% when the saved time is spent sampling more candidates and applying Minimum Bayes Risk selection. Training reaches competitive quality with 4-5x fewer tokens than strong diffusion baselines. Function evaluations fall 4-11x on unconditional generation and 6-16x on conditional generation, with diversity measured by unigram entropy staying close to the training data — diversity collapse is the problem that typically dogs few-step diffusion, and it does not appear here.

The Sudoku result is the most striking. On hard boards NCM solves 86.1% of puzzles using 25M parameters and 12 function evaluations, versus 81.3% for the best diffusion baseline at 29M parameters and 16 evaluations. The margin widens with difficulty — 1.8 points on easy, 3.7 on medium, 4.8 on hard — which the writeup reads as evidence that the coarse levels genuinely plan ahead on constraint-heavy problems.

The caveats

The post is candid about limits. Everything is tested at ~100M parameters on LM1B, OpenWebText, XSum and Sudoku; scaling evidence is preliminary, with no data on whether the hierarchy holds up with long sequences and large vocabularies at billion-parameter scale. The tokenizer is a second model the whole scheme depends on: if coarse levels encode ambiguity that fine levels cannot resolve, the generator cannot compensate for it no matter how strong the transformer. And the headline speedups are measured against unguided, undistilled diffusion baselines, while few-step samplers, distillation and block-autoregressive diffusion are independently chipping away at the same latency issue. What matters for production is how NCM fares against distilled diffusion, and that comparison has not been made.

The idea has clear lineage: next-scale prediction already powers VAR-style image models, and concept-level text modeling appears in MEGABYTE's byte patches, H-Net's learned chunking and Large Concept Models. NCM's contribution, per the writeup, is making the hierarchy discrete, residual and generative for text, with a decoder precise enough that the approach holds up on exact-match tasks such as Sudoku.

Why it matters

Sequential decoding is a structural bottleneck in inference cost — no amount of hardware parallelism shortens the token-by-token chain. NCM reframes the problem as a choice of generation order: plan first, details second, in a few parallel steps. If the speedups and the 4-5x training-efficiency claim survive scaling, they translate directly into lower serving costs. At ~100M parameters this is a research signal rather than a shipping technique, but it suggests the next big latency win in language models may come from changing how text is generated rather than from faster silicon.