Continuous diffusion language models make a comeback after near-extinction in 2023

A burst of new research into continuous diffusion language models is underway, ending a period of near-total dormancy that began in late 2023. That is the central claim of a retrospective by Sander Dieleman, a researcher who has worked on image and video generation systems including Imagen, Veo and Nano Banana, whose post reached the Hacker News front page and argues that the field's pivot away from continuous methods may have been a mistake worth revisiting.

Autoregression's long shadow

Nearly every modern language model is autoregressive: it produces text one token at a time, each step conditioned on everything that came before. As Dieleman explains, this decomposes a difficult generation task into many small, identical prediction steps, which enables parameter sharing across the sequence and — through teacher forcing — lets Transformers train in parallel across all positions. That combination proved exceptionally scalable and produced the LLM era.

Diffusion offers a different kind of iterative process. Instead of predicting the next element, a model learns to reverse a corruption procedure, canonically by gradually adding Gaussian noise until the original signal is drowned out. The technique came to dominate image, audio and video generation, which is what first led researchers to try it on text.

From discrete to continuous and back

The earliest language diffusion attempts arrived in 2021 and were discrete in character: multinomial diffusion, D3PM and SUNDAE swapped Gaussian corruption for operations suited to categorical data. According to Dieleman, the timing mattered. This was after GPT-3 had turned heads but before the ChatGPT breakthrough of late 2022, and discrete diffusion seemed to fix genuine theoretical weaknesses of autoregression, including exposure bias introduced by teacher forcing and the awkwardness of infilling and constrained generation.

In 2022 a second wave made the process continuous again. Diffusion-LM represented discrete tokens as continuous embedding vectors that Gaussian noise can corrupt directly, and a cluster of variations followed that year: DiffuSeq, SSD-LM, Difformer, SeqDiffuSeq, GENIE and LD4LG, plus two papers Dieleman co-authored, SED and CDCD. The attraction, he writes, was leverage: continuous methods could borrow the sampling and distillation techniques accumulating around continuous diffusion as it took over audiovisual generation, machinery that was often difficult or impossible to apply to discrete formulations.

The late-2023 die-off

Then continuous language diffusion essentially vanished. Dieleman cites a diagram from a 2025 survey of diffusion language models that makes the break unmistakable: from 2024 onward, almost all new work used discrete methods.

He offers two explanations, both explicitly speculative. One is that the ChatGPT moment redirected the field from theoretical appeal toward raw performance, and researchers apparently judged that discrete diffusion, being conceptually closer to autoregression, offered the better path to matching autoregressive models at scale. The other is that early scaling results were brutal for continuous approaches: in May 2023, Gulrajani and Hashimoto measured Plaid-1B, a likelihood-based continuous diffusion language model, at 64x lower training efficiency than an autoregressive baseline. In a research culture still optimising training compute against perplexity in the Chinchilla tradition, Dieleman argues, a gap of nearly two orders of magnitude was disqualifying — even though LLaMA 1, released in February 2023, had only just begun making the case that inference budgets deserve weight too.

Dieleman himself left the area around that point to build generative image and video systems, and writes that he watched the die-off from the sidelines with some surprise.

Why continuous is drawing fresh attention

The recent wave of activity suggests momentum is swinging back toward continuous methods. Dieleman contends they hold advantages the field surrendered perhaps too readily: the capacity to express uncertainty at the level of individual tokens, and access to the deep bench of sampling algorithms and tricks developed for continuous diffusion in other domains.

Why it matters

The commitments baked into autoregression shape both what today's LLMs can do and what they cost: exposure bias, clumsy handling of infilling and constrained generation, and a decoding process that is sequential by nature. If renewed work on continuous diffusion can close the efficiency gap that sank it in 2023, language models could inherit the refinements that made image diffusion fast and steerable, with practical consequences for inference speed, editing and controllability in future LLMs. Dieleman is candid that his account is subjective and invites dissent, but the arc he documents — rapid adoption, sudden abandonment, and now revival — is also a useful reminder of how sensitive research direction is to whichever efficiency metric happens to be in fashion.