After transformers: state space models, diffusion text and world models edge into production

The post-transformer field, mapped

A dev.to article published in September 2026 surveys the main candidates to succeed the transformer and argues that the next architectural shift is already forming — in research labs rather than press releases. The author frames AI progress as a roughly three-year cycle: transformers displaced RNNs and CNNs around 2017, diffusion models took over image generation between 2020 and 2022, and LLMs went mainstream in 2022. On that rhythm, hybrid post-transformer systems belong to the 2026–2028 window, and the article presents evidence that the transition has begun.

Four limits scaling cannot fix

According to the piece, four problems are closing in on transformers that additional compute will not solve:

• Self-attention compares every token with every other token, so cost grows with the square of context length. Doubling context quadruples compute, and the author contends the arithmetic breaks down outright at around ten million tokens.
• Training data is running out. Internet text has largely been consumed, and the article warns that synthetic data degrades over model generations as learners absorb the artifacts of the generation process, leaving the scaling laws of 2020–2024 with diminishing returns.
• Energy and cost form a hard ceiling. Frontier training runs cost tens to hundreds of millions of dollars, serving at scale demands gigawatts of power, and the piece cites Sam Altman acknowledging in 2024 that compute costs constrain everything.
• Token prediction is pattern matching, not causal understanding. Systems that must plan, simulate or adapt to new environments cannot get there by predicting the next word, the author argues.

State space models: linear scaling, compressed memory

State space models such as Mamba and Gated DeltaNet replace attention over the full history with a fixed-size internal state that summarizes it, giving linear rather than quadratic scaling and constant per-token memory at inference. The article traces the lineage: early S4-family models used fixed transition rules, which are fast but cannot weigh one token over another. Mamba made those parameters input-dependent so the model chooses what to keep, and runs the recurrence as a hardware-aware parallel scan inside GPU memory — kernel work the author compares to FlashAttention. Mamba-2 then introduced state space duality, showing that a selective SSM and a restricted form of linear attention compute the same underlying operation, letting SSM kernels reuse matrix-multiplication hardware.

The production milestone, per the piece, is Gated DeltaNet, a delta-rule refinement of Mamba-2 with better long-context handling. The article reports that Qwen3.5 adopted it as its core architecture in February 2026 and posted strong agent and coding benchmark results with only three billion active parameters in a hybrid mixture-of-experts configuration, and that NVIDIA shipped Nemotron 3 with Mamba-2 layers built in. The tradeoff is structural: compressing history into a fixed state can lose fine-grained detail that attention preserves, so the piece concedes transformers still win for exact copying or citation, while state space models suit summarization, tool use and long-document work where the gist matters more than verbatim recall.

Diffusion comes for text

Diffusion language models borrow the technique behind image generators: instead of emitting one token at a time, they refine a whole block of text in parallel, producing many tokens per forward pass. According to the article, NVIDIA's Nemotron-Labs-Diffusion, released in May 2026, reached nearly six times the tokens per forward pass of Qwen3-8B at competitive accuracy on benchmarks including HumanEval, GSM8K and Math500, and unifies autoregressive, diffusion and self-speculation decoding in one tri-mode model. Google's Gemini Diffusion is credited with a promised tenfold speedup for text generation. The catch, the author notes, is that diffusion optimizes globally across an output, which can help overall structure but sometimes sacrifices fine detail — potentially better for creative work, weaker for precise technical output.

World models as the third track

The final candidate is the world model. The article recounts that Yann LeCun spent 2025 arguing autoregressive LLMs are fundamentally limited and that AI needs models that predict the world rather than words, with his team publishing Joint Embedding Predictive Architecture papers for images, then video, then language in September 2025. In December 2025, the piece says, VL-JEPA arrived — a vision-language model with roughly half the parameters of its competitors that still performed better.

Why it matters

Most of these claims — benchmark figures, ship dates and vendor speedup promises — come from a single survey article relaying lab and vendor announcements, so they warrant independent verification. But the direction matters for anyone architecting AI systems for 2027 and beyond. If energy is the binding constraint, architectures with linear scaling and constant inference memory change what is economical to deploy, and hybrids mixing attention, state space layers and diffusion decoding are already shipping rather than remaining theoretical. Engineers need not abandon transformers tomorrow, but the assumption that attention-only models stay the default looks, on this evidence, increasingly dated.