deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

NVIDIA open-sources Nemotron-Labs-Sigma, a continuous diffusion language model family with 14 checkpoints

NVIDIA has released Nemotron-Labs-Sigma, an open-weight family of continuous diffusion language models, with 14 checkpoints across 3B and 8B sizes published on Hugging Face under a commercial-use license.

NVIDIA open-sources Nemotron-Labs-Sigma, a continuous diffusion language model family with 14 checkpoints

NVIDIA has released Nemotron-Labs-Sigma, an open-weight family of continuous diffusion language models. According to a dev.to post describing the release, which appeared on Hugging Face on November 20, 2026, the family ships 14 checkpoints in two sizes, 3B and 8B, covering base, instruct and few-step distilled variants (NFE 2, 4, 8 and 16), plus a self-speculative variant at each size. The weights are published under the NVIDIA Nemotron Open Model License, which permits commercial use, so teams can download, fine-tune and serve the models on their own hardware.

A third way to generate text

Continuous diffusion sits apart from the two familiar paradigms. Autoregressive models emit text one token at a time, strictly left to right, while discrete diffusion models such as LLaDA and Dream operate on discrete token IDs. Sigma instead works on continuous token representations, denoising them inside a learned 16-dimensional embedding space, and produces text in 64-token blocks. Each block passes through several denoising steps before it is finalized, after which the model moves on to the next block.

The training lineage is unusual as well. Per the dev.to summary, the models were warm-started from an autoregressive checkpoint trained on roughly 1 trillion tokens, then continued with about 0.3 trillion tokens of diffusion training — effectively a diffusion layer built on top of a conventional autoregressive foundation.

Self-reported results

All benchmark figures come from NVIDIA's internal evaluations and are not independently verified, a caveat the dev.to post states explicitly.

On NVIDIA's six-task suite, Sigma-8B base averages 71.16, ahead of NLD-8B at 70.22, Dream-7B at 58.74, LLaDA-8B at 41.78 and the autoregressive Qwen3-8B at 68.91. The 8B instruct variant reaches 91.96 on GSM8K. The clear weak spot is AIME25 at 17.00, meaning competition-level mathematics lags well behind the family's other results. The context window is 4,096 tokens, extended to 8,192 tokens for the 8B instruct variant.

Throughput and serving

NVIDIA reports roughly 120–140 tokens per second on a single H100 in bfloat16. The self-speculative 8B variant is said to decode up to 3.46x faster than the autoregressive average on a GB200 running SGLang, with a peak of 4.20x. The few-step distilled variants accept some extra compute in exchange for fewer network round-trips, a trade that pays off when serving latency is the bottleneck.

There is no first-party API. Everything runs on the user's own infrastructure through vLLM, SGLang or Docker Model Runner, with commercial use allowed under the license.

Why it matters

Nearly every widely deployed language model today is autoregressive, which caps decoding speed at roughly one forward pass per token. Diffusion language models generate whole blocks in parallel, and the distilled few-step variants push further in that direction. If the reported quality holds up under independent evaluation, Sigma gives self-hosting teams a genuinely different architectural option, with permissive commercial licensing and no dependency on external endpoints — attractive for anyone already operating a GPU fleet.

The caveats are just as real: the numbers are self-reported, competition-level math is a visible weakness, and the 4K–8K context window is far below what current autoregressive models offer. Still, as an open-weight release from a major vendor, Sigma is a strong signal that non-autoregressive generation is being pursued seriously beyond research labs.

  • #nvidia
  • #diffusion-models
  • #open-source
  • #language-models
  • #hugging-face

Related posts