Qwen3.8-Flash-Next hits 39 tok/s on a 24GB GPU, provided you bring 96GB of system RAM

What the model actually is

Qwen3.8-Flash-Next is described in Qwen's base repository as an experimental preview, and according to a guide published on dev.to by alanwest it is the first open-weight release of the architecture expected to underpin Qwen4. Its spec sheet explains the confusion around its size: 125B total parameters, only 6B active per token, a 51B n-gram embedding component, and a 4B multi-token prediction (MTP) head.

The mixture-of-experts design spreads 512 experts across 48 layers, with 10 routed plus 1 shared expert active per token. Layers alternate Gated DeltaNet and Qwen Sparse Attention blocks in a 3:1 pattern, the model uses a gated residual design with a Muon and AdamW training mix, native context is 262,144 tokens (stretchable to one million with YaRN), and a vision encoder is included. West flags that the release uses the qwen-community-1.0 licence rather than Apache 2.0, which deserves a read before any commercial deployment.

llama.cpp support landed via PR #27742, merged August 27, 2026, so builds older than that will not recognise the qwen4exp architecture.

Why a 6B-active model ships as a 111 GB file

West's central point is that MoE saves compute, not memory. All 512 experts must live somewhere readable, because the router selects a different ten on every token. "Flash" refers to decode speed, not footprint: the smallest quant in Unsloth's GGUF repo, a 1-bit file, is 66.4 GB, and the widely used UD-Q4_K_XL is 111 GB.

The easily missed cost is the n-gram embedding table — what an r/LocalLLaMA benchmark thread calls the per-layer token embedding (PLE) table. It is a lookup of bigrams and trigrams with roughly 20 million rows feeding layer 2, measured at 27.2 GiB inside an 87.2 GiB UD-IQ4_XS file. Unsloth's documentation lists it at 102.4 GB in BF16, 54.4 GB in Q8_0, 38.4 GB in Q5_1 and 28.8 GB in IQ4_NL, and ships it as a separate shard with a set_split.py script for swapping its precision. Because each token touches only a few rows, it can sit in system RAM or be paged from SSD via mmap. Putting it on the GPU is the one mistake to avoid: West reports 108.5 tok/s decode with the table in RAM, versus a drop the guide characterises as 55x slower when it is forced onto CUDA.

The KV cache is comparatively tiny — only 12 of the 48 layers are attention layers, each with 2 KV heads at head dimension 256, working out to roughly 24 KB per token in F16, about 6.4 GB at full 262K context and under 1 GB at 32K. West also warns that Unsloth's own requirements table lists "4-bit: 64 GB," which contradicts the 111.3 GB file and another figure on the same page of at least 75 GB. The advice: plan around the file size.

Real-world throughput

West has not run the model personally; the figures come from model cards and community benchmarks. An r/LocalLLaMA test on an RTX PRO 6000 96GB with 96GB of DDR5 decoded UD-IQ4_XS at 109.07 tok/s on a 2K prompt and 21.61 tok/s at 245K context. Simulating a 24GB VRAM budget on the same machine still produced 39.01 tok/s, and even an 8GB tier — with all 48 expert layers in RAM — managed 35.69 tok/s. CPU-only decode was 8.34 tok/s. West cautions that the simulated tiers show what offloading costs rather than how a real 4090 behaves.

Elsewhere, contextstudios.ai measured 24.2 tok/s on a DGX Spark with UD-Q4_K_XL plus the draft head, while lilting.ch got roughly 18 tok/s on an M1 Max 64GB using a heavy 1-bit mix — a result West distrusts, citing a KLD of 0.2277, a hallucinated game title, a rendering bug and a swapping machine.

Running it yourself

The quickest path is llama.cpp's OpenAI-compatible server pointing at unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL. Speculative decoding with the MTP draft head uses --spec-type draft-mtp with --spec-draft-n-max 5; Unsloth claims a 1.3–1.7x speedup, smaller on low-bandwidth hardware. Mainline MTP support merged October 1, 2026 (PR #29761), so fall back to the qwen4exp/mtp branch if a stock build ignores the head. Recommended sampling for thinking mode is temperature 1.0, top-p 0.95, top-k 20, min-p 0; non-thinking mode uses temperature 0.7, top-p 0.80 and presence penalty 1.5. Ollama can pull the GGUF directly, but Unsloth's docs do not mention it, support for the new architecture is uncertain, and the command downloads the full 111 GB.

Why it matters

The 39 tok/s result reframes self-hosting economics for large MoE models: the bottleneck is total system memory, not GPU VRAM. A modest 24GB card paired with 96GB of DDR5 becomes a viable way to run a frontier-class open model, which shifts hardware budgets toward RAM. The guide is also a caution against taking model branding and official requirements tables at face value — and a demonstration of how community benchmarks, even simulated ones, fill the documentation gap.