Dense local LLMs hit 87-91% of laptop memory bandwidth while MoE models trail at 43-67%

Benchmarks of eight local LLMs run on a single laptop, converted from tokens per second into memory traffic, show dense models reading the memory bus at 87-91% of its rated speed while mixture-of-experts models manage only 43-67% of the measured ceiling. The underlying measurements were published by a reader identified as maruhana, and the bandwidth arithmetic is by jaxsinclair in a dev.to post.

The setup

The test machine is a Surface Laptop Ultra on Windows 11 Arm64: an RTX Spark N1X pairing a 20-core Grace CPU with a 6,144-core Blackwell GPU and 128GB of LPDDR5X-9400 on a 256-bit interface, a platform NVIDIA's porting guide rates at 300 GB/s. Independent reviews measured CPU-side copies at around 122 GB/s, which the post describes as a different test rather than a contradiction.

Every model ran on the same llama.cpp build using llama-bench's tg128 generation test, speculative decoding off, three runs averaged, with standard deviation under 0.7 tokens per second. Two details frame the results: the roughly 82GB GPU carve-out is, per NVIDIA's guide, backed by system DRAM rather than separate VRAM, so the machine effectively has one memory pool. And the largest run peaked at 64.5GB of the available 128GB — no model hit capacity, so the bus, not memory space, was the limit.

Turning tokens into bytes

The conversion has three steps. File size divided by total parameters gives effective bytes per parameter. Multiplying by active parameters gives the bytes a generated token must read. Multiplying by tokens per second gives an implied read rate in GB/s. The method is exact for dense models, approximate for MoE — a token only reads its activated experts — and it ignores KV-cache traffic, so every figure is a lower bound on real bus traffic.

Dense rows define the real ceiling

The three dense models came in three file sizes, 14.3, 16.5 and 21.8GB for Mistral Small 3.2 24B, Qwen3.8-27B and Hermes 4.3 36B, and three speeds: 18.5, 16.5 and 12.0 tokens per second. Their implied bandwidth was 264.6, 272.2 and 261.6 GB/s — within 4% of each other, and 87-91% of the rated 300. jaxsinclair's argument is that three independent files converging on one number measures the memory system rather than the models. The usable ceiling on this laptop is therefore about 265 GB/s, some 11% below spec, and that is the figure MoE results should be judged against.

MoE models pay a routing tax

The five sparse models landed between 114.3 and 177.4 GB/s, or 43-67% of the measured ceiling. The spread is instructive. Nemotron 3.5 Bolt 74B-A9B, quantized to just 2.38 bits per weight, was the worst at 114.3 GB/s: fewer bytes did not buy speed, because the kernel spent the saved traffic unpacking 2-bit weights while the bus waited. gpt-oss-120B marked the efficient edge at 177.4 GB/s, or 67% of the measured ceiling.

The post spells out the practical bargain. gpt-oss-120B, with 117B total and 5.1B active parameters, reads 2.76GB per token against the dense Qwen3.8-27B's 16.5GB — a 6x discount in memory traffic. The speed gain is 3.9x, not 6x, because bus saturation falls from 91% on the dense row to 59% on the sparse one. Part of the MoE discount is paid straight back in routing overhead.

Token rates can mislead

maruhana's first published figure for one model was about 100 tokens per second; he corrected it himself, noting it included MTP speculative decoding with two draft tokens and counted thinking tokens. With drafting off, the same model ran at 75.7 tokens per second on llama.cpp and 81.5-81.7 on Ollama. The drafted figure implies 189.4 GB/s against 143-154 GB/s undrafted, but speculative decoding verifies several tokens per weight read, so tokens per second stops being a bandwidth proxy once drafting is on. The post's test for any published figure: was drafting on, and were reasoning tokens counted?

Prefill shows the compute side: 3,092 tokens per second on a 512-token prompt, dropping to 2,138, or 69%, on a 32,768-token prompt. Generation speed alone never tells you what a machine feels like in use.

Why it matters

For anyone running local models on unified-memory laptops, the planning number is the roughly 265 GB/s a dense model can actually reach, not the 300 GB/s on the spec sheet. The data also quantifies the MoE trade-off: a large model's quality at a discount on memory traffic, with part of the discount lost to lower bus saturation — and smaller quantized files can even be slower rather than faster. It is also a caution that token-per-second figures from different setups are not comparable unless drafting and token counting are disclosed. jaxsinclair is soliciting matching measurements — one dense and one MoE model, same method — from hardware such as discrete 24GB cards and Apple silicon machines, to give this laptop's ceiling some context.