Running MoE LLMs larger than RAM: a llama.cpp guide to expert offloading and SSD streaming

A guide published on dev.to tackles a scenario that sounds self-contradictory: running a language model whose weight file is bigger than the machine's total memory. It works, the author explains, only for Mixture of Experts (MoE) architectures, and it combines three ingredients: memory-mapped model files, llama.cpp's tensor-placement flags, and the operating system's page cache backed by a fast NVMe drive.

The piece was prompted by Janas, a project the author saw on Hacker News that streams expert weights from SSD on demand. The author's own starting point was an attempt to fit a 100B-plus-parameter MoE onto a 32GB laptop — an effort that first froze the machine outright and, once configured correctly, delivered roughly 8 tokens per second.

Why sparse models fit when dense ones don't

For a dense model such as Llama 3.1 70B, every token must flow through the entire weight set. If those weights don't fit in RAM, each token triggers many gigabytes of disk reads, and even a Gen4 NVMe drive at around 7 GB/s only manages about one token every few seconds, which the author calls effectively unusable.

MoE models behave differently. According to the guide, gpt-oss-120b carries roughly 117B parameters in total but activates only about 5.1B per token, while Qwen3-30B-A3B is 30B total with around 3B active. A router at each layer selects a handful of experts — typically top-4 or top-8 from pools that can reach 128 — and leaves the rest untouched.

Placement follows from that split. Attention layers, embeddings and shared components touch every token but are small, so they belong in VRAM. Expert FFN weights dominate the file size yet only a fraction is read per token, so they can live in RAM and page in from SSD when memory runs short. Because routers tend to reuse a hot set of experts for prompts of a similar kind, the Linux page cache naturally keeps those in memory while cold experts stay on disk. Janas and similar projects add prefetching and per-expert LRU caching on top, but the author argues that llama.cpp's default mmap behavior already captures most of the benefit.

Sizing it before the download

The guide includes a short Python script that parses GGUF metadata with the gguf package and sums tensor sizes, splitting expert tensors (identified by the ffn_*_exps naming convention) from everything else. For a Q4 build of a roughly 120B model, the author measured only a few GB of non-expert weights — enough for an 8GB GPU — against approximately 55 to 60GB of expert weights.

The author's rules of thumb: with RAM at 70 percent or more of the expert size, speed is close to an all-in-RAM run; between 40 and 70 percent, the model is usable but speed varies with the prompt and switching topics causes a brief slowdown; below 40 percent, a smaller quant or model is the better choice.

Configuring llama.cpp

Since around mid-2025, llama.cpp has exposed --n-cpu-moe and --override-tensor (-ot) for fine control over where tensors live. The author's example offloads all layers to the GPU with -ngl 99 while forcing experts to the CPU via -ot ".ffn_.*_exps.=CPU", or uses the shorter --n-cpu-moe 30 to keep the first 30 layers' experts on the CPU, tuning the number until VRAM fits.

The guide also lists pitfalls learned by trial and error. Adding --no-mmap when the model exceeds RAM forces a full load and gets the process killed by the OOM killer; default mmap lets the kernel decide which pages stay. --mlock should also be skipped, since locking every page into RAM fails immediately. Thread count (-t) should match physical cores rather than logical ones, because expert matmuls on the CPU are memory-bandwidth-bound and hyperthreads can make things slower. And when VRAM is spare, lowering --n-cpu-moe moves expert layers onto the GPU for a few percent of extra speed each.

Checking whether the SSD is the bottleneck

When generation is slower than expected, the author checks the disk first. Suggested tools include iostat -xm 1 nvme0n1 for read throughput — sustained multi-GB/s reads during generation signal thrashing, meaning the working set exceeds RAM — plus ps -o maj_flt for major page faults, where a burst over the first few dozen tokens is normal warm-up. vmtouch shows how much of the model file is resident and can pre-warm the cache before a demo. On Apple Silicon, unified memory removes the VRAM/RAM boundary, leaving total RAM as the only number that matters, with sudo fs_usage -f diskio filtered to the process revealing disk pressure.

Why it matters

The broader argument is economic: capable GPUs and rented cloud instances are expensive, so extracting maximum value from existing hardware is a worthwhile skill. The practical takeaways are to prefer MoE over dense models on constrained machines — the author says a 30B-A3B model on a 16GB laptop usually beats an aggressively quantized dense 14B — to buy a Gen4 NVMe drive before upgrading a GPU if large models are a regular workload, and to measure with iostat and vmtouch instead of guessing, since slowdowns usually stem from thrashing rather than CPU limits. Router-aware prefetch projects like Janas are pushing the approach further, but the mmap-plus-tensor-placement baseline alone now lets consumer hardware run models that, a year ago, would have required a server.