dev.to guide explains running Qwen3-Coder-Next 80B MoE model locally with llama.cpp

A tutorial on dev.to walks through hosting a large Mixture-of-Experts (MoE) coding model on a mid-range home computer with llama.cpp, with no high-end GPU required. The author, who previously covered the same goal using Ollama, now focuses on llama.cpp and on the trade-offs — model size, quantization level, VRAM and system RAM — that decide whether a big model runs comfortably on modest hardware.

Why Mixture-of-Experts changes the memory math

The guide likens an MoE model to a team of specialists: a dispatcher hands each task to the people most likely to help rather than involving the whole staff. Two figures matter when planning a local setup. The total parameter count covers every stored weight and dictates how much RAM, VRAM and disk you need, while the much smaller active count — the weights exercised for a given token — largely governs generation speed.

That combination is what makes large MoE models attractive on consumer machines: answer quality can approach that of a big model while the per-token cost sits closer to a small one. As the author cautions, it is not free. The weights still have to live somewhere, so memory and disk remain the binding constraints.

Hardware targets and quantization choices

The target machine is a fairly ordinary desktop: an NVIDIA or AMD card with 8 to 16 GB of VRAM, 32 GB of system RAM (16 GB is workable but cramped), a modern six- or eight-core CPU, and an SSD with at least 50 GB free, running Windows or Linux.

The example model is Qwen3-Coder-Next in GGUF form, an MoE release aimed at coding agents with 80 billion total parameters and roughly 3 billion active per token. File size is the catch. According to dev.to, the UD-Q4_K_M build listed on the model page weighs about 49 GB, which the author judges too much for an 8 GB GPU alongside 32 GB of RAM. The advice is to begin with a smaller 2-bit quantization, such as unsloth's UD-IQ2_M build, or a smaller MoE altogether, and to regard Q4 as viable only on machines with more than 45 GB of combined RAM and VRAM plus extra room for the context window. The model page, cited in the guide, suggests more than 30 GB of combined memory for its 2-bit XL variants, which makes an 8 GB card with 32 GB of RAM a realistic, if not speedy, learning setup.

Installing llama.cpp and fetching the model

On Windows the quickest route is the official WinGet package:

winget install llama.cpp

After reopening the terminal, llama --version confirms the install; prebuilt release binaries or a self-compiled build work as well. llama.cpp needs GGUF files, the format consumed by this project and various other local AI tools. One command pulls the quantized model straight from Hugging Face and opens a terminal chat:

llama cli -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096

Running a local service and tuning the split

To let other applications reach the model, llama serve starts a server exposing a browser interface and an OpenAI-compatible API. Binding it to 127.0.0.1 on port 8080 restricts access to the same machine, keeping a private coding endpoint off the network; older Windows packages use llama-server.exe with -m pointing at a saved GGUF path.

The setting that matters most is not the model name but how the workload is divided. The -ngl flag sets how many layers move to the GPU — more layers buy speed at the cost of VRAM — and --cpu-moe pins the expert weights to system RAM while the rest of the model can still occupy the card:

llama serve -hf unsloth/Qwen3-Coder-Next-GGUF:UD-IQ2_M -c 4096 -ngl 999 --cpu-moe --host 127.0.0.1 --port 8080

This hybrid arrangement runs slower than a fully GPU-resident model, but it is what lets a large MoE fit on a small card. With 12 or 16 GB of VRAM, the guide recommends dropping --cpu-moe and comparing throughput and memory use. If the process exhausts memory, the suggested fixes are to cut the context from 4096 to 2048, step down to a smaller quantization or model, keep more expert weights on the CPU, close GPU-heavy applications, and lower the -ngl value below 999.

Why it matters

The walkthrough illustrates where local AI is heading: models with huge total parameter counts but small active footprints are pulling capable coding assistance onto consumer PCs, as long as users accept the storage burden and a slower, split GPU/CPU execution. llama.cpp's offload controls make that split tunable machine by machine rather than fixed, and the OpenAI-compatible endpoint means existing tools and coding agents can point at a private, offline model instead of a cloud API. The guide is also candid about limits — an 8 GB card will not run Qwen3-Coder-Next at Q4 fidelity or full speed — but with a 2-bit quantization and CPU-resident experts it becomes a workable experiment and a stepping stone as hardware grows.