llama-modchip claims 284B-parameter DeepSeek model runs on a 64GB CPU with no GPU

What the project claims

An open-source project named llama-modchip reached the Hacker News front page on November 8, 2026 with a striking claim: a 284.33-billion-parameter mixture-of-experts model, DeepSeek-V4-Flash, can run entirely on a CPU-only desktop with 64GB of RAM and no graphics card at all.

According to the project's GitHub README, the model — 284.33B total parameters with roughly 13.17B active per token, spread across 43 MLA layers and 256 routed experts plus a shared one — occupies 56.56 GiB on disk and peaks between about 57.6 and 58.9 GiB of resident memory even at a 256,000-token context. The unmodified checkpoint reportedly weighs 76.87 GiB, enough to trigger an out-of-memory kill on the same machine, so compression is what makes the fit possible.

Reported throughput is 44.36 tokens per second on prefill, with decode between 9.06 and 9.95 tok/s in lossless mode and up to 11.5–14.7 tok/s once speculative drafting kicks in.

How the compression works

The README credits three families of techniques. First, an analytical second-order Hessian compression of the SwiGLU layers, called SwiGLU-H2, which the authors say removes 20.31 GiB on its own. Second, a "Fractal-64" quantization scheme at 3.75 bits per weight, built on llama.cpp's Q2_0 format, applied to the attention weights alongside MoE pruning. Third, kernel-level tuning: AVX-512BW register tiling across the L1, L2 and L3 caches plus a register-stationary MQA8 FlashAttention kernel, together credited with lifting prefill 77 percent, from 25.0 to 44.36 tok/s. An exact SVD step on the output projection is reported to retain 99.82 percent cosine similarity at 25 percent lower rank.

The decode-side gain comes from a 1.64MB speculative drafter the authors call Stockfish-PVS v2, sized to stay resident in L2/L3 cache, with a claimed 96.4 percent acceptance rate.

Where smaller models land

The same toolchain targets 30B-class MoE models that run on ordinary laptops. NVIDIA-Nemotron-3.5-30B-A3B, a hybrid that combines Mamba-2 state-space layers with MoE routing (30.2B total, 3.0B active), takes 14.35 GiB and is reported at 25.0–60.4 tok/s on a 16-core Zen 5 CPU, 44.97 tok/s on an AWS Graviton3 ARM instance, 80.8 tok/s in a hybrid setup with a 16GB GPU, and 111.6 tok/s on a Threadripper workstation whose eight-channel DDR5 saturates memory bandwidth. Qwen3-30B-A3B is also supported, at 34.8 tok/s in pure-CPU mode.

For machines with a modest GPU, a hybrid mode keeps the attention trunk in VRAM while the routed experts stay in system RAM.

Agent use and a compatibility catch

The project also ships a Claude Code launcher that wires the coding agent to the local engine with all 40 tools active. A /dev/shm-based warm restore of the KV cache takes 13.1–32.6ms on the 284B model, which the authors say eliminates a roughly 45-second prompt re-evaluation pause on each agent turn.

One notable limitation: Fractal-64 files use a custom tensor encoding (GGML type ID 40) that unpatched llama.cpp, Ollama and LM Studio will refuse to open. Users must run the project's bundled pre-compiled binary from its Hugging Face repositories or GitHub releases. Windows users get a one-click script that detects cores, free RAM and VRAM, then computes an optimal CPU/GPU split automatically.

Why it matters

Every number above is self-reported by the project; the README offers a reproducible evaluation command, but the benchmarks have not been independently verified, and output quality at 3.75 bits per weight — where the authors report a perfect score on their own 100-point gate for the 30B builds — is the obvious open question.

If the claims hold even approximately, the GPU barrier to frontier-scale local models effectively disappears. A 64GB desktop is a commodity purchase; an 80GB-class accelerator is not. Ten tokens per second is not chat-snappy for everyone, but it sits comfortably in the range useful for agents, summarisation, retrieval and batch work — precisely the workloads where running locally, with no API bill and no data leaving the machine, matters most.