Janas streams MoE experts from SSD to run 80B models on a 32 GB laptop

A new open-source project called Janas takes aim at one of local AI's hard limits: running models that don't fit in memory. Janas-LLM, its first component, keeps a model's weights on disk and streams just the experts that each token activates from an NVMe SSD, while the most frequently used experts stay cached in RAM. On a laptop with 32 GB of memory, the project reports running Qwen3-Next-80B — a 48.4 GB model — at 24 tokens per second where llama.cpp manages 2.8, and at 37 tok/s when a draft model assists with speculative decoding. The claims come from the project's GitHub repository, whose announcement reached Hacker News's front page.

How it works

Janas is written in C with no dependencies beyond the C library, and it runs on CPU or GPU. Its target is mixture-of-experts models, where only a small slice of the parameters fires for any given token; the engine exploits that sparsity by treating the SSD as an extension of memory instead of loading the whole model up front. According to the repository, the engine measures the machine it runs on and chooses its own configuration: it probes thread counts across core types, and on the developer's laptop it hands prompt processing to the integrated Arc GPU. A bundled tool, janas-bench, reproduces the measurements on other hardware. Models ship pre-converted as .jns files on Hugging Face, and builds exist for Linux (Ubuntu 22.04, Debian 12 and later) plus Windows 10 (1903 and later) and 11. The project was designed and written by Maurizio "camauri" Cammalleri.

The benchmark numbers

All figures come from one machine: Debian 13 on an Intel Core Ultra 9 185H with its integrated Arc GPU, 32 GB of DDR5 and a 1 TB NVMe SSD reading at roughly 5.7 GB/s, measured between 9 and 11 October 2026. Janas was faster than llama.cpp on all 13 models compared in Q4_K_M.

The headline gaps are on the oversized models. Reading a 1,024-token prompt with Qwen3-Next-80B, Janas reached 62 tok/s against llama.cpp's 16; Qwen3-Coder-Next came in at 66 against 14. Generation ran at 24 versus 2.8 tok/s and 22 versus 2.2 tok/s respectively. The margin narrows on small dense models but never closes: Qwen3-4B reads at 193 versus 164 tok/s and writes 27 versus 18, while Qwen3-0.6B writes 130 tok/s against 45.

There is one exception: a 3-bit Q3_K_XL build of Qwen3-30B-A3B read prompts slower than llama.cpp (100 versus 112 tok/s), because those weights don't yet reach the GPU's fast prompt path — though it still wrote 36 tok/s to llama.cpp's 11. Under Windows 11 on the same laptop, writing speeds matched Linux, and the dense Qwen3-4B read prompts 1.8 times faster (346 versus 193 tok/s) through Intel's own graphics driver.

An experiment, not a product

The repository is unusually blunt about its maturity, warning up front that it is "an experiment, not a product." The API and command line change several times a week; only the .jns model format is promised to remain readable. The engine deliberately fills free memory with its expert cache and reads the disk flat-out using O_DIRECT — on machines with less headroom, the repository says, it can push a desktop into swap, and GPU use once triggered a driver reset that took the desktop down. Everything was developed and tested on that single laptop; other CPUs (especially AMD or chips without AVX-VNNI), discrete GPUs and slower disks are untried. Escape hatches include Ctrl-C to stop a reply, a --cache flag to cap memory use, and JANAS_GPU=0 to stay off the GPU entirely.

Why it matters

Local LLM inference has been gated by RAM: if a model doesn't fit, you quantise harder or you don't run it at all. For sparse MoE architectures, Janas reframes the problem by making a fast SSD the memory tier, and a gap of more than eight times over llama.cpp on an 80B model suggests the idea is more than a curiosity. If the numbers hold on other hardware, 80B-class models become practical on ordinary laptops without any cloud access. The caveats are real — one machine, early development, aggressive resource use — but as a demonstration that the memory wall can be traded for an I/O problem, this is one of the more interesting local-inference results in some time.