slotstream runs a 104GB Qwen MoE on a 48GB Mac by streaming weights from SSD

A developer using the handle carloslfu has released slotstream, an open-source tool that runs a language model far larger than a Mac's memory by streaming it from the internal SSD. According to the project's README, posted to Hacker News as a Show HN item on September 1, the tool runs Qwen3.8-Flash-Next — a 125-billion-parameter mixture-of-experts model that occupies 104 GB at 4-bit quantization — on machines with a fraction of that RAM, down to a planned floor of 8.1 GB.

Measured on a 48 GB Mac

On the developer's test machine, an M5 Pro with 48 GB of RAM, the README reports roughly 12 tokens per second during warm decoding, a cold start to first token of about three seconds, and peak memory use of 32 GB, with all 104 GB of weights held on disk. Lower-memory tiers are estimates: about 5 tok/s at 16 GB, 8 at 24 GB and 10 at 32 GB. Only the 48 GB figure comes from real hardware — the rest are extrapolated from the same measured curve, and the README cautions that smaller Macs also tend to have slower SSDs. A slotstream doctor command previews what a given machine should get, and whether it has room for the weights, before anything is downloaded.

How the streaming works

Almost all of the model's bytes sit in two structures: 68 GB of routed experts — 512 per layer, of which only 10 are active per token — and a 32 GB n-gram table. The dense trunk is just 3.8 GB and stays resident. Because so few experts fire at any instant, slotstream reads them on demand with pread into a fixed pool of cache slots shared by all 48 layers, so busy layers borrow capacity from quiet ones.

Cache size changes speed, never output: greedy decoding is byte-identical between a 4 GB cache and a 24 GB one, and the README says that equivalence is checked as a standing test.

The obvious alternative — memory-mapping the weight file — fails here, according to the README. MLX cannot materialize part of a memory-mapped tensor, so a top-10 expert gather evaluates all 512 experts of a layer and a 16-row n-gram lookup reads an entire 250 MB shard. An mmap path would load roughly 100 GB and die; the stock mlx_lm.load() loader reportedly pushed the 48 GB test machine into 48 GB of swap without producing a token.

Disk and download realities

Free storage, not RAM, is the first hurdle: the tool needs about 110 GB free, making a 512 GB Mac the realistic minimum. The one-time 103.8 GB download is limited by Hugging Face rather than the user's connection — throughput plateaus at 36–57 MB/s beyond four parallel connections — and a real install took 35 minutes. Interrupted downloads resume at the exact byte where they stopped, and all 24 files are verified against SHA-256 hashes compiled into the binary.

Memory sizing is automatic by default: the process takes the lowest of a 33 GB target, 70% of installed RAM, and the Metal working-set limit. Sweeping the target one GB at a time found that nothing between 34 and 84 GB improved decode or prefill speed, so 64 GB and 128 GB machines request the same 33 GB allocation — the extra memory buys nothing, and the tool says so rather than leaving users guessing. While running, it re-checks every 15 seconds and resizes the cache between requests, shrinking under memory pressure and growing back afterwards, with byte-identical output across resizes.

An Ollama-shaped API, with limits

A single Swift binary serves the chat and generate endpoints used by Ollama clients on port 11434, plus the matching OpenAI subset; the Ollama CLI, Open WebUI and the OpenAI SDKs are tested against it. Features the server does not implement — tools, images, JSON-schema output, logprobs — return an explicit 400 error instead of being silently ignored. Context is capped at 32,768 tokens. Long prompts are the slow path: prefill runs at about 125 tok/s on the 48 GB machine, so an 8,000-token prompt waits between one and three minutes for its first token. A prefix cache keeps time-to-first-token flat as a conversation grows — 6.0 seconds instead of climbing to 25.8 over eight turns — though cached reuse is not bit-identical to recomputation, and a flag disables it when exact reproducibility matters. Apple Silicon and macOS 14 or newer are required.

Why it matters

The usual answers to a model that will not fit in RAM are to buy more memory or quantize harder. slotstream demonstrates a third route: treat a fast SSD as tier-two memory for the parts of a sparse model that are rarely touched. If roughly 12 tok/s holds up beyond one benchmark machine, a 104 GB model becomes practical for interactive use on a mid-range Mac, and the binding constraint for local inference shifts from unified memory — expensive and fixed at purchase — to disk capacity and SSD bandwidth. The caveats are real: one machine measured, extrapolated figures below 48 GB, and slow prefill on long prompts. But for mixture-of-experts architectures whose weights mostly sit idle, streaming from storage is a credible way to run models that were previously out of consumer reach.