A 30-minute local LLM stack: 35B MoE model on a 48GB M4 Pro Mac mini

The setup

A developer has published a working recipe for a local large language model stack that runs entirely on an M4 Pro Mac mini with 48 GB of unified memory and, according to the author, takes about 30 minutes to assemble. The post on lws.io, which reached Hacker News's front page, describes the machine as an always-on server that powers everything from an agent backend to quick chat queries issued from a phone.

Four components make up the stack: Qwen3.6-35B-A3B-OptiQ-4bit as the main model for reasoning and depth, Gemma-4-E4B-it-OptiQ-4bit as a lightweight option for routine chat and formatting, oMLX as the inference server, and Tailscale as the private network linking the Mac mini, an iPhone and a MacBook.

How the pieces connect

Hermes, an agent framework, runs on the Mac mini and draws on the same local models. The author reaches it through Telegram on their phone and through the Hermes desktop app on the MacBook, which acts as a shell for the remote backend. Sharing one backend means conversation history and configured skills stay in sync across devices.

Around the agent sit several clients, all pointed at the same server: Apollo on iOS for casual questions, Raycast AI on the Mac for ad-hoc queries, and Pi as a coding agent. Connecting Apollo, the author notes, is as simple as pointing it at the Mac mini's tailnet address.

The case for going local

The author's motivations are less about raw capability than control. Cloud APIs are compared to renting rather than owning: providers can change pricing, enforce usage limits, or swap the model being served without notice. The author says they were regularly maxing out two $200-per-month subscriptions and noticed output quality degrading at times with no announcement.

Privacy and sovereignty also factor in. Data sent to a third-party API is a one-way decision, the post argues, which matters for sensitive code or client work. Government restrictions on specific models — the author points to limits on US rollouts — could strand a workflow that depends on a cloud provider. Local hardware brings flat costs, lower latency, offline operation and no rate limits.

Notably, the goal is not to replace frontier models. The author frames local inference as covering roughly 80 percent of daily requests, with cloud models still available when a task genuinely needs them.

Mixture-of-experts changes the math

The most technical section of the post concerns memory. Dense and mixture-of-experts (MoE) models behave very differently on consumer hardware, and the author argues that parameter counts alone mislead. In the Qwen3.6 model, 35 billion parameters are spread across 256 experts, but only about 3 billion activate for any given token. On the 48 GB machine, the 4-bit build occupies about 20 GB of RAM, leaving 28 GB for context windows, the operating system and everything else. The Gemma model needs roughly 2.4 GB.

By contrast, the author cites a friend's 16 GB MacBook Air: a dense 27B model at 4-bit requires about 14 GB, which exhausts the machine and forces slow SSD swapping. Because an MoE model's active weights resemble those of a much smaller dense model, the author suggests this architecture is what makes large models viable on consumer machines. Quantization helps as well: the OptiQ 4-bit scheme reportedly costs about one to two benchmark points against the uncompressed BF16 baseline, a tradeoff the author considers acceptable.

A pre-flight checklist

The post lays out rules for judging whether a model fits your hardware: check the quantized file size first (a 4-bit 35B model runs roughly 17–20 GB), subtract 6–8 GB of macOS overhead on Apple Silicon, budget 8–16 GB for the KV cache on long conversations, and look at active rather than total parameters for MoE models. Leave a 10–15 percent buffer; anything tighter risks swapping to SSD.

Swapping models is a download and a server restart — oMLX auto-discovers files dropped into a models directory, and its dashboard includes a Hugging Face browser. Tailscale keeps the server off the public internet, with oMLX listening on port 8000 for every device on the tailnet, and its KV cache persistence helps coding agents that repeatedly revisit earlier context.

Why it matters

The post is a concrete demonstration that the gap between local and cloud models has narrowed enough for ordinary desktop hardware to carry a real, multi-device workflow. Its memory arithmetic is reusable by anyone evaluating their own machine, and its architectural argument — that MoE designs, not raw parameter counts, determine what fits — is a useful lens as open-weight releases increasingly favor that structure.