Apple's MLX runs 816B-parameter Muse Spark Open across Mac Studio clusters at 36+ tokens per second

What Apple released

Apple's machine learning research team has detailed how to run Muse Spark Open, a newly released model with 816 billion parameters, entirely on desk-side hardware using the newest version of MLX, the company's open-source array framework for Apple silicon. According to the post, a cluster of four Mac Studios generates more than 36 tokens per second with the model at an 8,192-token context size, a pace faster than most people read.

The pitch is privacy and ownership: MLX taps the unified memory architecture of Apple silicon so researchers and developers can test new inference and fine-tuning techniques on hardware they control, rather than sending data to a hosted API.

What fits on one machine

A single Mac Studio can be configured with up to 512GB of unified memory. Apple benchmarked two dense open-weight models — Qwen 3.8 (27B, BF16) and Mistral Medium 3.5 (128B, FP8) — and reports they occupy up to 68GB and 158GB respectively when running with 65,536 tokens of context, comfortably inside one machine.

Muse Spark Open is a different case. It is a mixture-of-experts model with 816B total parameters but only 42B active at inference. In its native BF16 precision the weights alone would require more than 1.6TB of memory; quantized to 4-bit NVFP4 the footprint drops to roughly 493GB, which still leaves no headroom for inference overhead on a single box.

Scaling out with RDMA over Thunderbolt

To move past the one-machine ceiling, the new MLX release (v0.30.1 and later) adds distributed support built on a low-latency RDMA driver that shipped in macOS 26.2 over Thunderbolt 5. Apple's setup links machines with two Thunderbolt cables to double the connection bandwidth, and the framework ships a new set of communication collectives called jaccl — pronounced "jackal" — plus an mlx.launch helper that starts jobs across a list of hosts.

Two parallelization schemes are supported. Pipeline parallelism splits a model by depth: one Mac executes the first half of the layers and hands its output to the next, an approach Apple describes as needing only minimal code changes. Tensor parallelism splits the model by width instead. The headline throughput figure comes from tensor parallelism across four Mac Studios, with pipeline and two-machine tensor results also shown in the post's charts.

M5 Ultra versus M3 Ultra

The post compares the latest M5 Ultra Mac Studio against the previous M3 Ultra generation. Time to first token is compute-bound, and the M5 Ultra's GPU Neural Accelerators — dedicated matrix-multiplication hardware — deliver up to 4.6× faster first-token times. Generation speed is bound by memory bandwidth instead: the M5 Ultra provides 1.2 TB/s of unified memory bandwidth versus 819 GB/s on the M3 Ultra, which translates into 1.6× faster decoding.

Power draw stays modest by Apple's measurements: peak consumption of roughly 200W for the M3 Ultra machine and 300W for the M5 Ultra, figures the company contrasts with an ordinary household toaster. Apple argues that envelope makes the hardware practical for AI work in everyday homes and offices.

The team also notes that agentic coding applications, which frequently need very large contexts, are a popular local workload, which is why results are reported at both 8,192- and 65,536-token context sizes. All measurements were produced with the mlx_lm.benchmark tool from the MLX LM package.

Why it matters

Running a frontier-scale model has mostly meant renting datacenter GPUs and shipping prompts to someone else's servers. Apple's post is a working demonstration that, with a mixture-of-experts architecture, 4-bit quantization, and an inexpensive interconnect like Thunderbolt 5, a near-trillion-parameter model can run on desk-side hardware under a fully open-source stack. For developers, that means agentic and large-context workloads can stay entirely private, at a predictable power cost of a few hundred watts per node. It is also a strategic signal: Apple is investing in the surrounding software — OS-level RDMA drivers, cluster communication libraries, and multi-node launch tooling — to position Apple silicon as a serious platform for local AI experimentation rather than just a client for cloud APIs.