Qwen3.5 and Qwen3.6 MTP head reaches 1.4x on Apple Silicon after three fixes

A prediction head shipped with the model

According to a write-up from the OptiQ team published on dev.to, Qwen3.5 and Qwen3.6 both include a compact multi-token prediction (MTP) head inside the model package, roughly 185 MB on the 9B variant. Its purpose is speculative decoding: the head drafts K candidate tokens quickly, and the main model checks all of them in a single parallel forward pass. When the drafts agree with what the main model would have emitted anyway, those tokens cost almost nothing extra. The OptiQ team assumed wiring this through their mlx-lm-based serving path would take an afternoon. It took three separate fixes.

Duplicate weights in unified memory

OptiQ's integration patched stream_generate in mlx-lm so that MTP-enabled generation routed through its own engine. On first request that engine loaded the model again, even though mlx-lm's server had already loaded it, leaving two full sets of base weights resident in Apple's unified memory. Peak memory for Qwen3.5-9B jumped from 6.5 GB without MTP to 16.7 GB with it, and the 27B model could not run with MTP at all on a 24 GB Mac.

The fix was a classmethod, OptiqEngine.from_loaded, which attaches MTP support to the already-loaded model in place. After that, the 9B peak fell to 6.7 GB and the 27B fit at 17.6 GB, which is what made benchmarking the larger model feasible.

The verify was wrong under sampling

Greedy verification, comparing the draft token to the main model's argmax, worked. But at temperature the code compared two independent samples, one from the main model and one from the draft, and two independent draws from different distributions almost never match exactly. Measured on the 4B model with Qwen's recommended sampler (temperature 1.0, top_p 0.95, top_k 20), acceptance was 28 percent versus 67 percent for greedy.

The correct approach is rejection sampling, attributed in the write-up to Leviathan et al. and Chen et al.: accept a draft token with probability min(1, p_main(dt) / q_draft(dt)), and on rejection draw a replacement from the clipped, rescaled difference of the two distributions. That change lifted 4B acceptance from 28 to 57 percent and moved throughput from a 0.87x regression to 1.11x.

Truncation has to match on both sides

OptiQ was still short of the roughly 83 percent acceptance unsloth publishes for this model class. The remaining gap was that the draft path applied top_p and top_k while the verify used the untruncated softmax. Restricting both distributions to the same support, the top 20 tokens and the top 0.95 of cumulative mass, raises the overlap between them, and that overlap is what rejection-sampling acceptance measures. On the 9B model acceptance rose from 32 to 63 percent and throughput from 1.00x to 1.28x. OptiQ also made the settings in Qwen's generation_config.json the default sampler in its serve and lab commands, so users setting only a temperature flag would not hit the same diffuse-distribution problem.

Why K=1 wins on Metal

vLLM, HuggingFace Transformers and unsloth's guide all default to two or more draft tokens. OptiQ tested K=1 through K=4 across all three model sizes and found K=1 consistently best on Apple Silicon: 1.28x versus 1.19x at K=2 for the 9B, dropping below baseline at K=3 and K=4. The reason is hardware: on CUDA, verifying extra tokens is nearly free because Tensor Cores have spare throughput during single-token decode, while Metal's compute-to-bandwidth ratio is roughly 10x lower than an H100's, so a K=2 verify costs about twice a K=1 verify. HuggingFace's dynamic-depth heuristic, which raises K after fully accepted cycles, lost 4 to 17 percent against fixed K=1.

Final numbers

On an M4 Pro with 24 GB, greedy decoding, 512 generated tokens, OptiQ 0.1.0 reports 1.20x on Qwen3.5-4B (29.2 to 35.0 tok/s), 1.32x on the 9B (19.5 to 25.8 tok/s) and 1.40x on Qwen3.6-27B (6.0 to 8.4 tok/s). With Qwen's sampler the figures become 1.09x, 1.17x and 1.30x. The 27B greedy result lands within five percent of the 1.4x unsloth publishes for the same model class on an RTX 6000.

Why it matters

The MTP head in Qwen3.5 and Qwen3.6 is not a flag you simply turn on. Memory layout, the verification rule at temperature and sampler-truncation consistency each decide whether speculative decoding helps or actively hurts, and none of these issues show up until you measure. Ecosystem defaults such as K of two or more are tuned for CUDA hardware and mislead on Apple Silicon, where single-token drafting is the sweet spot. For anyone running these models locally, the practical takeaway is that a correctly wired MTP head gets a Mac most of the way to workstation-GPU speedups, provided the draft and verify paths share the same truncation and the implementation reuses already-loaded weights.