Benchmark finds Q4_K_M nearly twice as fast as MXFP4 for local LLMs on M2 Mac

A hands-on benchmark posted on dev.to has found that Q4_K_M, the long-standing workhorse quantization format of the GGUF ecosystem, ran a local LLM workload roughly 1.8x faster than OpenAI's newer MXFP4 microscaling format on the same Apple M2 laptop. The result runs against the expectation that the newer format, chosen by OpenAI for its local-first gpt-oss models, would be the faster option on Apple Silicon.

What was tested

The author, publishing as pitambarmahato on dev.to, ran two models through Ollama 0.12.8 on an Apple M2 with 24 GB of unified memory: Alibaba's Qwen3-14B in Q4_K_M (9.3 GB on disk) and OpenAI's gpt-oss-20B in MXFP4 (13.8 GB on disk). Each model answered the same 263-character code-completion prompt three times, with the first, cold run reported separately from the two warm runs. The author also cites a figure of roughly 70 percent of local model downloads on Hugging Face going to Q4_K_M in 2026, underscoring how widely used the older format still is.

The pairing was a pivot rather than the original plan. The author had set out to compare Q4, Q5 and Q8 variants of a single model, but switched to the two models already cached locally after estimating the extra downloads at 30-60 minutes over a home connection.

The numbers

On the warm averages across trials two and three, Qwen3-14B at Q4_K_M generated 4.7 tokens per second and completed a 200-token response in 44.0 seconds. gpt-oss-20B at MXFP4 managed 2.6 tokens per second and needed 70.8 seconds for its output, a gap of about 27 seconds for the same nominal workload. The cold-load difference was wider still: 39.1 seconds for the Q4_K_M model versus 165.7 seconds for the MXFP4 model.

Scaled up by the author, a 1,000-token completion at these rates would take roughly six minutes on gpt-oss-20B versus about three and a half minutes on Qwen3-14B.

Why MXFP4 lost, according to the author

The writeup offers a theory rather than a proof, built on three observations. First, MXFP4 on Apple Silicon runs through the AMX matrix unit rather than the Neural Engine, and the M2's AMX has no native FP4 support, which the author says arrived only with the M4, so MXFP4 weights must be dequantized to FP16 before each matmul. That cost should vanish on newer chips. Second, Q4_K_M is native to llama.cpp and benefits from years of kernel micro-optimization, while MXFP4 support in the software stack is comparatively young. Third, gpt-oss-20B emitted roughly 400 total tokens, including hidden thinking blocks, for 200 visible tokens, versus about 250 for Qwen3, so its reasoning overhead taxed measured throughput harder.

What the test does not show

This is not a clean format-versus-format experiment. It pits two different models, of different parameter counts and disk sizes, against each other because each shipped in only one quant. The author did not measure output quality, did not test Apple's MLX format, M3 or M4 hardware, very long contexts, or Q5 and Q8 variants. The practical conclusions are correspondingly narrow: on M2 and M3 Macs, Q4_K_M and Q5_K_M remain the fastest picks, while MXFP4 should become competitive once M4-class Macs and NVIDIA Blackwell GPUs, with native FP4 support, are common. The author also cautions that gpt-oss-20B's advertised 20 tokens per second assumes newer hardware; on the M2 it delivered 2.6.

The benchmark script is published in a public repository, and the author is asking for re-runs on M4 Macs to settle the open question of whether MXFP4 wins on the hardware it was designed for.

Why it matters

Most people running local models are not on bleeding-edge hardware, and this test is a useful reminder that quantization speed depends on the whole stack, including format, kernel maturity, hardware support and the model's own token habits, rather than on bit count or release date. For the large population of M2 and M3 Mac owners, the safe choice for speed is still the old integer quant, and headline throughput figures for new formats and models should be read as best-case numbers for machines they may not own. It also frames a concrete, falsifiable question for the community: whether MXFP4's promised gains actually materialize on M4-class chips, or whether the format is simply slower everywhere today.