Qwen3.8 27B on a Mac Studio: 14 tok/s, but terse answers keep pace with its predecessor

A developer who has spent ten days running Qwen3.8 27B as an always-on local assistant — summarising RSS feeds and filing scanned PDFs — has published real throughput numbers for the model on a Mac Studio M3 Ultra, filling a gap left by the capability-focused charts that circulated during the model's first week. The write-up on Terminal Bytes, which reached the front page of Hacker News, covers generation speed, power draw, quantisation trade-offs and the RAM budget each quant actually needs.

Per-token speed halves, answer times hold steady

According to the post, the default Ollama Q4_K_M quant of Qwen3.8 27B — a roughly 17GB file — generates at about 14 tokens per second on the M3 Ultra with 256GB of unified memory. The previous qwen3.6:27b release, at the same quant size on the same machine, manages 28.6 tokens per second, and with a far tighter run-to-run spread: 28.5–28.8 versus 13.2–15.4 for the new model. Prompt processing is nearly identical between the two, at 93.1 and 95.0 tokens per second. The author attributes the generation gap to the model's new hybrid attention design and Metal kernels in Ollama that have not yet caught up, and expects it to narrow as runtimes mature.

The slower token rate does not translate into slower answers. Across five timed runs per model on the same technical prompts, Qwen3.8 used roughly 890–1,090 tokens per answer where qwen3.6 rambled through 1,950–3,340. The author's arithmetic: 2,058 tokens at 28.6 tokens per second takes 72 seconds, while 955 tokens at 14.2 tokens per second takes 67 — so the newer model actually finishes slightly ahead in wall-clock time.

Monitoring mid-generation showed inference running almost entirely on the GPU via Metal: all 60 GPU cores at 100%, the GPU drawing about 64W, the CPU near 6W, and whole-system power peaking around 291W.

The 1-bit quant keeps facts, loses commitment

The 1-bit Unsloth quant (UD-IQ1_M, 6.7GB) that topped community discussion last week was also tested. It ran in llama.cpp at 27.2 tokens per second generation with 309 tokens per second prompt processing, using under 8GB of RAM — nearly double the Q4 generation speed. General knowledge held up: the model knew Canberra is Australia's capital and the history behind it. But asked for a simple bash one-liner it produced a working command and then burned about 400 tokens cycling through alternatives without committing to one. The author notes this matches Unsloth's own documentation, which advises against 1-bit for agentic or tool-calling use and names the 9.8GB UD-Q2_K_XL quant as the minimum for those workloads. The broader lesson: quantisation does not degrade a model evenly.

What hardware each quant needs

Summarising Unsloth's full GGUF ladder, the post maps file sizes to realistic RAM: the 1-bit file (6.7GB) and Q2_K_XL (9.8GB) fit 16GB machines; Q4 quants (16–17.6GB) need 32GB; Q6_K (22GB) wants 48GB for comfort; Q8_0 (29GB) needs 48–64GB; and BF16 (54.7GB) starts at 96GB. A 32GB mini PC can run Q4 on CPU inference, but the author expects single-digit tokens per second — acceptable for background summarisation jobs, painful for interactive chat.

Outside Apple hardware, the community reference point is AMD's Strix Halo: the strix-halo-guide project measured the official Q4_K_M at 20.4 tokens per second generation and 292 tokens per second prompt processing on a Ryzen AI Max+ 395, with the 64GB GMKtec EVO-X2 cited as a $1,999 entry point. The post also warns that DDR5 prices remain inflated — a 64GB SODIMM kit costs $750–870 — so buying a mini PC with RAM preinstalled is currently cheaper than upgrading later. Community reports cited put dual RTX 3090 setups around 60 tokens per second and an RTX 5090 between 75 and 140 depending on the runtime.

A version gotcha, and what the model offers

One practical trap: the GGUF carries the architecture tag qwen35, and llama.cpp builds more than a couple of weeks old fail with "unknown model architecture: 'qwen35'" — an error the author hit personally. The model itself is a 27.3B-parameter dense release with image and video understanding, a 262,144-token native context window and an Apache 2.0 licence. Its model card claims 61.7 on SWE-bench Pro and 89.2 on GPQA Diamond, while early community activity centred on wiring it into coding pipelines in place of paid APIs and OCR comparisons against commercial cloud tiers.

Why it matters

Launch-week coverage of a new open model measures capability; self-hosters also need throughput on hardware they already own. This post supplies exactly that for Apple silicon, and two findings stand out. First, raw tokens-per-second comparisons mislead: a model half the speed can still answer faster if it is three times as concise. Second, the quantisation results draw a clear boundary between trivia-friendly and agent-safe file sizes, useful for anyone deciding whether a 16GB machine is enough. The Metal kernel lag and llama.cpp compatibility break are also a reminder that brand-new architectures come with real setup friction in their first weeks.