Qwen3.8 27B at 4-bit matches full-precision coding scores on a 24GB GPU

Four-bit quantization holds its ground

How much GPU memory does it take to run Qwen3.8 27B well? According to benchmarking published by Quesma, which surfaced on the Hacker News front page, the answer is around 17 GB. That is the size of the 4-bit Q4_K_M GGUF quantization, which matched the full BF16 model — a 55 GB download — on an agentic coding benchmark. It fits on a 24 GB consumer card such as an RTX 4090 with room left for roughly 64k tokens of context.

What was tested

The motivation was partly skepticism toward a common complaint on Reddit that every quantization, even 8-bit, makes local models noticeably dumber. Token-level metrics such as KL-divergence can measure drift but not whether tasks actually fail, so Quesma measured benchmark outcomes directly across three suites: GPQA Diamond for graduate-level science, IFBench for instruction following, and Terminal-Bench 2.1 for agentic coding.

The candidates were Unsloth's GGUF quantizations on Hugging Face: 8-bit Q8_0 (29 GB), 4-bit Q4_K_M (17 GB), 2-bit UD-Q2_K_XL (10.7 GB) and 1-bit UD-IQ1_S (6.2 GB). Everything ran through llama.cpp using a build from 16 August 2026, since earlier builds did not support this model, with an F16 KV-cache of roughly 2.3 GB per 32k tokens regardless of quantization. Quesma first replicated Qwen's official BF16 scores, then measured each quantization against them, spending around $3,000 on rented Modal GPUs. One caveat: Unsloth replaced the v2 quant files on 19 August 2026, so the exact artifacts used for most tests are no longer downloadable.

Scores barely move above 4-bit

On GPQA Diamond, results sat within statistical noise from BF16 down to 4-bit, with only the 2-bit variant scoring noticeably lower. Reasoning effort mattered far more than compression: the best scores came at the default xhigh setting, consuming around 8k reasoning tokens, though Quesma cautions that this setting can cause the model to overthink.

IFBench showed no measurable degradation at all, even at 2-bit, for a model weighing under 11 GB.

Terminal-Bench 2.1, run with a 3-hour timeout and 98k tokens of reserved context, produced the headline result: Q4_K_M replicated both the full model's measured score and Qwen's officially reported figure. Q8_0 was accidentally skipped, which Quesma considers harmless since its result can be safely interpolated between 4-bit and full precision. The 2-bit model dropped visibly but still landed around the level of Opus 4.7 or Gemini 3.1 Pro. On the same solved tasks it needed roughly the same number of agent turns as BF16 but emitted about a quarter more output tokens.

One bit is one bit too few

Compression damage turned out to be nonlinear: first no measurable change, then a mild decline, then a collapse. Both 1-bit quantizations scored at roughly random-guess level on GPQA Diamond, with the smallest one below chance. Longer reasoning made things worse, because the model frequently reasoned until the token budget expired and returned an empty answer. Unsloth advertises UD-IQ1_S as retaining around 72% top-1 accuracy while being 89% smaller, but according to Quesma the missing accuracy is decisive for real tasks — consistent with an r/LocalLLaMA thread describing the 1-bit quant as unusable.

What the runs cost

The Terminal-Bench runs alone cost $2,308 in GPU time, with GPQA and IFBench adding another $663. A single BF16 configuration cost $804 against $502 for Q4_K_M, run on rented NVIDIA L40S, H100 and H200 hardware via Modal. For scale, Quesma points out that DeepSeek V4 Flash 0731, a 284B model, costs about $0.10 per million output tokens from the cheapest OpenRouter providers, though it is unclear how much of that gap reflects scale efficiency, pricing strategy or popularity.

Why it matters

For anyone running local models, the guidance is concrete: pick the largest quant that fits in GPU memory alongside the KV-cache, and on a 24 GB card that means Q4_K_M at effectively full quality for coding and agentic work. The results also push back on the recurring belief that quantization inherently degrades output — above 4 bits, none of these benchmarks could tell the difference — while showing that reasoning-effort settings are a far bigger lever over quality than compression. Below 2 bits the savings stop being worth it: tiny quants may fit in modest memory, but on this evidence they cannot solve the tasks.