NVFP4 decodes 2.3x faster than BF16 on Ada GPU in same-model quantization test

A developer has published a same-model comparison of NVFP4, FP8, MXFP8 and GGUF quantization, and the headline number is a 2.3x decode speedup for NVFP4 over BF16 with GSM8K accuracy holding steady. The test, posted on dev.to by Efrain Garay, ran NVIDIA's Nemotron Labs Dory 1.0 4B — released on October 16 simultaneously in BF16, NVFP4, MXFP8 and GGUF — on an RTX 4070 Ti SUPER, a 16 GB Ada Lovelace card with no FP4 execution hardware.

One model, five formats

Quantization comparisons are usually muddied by unrelated checkpoints compressed by different people. Here every format derives from the same base weights, which is what makes the exercise unusual. Garay also says he fixed the measurement rules before running anything, pre-registering the methodology and publishing the scripts alongside the results.

The GPU choice matters for interpreting the numbers. Because Ada silicon has no native FP4 units, NVFP4 runs through a weight-only Marlin kernel: the checkpoint stays small in memory, but the arithmetic itself is carried out at 16-bit precision.

Decode throughput

Measured with vLLM at batch size 1, generation speed tracked weight size:

• BF16 — 51.0 tokens/s, 7.69 GiB of weights
• FP8 (per-tensor W8A8) — 78.5 tokens/s, 4.67 GiB
• MXFP8 — 81.4 tokens/s, 4.76 GiB
• NVFP4 — 116.8 tokens/s, 3.28 GiB

NVFP4 topped the table at 2.3x BF16's throughput. GGUF Q5_K_M, run through llama.cpp rather than vLLM, landed near BF16 at 58.8 tokens/s with a 3.42 GiB file — a useful reference for local-LLM users, though the differing engine means the format is not the only variable in that comparison.

Prompt processing flips the ranking

Ingesting a 2,048-token prompt told a different story. FP8 led at 10,118 tokens/s, ahead of BF16 at 7,003, NVFP4 at 6,978, MXFP8 at 6,061 and GGUF at 1,710. According to Garay, FP8 is the only format in the set whose multiplications Ada actually executes in 8-bit arithmetic — the likely reason it pulls ahead when the job is digesting a long context rather than emitting one token at a time.

Accuracy holds on GSM8K, diverges token by token

On the full GSM8K set — 1,319 questions with an 8,192-token cap — the vLLM formats effectively tied: BF16 answered 1,249 correctly, MXFP8 1,250, FP8 1,241 and NVFP4 1,238. All three quantized variants sit within ±3 points of BF16, and Garay reports their 95% confidence intervals stay inside ±2 points.

Token-level agreement is where the formats separate. On fixed English and Spanish texts, NVFP4 picked BF16's exact next token 89–90% of the time, against 94–96% for FP8 and MXFP8, while perplexity barely moved. MXFP8 tracked the BF16 distribution most closely — a reminder that benchmark parity and token-level divergence can coexist.

What broke before the clean run

The post also documents the problems that preceded measurement: the llama.cpp converter rejected the public checkpoint, a prefix cache initially inflated prefill numbers, and truncated reasoning was at one point scored as an answer. An audit caught a mistake in Garay's own first fidelity run as well. For anyone attempting to replicate the results, these notes may be as valuable as the headline figures.

Why it matters

For local LLM practitioners on GPUs without FP4 hardware, the result suggests NVFP4 deserves a look even before upgrading: the weight-only kernel path converts a smaller checkpoint into a 2.3x decode speedup at matched GSM8K quality, with the trade-off being somewhat lower next-token agreement than the 8-bit formats. Garay's numbers also imply a practical split — NVFP4 for bandwidth-hungry generation, FP8 for long-prompt workloads. And by shipping one model in every major format on the same day, NVIDIA made it possible to measure the formats themselves rather than the choices of whoever quantized them; other model releases could usefully copy that pattern.