Reproducible benchmarks put vLLM ahead of SGLang and llama.cpp for Qwen3-8B on workstation Blackwell

What the benchmarks covered

A benchmark writeup published on dev.to by Conatus AI compared three inference stacks serving the same model on the same GPU: vLLM 0.27.1, SGLang 0.5.9 and a CUDA build of llama.cpp, all running Qwen3-8B on an RTX PRO 6000 Blackwell with 96 GB of memory. The headline result is that engine choice barely matters for a single user, but becomes a roughly 4x throughput decision under concurrent load, where vLLM led decisively. The author also ran an FP8 pass on the winning stack and picked up another 1.5x in throughput.

According to the post, every figure comes with raw CSVs, an environment manifest and a one-command reproduction script, and the script was re-run end to end after the report was written, with all figures reproducing within 6 percent.

Methodology

The test used an identical prompt set and sampling settings across all three engines, with greedy decoding for determinism. Aggregate numbers were measured at concurrency 32 after warmup, and reported as steady-state figures over the full prompt set rather than peaks. Each engine was given the same maximum batch and context settings where the corresponding knob exists.

One methodological detail stands out: output token counts were matched across engines before any timing was compared. The author notes that engines disagree about stop conditions and detokenization, so a 10 percent difference in generated tokens can silently become a 10 percent throughput distortion.

BF16 results: a wide spread under load

At BF16 and concurrency 32, the spread across stacks was large:

• vLLM 0.27.1: 1,725 tokens/s aggregate, 39 ms median time to first token, 3.4 s end-to-end p99.
• SGLang 0.5.9: 1,327 tokens/s, 42 ms median TTFT, 5.0 s end-to-end p99.
• llama.cpp (CUDA): 428 tokens/s, 316 ms median TTFT, 16.3 s end-to-end p99.

For single-user scenarios the picture flattens out. The author measured single-stream decode in the 83 to 96 tokens/s band on every stack, and concludes that a service with one concurrent user should simply pick whichever engine is easiest to operate. Under load, however, the 4x throughput gap and the 4x spread in time to first token (39 ms versus 316 ms median) dominate, and TTFT in particular drives how responsive a chat workload feels.

The FP8 pass

Running vLLM with the official FP8 checkpoint under the same settings moved single-stream throughput from 86 to 130 tokens/s, aggregate batch throughput from 1,725 to 2,597 tokens/s, and median latency from 0.74 s down to 0.49 s. A fixed 20-prompt factual check run before and after the switch showed zero regressions. The author points out that on a 96 GB card the VRM saving is irrelevant for an 8B model; the real benefit is throughput.

The sm_120 wrinkle

The report also documents a hardware-class gotcha. Workstation Blackwell (compute capability sm_120) is not datacenter Blackwell, and default code paths tuned for the datacenter parts can fail outright on workstation silicon. Getting FP8 to run required routing around a kernel assertion in the default FP8 path: DeepGEMM asserts on sm_120, and the fix was falling back to the CUTLASS path.

The author cites a related example of the same gap, vLLM issue 53748, where the Triton MLA decode kernel overflows the 101,376-byte per-block shared-memory limit that GB10 and workstation Blackwell share but datacenter parts do not. The practical advice for anyone serving on an RTX PRO 6000, GB10 or consumer Blackwell: budget time for this class of issue, because the performance is there but the default configurations often are not.

Why it matters

Most published LLM serving benchmarks target datacenter GPUs, leaving teams deploying on workstation cards with thin guidance. This benchmark is useful precisely because it runs on workstation Blackwell, covers three mainstream stacks at matched settings, and is reproducible end to end. Its two practical takeaways are that engine choice is largely irrelevant at single-user scale but a 4x decision under concurrency, and that FP8 is a comparatively cheap 1.5x throughput win, provided you can work around kernel paths that assume datacenter hardware.

The usual caveats apply: this is a single self-published source covering one model on one GPU, and both vLLM and SGLang iterate quickly, so the specific numbers should be read as indicative rather than definitive.