vLLM benchmarks five speculative decoding methods on AMD Instinct GPUs

A faster decode loop without changing the model

Most LLM serving systems still run standard autoregressive decoding: the model emits one token, appends it to the context, and repeats. The vLLM team, in a blog post that reached the Hacker News front page, describes this loop as dependable but inherently serial — every output token costs one pass through the model, which can dominate latency and cap throughput during long generations.

Speculative decoding attacks that serial bottleneck without replacing the model. A small draft component proposes several candidate tokens ahead of the current position, and the target model then checks the whole proposal in a single verification pass. Checking runs left to right: accepted candidates are committed, but the first rejected token halts the run — the target model supplies the corrected token at that position and everything drafted after it is discarded. When most candidates survive, one target-model pass commits several output tokens, and the post emphasizes that the target model's output behavior is preserved throughout.

A worked example from the post: given the prompt "The weather today is", a drafter proposing "sunny and warm outside" might see the first two tokens accepted, "warm" replaced by the target model's choice of "clear", and "outside" thrown away because it followed the rejection.

Five drafters, three architectural camps

The post evaluates five drafting approaches — native MTP, Gemma 4 MTP, EAGLE-3, DFlash and DSpark — and sorts them into three groups based on how the drafter receives information from the target model and whether candidates are produced sequentially or in parallel.

Native MTP modules live inside the target model's own architecture and generate candidates sequentially through an auxiliary prediction path. Separate MTP drafters, the camp Gemma 4 MTP falls into, pair a distinct checkpoint with a specific target model, drawing on target-model activations and shared KV-cache information while still drafting one token at a time. The third camp, dedicated target-conditioned draft networks, covers EAGLE-3, DFlash and DSpark: EAGLE-3 drafts autoregressively from the target model's hidden states, DFlash drafts token blocks in parallel from those hidden states, and DSpark adds lightweight causal correction plus confidence-based prefix selection.

These categories describe the drafting side only, not the target-model family. A single target model, the vLLM team notes, may support native MTP while also having separately trained EAGLE-3, DFlash or DSpark draft models available.

Results on MI300X and MI355X

The measurements were run on AMD Instinct MI300X and MI355X GPUs using the ROCm open software platform. The headline finding is deliberately unspectacular: the effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, the draft checkpoint, the workload, and how often the target model accepted the drafts. No configuration won everywhere.

That variance follows mechanically from the verification rule. Because a proposal dies at its first rejection, longer proposals only pay off when the drafter is accurate enough for that particular workload — which is why the post pairs its measurements with instructions for enabling each method and a discussion of tuning and observability considerations.

Why it matters

Decoding throughput translates directly into serving cost: every draft token the target model accepts is an output token produced without an extra full-model pass, and the central claim of the technique is that this speedup comes without changing what the model actually says. The benchmark's practical lesson is that speculative decoding is not a switch to flip but a per-workload tuning problem, so operators on Instinct hardware should measure their own model, drafter and proposal-length combinations before committing to a configuration. For AMD, vLLM publishing concrete tuning data for MI300X and MI355X under ROCm is also a signal of intent in LLM serving, a space where most published optimization guidance has accrued around NVIDIA hardware.