Speculative decoding speeds up LLM inference by trading memory-bound for compute-bound work

The misconception, corrected

A technical post on jbarrow.ai that reached the Hacker News front page takes aim at a common explanation for speculative decoding's speedup. The popular story is that verifying a draft costs fewer FLOPs than generating those tokens would. According to the author, this is wrong: for the same output sequence, speculative decoding executes more floating point operations in total, not fewer. What an accurate draft model actually buys you is a different mix of GPU work, not compute savings.

How the technique works

Speculative decoding pairs a small draft model with a large target model. The draft model proposes a short run of likely tokens, and the large model checks all of them in one parallel forward pass, keeping the tokens it would have produced itself. Conventional autoregressive decoding generates a single token per forward pass instead.

Two kinds of GPU bottleneck

The explanation hinges on what limits the GPU at any given moment. Alongside the tensor operations themselves, such as matrix multiplies and dot products, the GPU also spends time moving weights between global and local memory. When it is waiting on those transfers, a step is memory bound; when the data is resident and arithmetic is the constraint, it is compute bound.

Single-token decoding is strongly memory bound: every generated token forces the GPU to stream all of the model's active parameters, which the post illustrates with Qwen3.5-27B and its 27 billion parameters. Batched decoding that processes thousands of tokens per step sits at the opposite, compute-bound extreme.

That memory-bound regime leaves arithmetic units idle. The author points out that decoding two sequences takes no longer per token than decoding one, because the extra compute capacity was going unused anyway. Speculative decoding, at low batch sizes, is a way to spend that surplus.

Batching sequences versus speculating deeper

The post makes the trade concrete with an example: suppose five tokens per forward pass is the optimum for a given setup, where any fewer leaves the GPU memory bound and any more makes it compute bound. You can reach that optimum in two ways. Batch five sequences and decode one token for each, or keep one sequence and verify five speculative tokens for it. The GPU does roughly the same work either way; the difference is whether five sequences each advance a little or one sequence advances a lot.

The same reasoning exposes the technique's limits. Combine a large batch with a fixed speculation depth and you push past the compute-bound threshold, slowing every request down. Rejections make it worse, since each rejected token is pure overhead: work performed that yields no accepted output.

Consequences for serving and research

This is why the technique is a low-batch optimization rather than a universal win. The post notes that vLLM exposes a --speculative-disable-by-batch-size flag so operators can switch speculation off entirely once batches grow large enough.

The analysis also motivates a research direction: if the right depth depends on how much compute is spare, why speculate to the same depth at every timestep? The author points to work such as dSpark as building on exactly this intuition, speculating deeply when batches are small and compute is free, and shallowly, or adaptively per sequence, when they are not.

Why it matters

For inference engineers, the post replaces a vague "less work" mental model with one you can act on. Whether speculation helps depends on where a workload sits between memory-bound and compute-bound operation, and on the draft model's acceptance rate. As deployments scale up throughput, a fixed speculation depth quietly turns from an accelerator into overhead, which is why dynamic, load-aware depth is becoming the interesting frontier both in production serving stacks and in the research literature.