How vLLM gets real throughput on H100s: PagedAttention and continuous batching explained

A technical deep dive republished on dev.to, originally from the g factor engineering blog, walks through how vLLM achieves high-throughput LLM serving on NVIDIA H100 and H200 hardware. Its central argument: generation speed is usually limited by how fast model weights and cached attention tensors move through GPU memory, not by raw compute. The author notes that watching nvidia-smi during a production run often shows compute utilization in the low double digits while users wait on slow generations.

Decode is a memory problem, not a math problem

According to the post, the two phases of inference behave very differently. Prefill processes the entire prompt in parallel, which amounts to dense matrix multiplication that GPUs handle well. Decode, where tokens are emitted one at a time, relies on sequential matrix-vector work, and each step re-reads model weights plus the accumulated Key-Value cache from high-bandwidth memory. Tensor cores spend most of that time idle.

The KV cache exists so attention does not have to recompute history at every step, but it grows with sequence length. Using an example GQA-based open-weight architecture (the post names Qwen3.6-27B, with 16 full-attention layers, 4 KV heads and a head dimension of 256), the author calculates that one 8,192-token sequence in BF16 consumes roughly 0.54 GiB. Standard PyTorch or Hugging Face generate() pipelines allocate these buffers on the fly, and under multi-tenant load that produces external fragmentation: gigabytes of VRAM may be nominally free, yet no contiguous block of the required size exists, so the next request fails with a CUDA out-of-memory error.

PagedAttention borrows from operating systems

vLLM's fix applies virtual-memory paging, an idea dating back decades in OS design, to GPU memory. Instead of reserving a worst-case contiguous chunk per sequence, the KV cache is divided into fixed-size physical blocks, typically holding 16 or 32 tokens. A block table maps each sequence's logical token positions to wherever those blocks physically live.

Allocation happens only as tokens are actually generated, which the post says cuts internal fragmentation to under 4 percent and lets the same hardware carry roughly two to four times more concurrent streams. The design also enables copy-on-write sharing: in reinforcement learning workflows such as GRPO, where 8 or 16 candidate rollouts are generated from a single prompt, all candidates share the prompt's physical KV pages, and duplication happens only where completions start to diverge.

Continuous batching removes straggler bubbles

The article contrasts this with static batching, where a batch waits for its longest request to finish. If four requests generate 50, 120, 240 and 1,024 tokens, three of the four slots sit idle for hundreds of iterations while the GPU waits on the straggler. The post calls these idle gaps GPU bubbles.

vLLM instead follows the iteration-level scheduling approach pioneered by the Orca system: the scheduler runs at every forward pass rather than at request boundaries. When a sequence emits its end-of-sequence token, its blocks return to the pool immediately and a queued request occupies that slot on the very next iteration, keeping the compute cores fed continuously.

Hopper-specific tuning

Beyond memory and scheduling, the piece turns to squeezing throughput out of H100 and H200 silicon by attacking kernel dispatch overhead. In PyTorch eager mode the Python runtime launches many operations to produce a single token, so vLLM can capture execution as CUDA graphs by disabling enforce_eager mode. The article's remaining sections, partially truncated in the dev.to feed, also cover chunked prefill and FP8 precision as further levers. All benchmarks and telemetry cited were collected on dedicated H100 and H200 clusters hosted on gft-studio.

Why it matters

For anyone serving models in production or running large-scale RL rollouts, the post reframes throughput as a systems engineering problem rather than a hardware purchasing problem. Understanding why decode is bandwidth-bound, why naive caching fragments VRAM, and how paged allocation plus iteration-level scheduling work accounts for most of what made vLLM the default serving stack. The shared-page mechanics for branching rollouts are particularly relevant as grouped RL sampling becomes a common production workload, since they cut memory costs in proportion to how many candidates share a prompt.