vLLM 0.28.0 ships 584 commits with Kimi-K3 and DeepSeek V4 performance work

vLLM 0.28.0 arrives with 584 commits

The vLLM project has released version 0.28.0, a large update that pulls together 584 commits from 270 contributors, 76 of whom are new to the project. According to the release notes published on GitHub, the bulk of the work targets inference throughput and memory efficiency, spanning new parallelism techniques, fused kernels, speculative decoding improvements and architectural changes aimed at the largest open-weight models.

Kimi-K3 gets a full-stack optimization pass

The headline effort is a coordinated performance push for Kimi-K3. The release adds Decode Context Parallel (DCP) support, fused FlashKDA kernels for decode and prefill, SiTU activation support for MegaMoE and GEMM-RS for sequence parallelism. Combining all-gather operations reportedly yields a 1.5x to 3x kernel-level speedup, while an adaptive speculative token budget improves DSpark time-to-first-token by roughly 60%. An optional shared-expert sharding mode saves around 17 GiB of memory per GPU. Kimi-K3 also now runs on AMD ROCm through the V2 model runner.

DeepSeek V4 and speculative decoding

DeepSeek V4 support advances as well. Sparse MLA now works end-to-end for plain decode, MTP and DSpark speculative decoding, and is joined by AMD Quark NVFP4 support, reasoning-effort prompts and mappings, optimized sparse top-k metadata kernels, narrowed eager CUDA graph regions and ROCm enablement on gfx11 and gfx950 hardware.

Speculative decoding gains DFlash2 with local convolution and a candidate selector, plus confidence-scheduled verification for DSpark. Asynchronous scheduling is now switched on automatically for draft models.

Model Runner V2 and tiered KV cache

Model Runner V2, the engine's newer execution path, continues to mature. The release adds E/P/D disaggregation, weight offloading, multi-layer MTP KV cache support, encoder CUDA graphs, token-wise decoder pooling, Transformers pooling models, attention-free model support and a thinking_token_budget parameter.

Tiered KV cache offloading also expands, with disk offloading, out-of-tree secondary-tier managers loaded via a module_path option, partial secondary-tier load results, tiering metrics and a canonical CPU layout designed to be agnostic to the parallelism strategy in use.

Rust frontend, gRPC and new defaults

The Rust frontend and gRPC API grow a standalone renderer, multimodal image inference over gRPC, explicit data-parallel rank routing and reinforcement-learning lifecycle control. The project's protobuf schemas are now published to Buf.

Several defaults change with this release. The max_num_batched_tokens ceiling doubles from 8192 to 16384, prefix caching is enabled by default for Mamba models, and the Blackwell CUDA graph capture default rises to 1024.

Breaking changes and broader model support

Upgraders should watch the breaking changes: bitsandbytes support has migrated to an out-of-tree plugin, Transformers is bumped to 5.15.0, and both the deprecated calculate_kv_scales runtime KV scale calculation and override_attention_dtype have been removed.

Model coverage widens with new arrivals including Muse Glimmer, Ling 3.0 Flash (with an FP8 variant and hybrid MXFP4 routed experts), Dots3 NOTE with native multimodal support, and Interns2mobius. Qwen models pick up ROCm enablement for Qwen3.8, a fused CUDA MTP decode kernel for Qwen3.5 GDN and fixes for text-only checkpoints. On the hardware side, NVIDIA gains FlashInfer XQA decode on SM12x parts, a CuTeDSL fused query kernel on SM100 and GB10 FP8 tuning configurations, while AMD gets a torch 2.12 / Triton 3.7 stack bump plus AITER and FP8 inference on GFX120x.

Release artifacts include PyPI wheels for CUDA 13.0, ROCm wheels from the project's own index, and Docker images spanning CUDA 12.9 and 13.0, Ubuntu 24.04 variants, ROCm, CPU and XPU.

Why it matters

vLLM has become a default serving layer for teams running open-weight models in production, so its release cadence is a useful signal of where inference engineering is heading. Two trends stand out here. First, optimization work is increasingly model-specific: kernels, parallelism schemes and memory layouts are being written around individual architectures such as Kimi-K3 and DeepSeek V4 rather than generic transformer serving. Second, the stack keeps stretching beyond single-node GPU serving, toward disaggregated prefill and decode, tiered memory including disk, and gRPC-based control planes for RL workflows. The changed defaults also matter operationally: doubled batched-token limits and shifted caching behavior can alter latency and memory profiles, so teams upgrading should benchmark rather than assume unchanged performance.