vLLM 0.31 constrained decoding lifts tool-call schema accuracy from 92% to 100%

A 92% problem and a 100% fix

When Ubicloud served Moonshot AI's Kimi K2.6 on vLLM 0.28, the model produced tool calls matching the expected schema only 92% of the time — roughly two failures in every 25 attempts. In a blog post published October 20 and since surfaced on the Hacker News front page, Ubicloud senior software engineers Jesse Donkervliet and Junhao Li explain why that gap exists and how to close it. vLLM recently added support for constrained decoding, released as a server-side flag in vLLM 0.31, and with the feature enabled Ubicloud reports 100% schema accuracy on the newer Kimi and DeepSeek models.

How tool calling works

Enabling an LLM to call an external tool takes two steps, the authors explain. First, an agent harness such as OpenCode sends the model a JSON array describing the available tools — each entry gives the function's name, description, parameter types, defaults, and which parameters are required. The inference engine then wraps that description in model-specific tokens matching the format the model was trained on. When the model decides to act, it emits its own special tokens marking the start of a call, the function name and index, and the JSON arguments. The server's parser strips those markers and returns a JSON object to the harness, which runs the tool and feeds the result back. The call only succeeds if it adheres to the tool's schema.

Where malformed calls come from

The simple approach is to let the model generate freely and trust its training to produce well-formed calls. That usually works, but a language model samples tokens by probability; nothing in the loop compiles or type-checks the output. The post points to a structural cause: models are trained on many tool-calling datasets, each with its own tools and naming conventions, so a model may emit search_web when the live tool is web_search, or city_name when the active schema expects location. Each malformed call returns an error, unwanted content builds up in the context window, the agent retries, and both quality and cost suffer — particularly in multi-turn agentic sessions where errors compound.

How Moonshot measures serving quality

To let providers verify they serve Kimi at the same quality as the original service, Moonshot AI published its K2VV ToolCall benchmark. The workload consists of 2000 requests and quantifies two metrics: trigger similarity, which measures how closely a service's decisions about when to invoke a tool match Moonshot's own service, and schema accuracy, the fraction of generated tool calls that adhere to the schema and can therefore be executed successfully.

Constrained decoding, token by token

Constrained decoding addresses the problem at the point of sampling. Instead of letting the model generate freely and validating the result afterwards, the inference engine steers generation step by step, allowing only tokens that keep the tool call syntactically valid. What counts as valid is derived from the tool's schema. According to the post, vLLM implements this using structural tags from the XGrammar library, which encode a model's tool-call syntax: where a call begins, where the function name and index appear, and the region where schema-constrained JSON arguments must be produced.

Why it matters

The post opens with a familiar complaint from developers working with AI models: results that fall well short of what quality benchmarks promise. Part of that shortfall, it turns out, belongs to the serving stack rather than the weights — the same model can show very different tool-calling quality depending on the inference engine and its configuration. Constrained decoding moves the correctness guarantee out of the model's probabilistic training and into deterministic machinery in the inference server, which matters for anyone running agents in production, where a single malformed call can derail a long session. The guarantee is also bounded: the benchmark's own separation of trigger similarity from schema accuracy is a reminder that valid, invocable calls are not the same thing as choosing the right tool with the right arguments.