Same model, different providers: Kimi K2 tool-call accuracy ranges from 84.63% to 100%

Identical weights, sixteen points of spread

When reports circulated that Moonshot's open-weights K2 model was weak at tool calling, the company built a benchmark to find out whether the model or its deployments were at fault. According to a dev.to analysis of the resulting data, published in Moonshot's K2 Vendor Verifier repository, the answer was largely the deployments.

The harness sends the same 4,000 requests to the same model weights hosted by different inference providers, and scores each response against Moonshot's own API as the reference. On Kimi K2 Thinking at temperature 1.0, Moonshot's official endpoint and Fireworks both returned perfectly schema-valid tool calls, while Chutes managed 83.05% and Together 84.63% — a sixteen-point gap on identical weights. On the older K2 0905 release the spread stretches to roughly 28 points, with a cluster of deployments between 72% and 76%.

The underlying problem, as the dev.to piece frames it, is distribution itself: the more openly weights circulate, the more serving stacks they pass through, and from the outside a model defect is indistinguishable from an engineering deviation. Users blame the model either way.

Two metrics that catch different failures

KVV separates two behaviors. Tool-call F1 asks whether a deployment decides to invoke a tool at the same moments the official API does, derived from finish reasons. Schema accuracy looks only at responses that already committed to a tool call and checks whether the arguments validate against the JSON schema.

A deployment can pass one and fail the other. The dev.to post points to Groq on the K2 0905 table: 100% schema accuracy next to a 69.52% trigger F1, meaning tool calls fired at the wrong moments while the arguments themselves were clean. Nebius shows the same pattern at 84.47% schema accuracy and 50.60% F1.

Schema failures are the errors that reach application code: the response declares a tool call, the dispatcher extracts the arguments, and validation rejects the payload because a required property is missing. A retry sometimes recovers, but it costs a round trip and context, and it happens without the model knowing why.

Per-call scores compound over a session

The post's sharpest argument is that schema accuracy is the wrong unit for agents. If a single call validates with probability p, a session of N sequential calls fully validates with probability p to the Nth power. SiliconFlow's 98.96% per-call score leaves about 77% of 25-call sessions intact; Together's 84.63% leaves 1.5%.

Two caveats are acknowledged. Providers triggered different numbers of tool calls, so the samples are not perfectly paired, and production failures are usually retried rather than fatal, which makes these figures a lower bound on session success rather than a predicted crash rate. Either way, small per-call differences grow large under repetition.

Provider scores are per configuration, not per brand

DeepInfra scored 100% serving k2-0905-preview at temperature 0.6 but 86.91% on k2-thinking at temperature 1.0. Together appears at 71.96% on one table and 84.63% on the other. A score is a property of the model version, decoding parameters and serving stack combined, not something you can carry forward as a vendor fact.

Moonshot's own fixes were deployment-side: enforcing temperature 1.0 and top-p 0.95 in thinking mode, and requiring that thinking content be passed back to the API. The repository also specifies a tool-call ID format that Moonshot's API rewrites automatically but other providers may not, and it recommends vendors add guided decoding for schema enforcement rather than trusting prompts, pointing to vLLM v0.11.0 and SGLang v0.5.3rc0 or newer.

Routing adds yet another variable. OpenRouter's default is price-based load balancing with fallbacks enabled, so the serving endpoint can change between runs with no configuration change on the client side. The article recommends pinning a provider for evaluation runs, for example by setting a provider allowlist with fallbacks disabled.

Why it matters

For anyone building agents, this reframes provider selection as a correctness decision, not just a cost or latency one. A model that performs well in testing can silently degrade when served elsewhere, and an eval attached to an unpinned router is measuring a moving target. The practical response is cheap: pin the model version and decoding parameters, log the finish reason and a schema validation result per call, report schema accuracy alongside trigger F1, and re-run whenever the provider or model changes. The KVV table is a deployment check rather than a model review — but it shows that the gap between deployments is wide enough to decide whether an agent works at all.