Claude Code 2.1.269 opens official plugin eval runner; maintainer migration notes surface four gotchas

Claude Code 2.1.269 makes claude plugin eval a public, official command: a first-party runner for the evaluation suites attached to Claude Code plugins. The release, along with a same-day migration account, is documented in a field report published on dev.to by the maintainer of config-drift-checker, a tool that had already shipped its own compatible runner while the official version was still gated.

According to the report, the official runner can generate starter eval cases, runs a no-plugin ablation arm by default so every result comes with a built-in control comparison, and emits both an HTML report and JSON output. For authors who previously rolled their own harness, that reframes the problem: the harness stops being the differentiator, and the eval cases become the product.

Migration notes from day one

The maintainer migrated onto the official command the day it shipped and recorded four friction points.

Frontmatter is a whitelist, not a warning system. Cases carrying a custom covers: key, used for coverage tracking under the tool's earlier, more lenient parser, failed to load outright with an "unknown frontmatter key" error. The report's workaround is to move custom metadata into a sidecar file next to a case's prompt file, which the runner leaves alone because it does not recognize it.

The YAML parser is strict. A description containing an unquoted colon followed by a space fails to parse; two of the maintainer's six cases hit this. Quoting any such description avoids the failure.

The --case flag matches a case's declared name, not its directory. Filtering on a directory named spring-work-triggers-skill matched nothing when the case's actual name was "Spring work triggers the conventions skill."

Transcripts are ephemeral. Full tool calls and responses go to a temporary trace file that is deleted when the command exits. The JSON that persists carries scores, turn counts and cost, but not what the agent actually did.

That last limit had a concrete consequence. The maintainer's tool flags suspicious diffs as probable model refusals when it sees zero tool calls, a single turn and a short reply, so that an upstream guardrail change does not masquerade as a broken local setup. Without transcripts, "zero tool calls" and "unknown" become indistinguishable, so the label now requires positive evidence and never fires on official-runner output. The broader warning for anyone building on the JSON: decide early what "unknown" means in your pipeline.

What the official runner deliberately omits

Per the report, the official command answers one question — does the plugin work right now, on this machine — and deliberately remembers nothing. There is no stored baseline, no run history, and no way to ask whether a case regressed since last month, which cases are simply flaky, or which Claude Code release broke something. The documentation instead advises pinning a model so results stay comparable across runs.

That gap is where config-drift-checker v0.5.0, released the same day as the official runner, positions itself. Using official-format cases with the official runner underneath, it layers on a pinned baseline, history across every release, per-case noise bands learned from each case's own past (one of the maintainer's cases naturally swings by 0.75, a spread that would defeat any fixed threshold), a canary track that runs when a Claude Code release actually ships, and an automated version-bump pull request once a new model proves green twice.

Why it matters

An official eval runner signals that Claude Code plugins are being treated as a real platform rather than an experiment: authors now get reproducible scoring, control comparisons and machine-readable output without building harnesses themselves. The strictness that tripped this maintainer — rejected frontmatter keys, strict YAML, name-based filtering — is the cost of standardization, and the ephemeral transcript caps how much downstream analysis can honestly claim. This account is one maintainer's day-one experience reported on dev.to, so the sharp edges may soften in later releases. But the design boundary it describes — a stateless scoring command, with baselines, history and trend analysis left to the ecosystem — looks deliberate, and it defines exactly where third-party tooling around Claude Code evals will live.