Tymfi Labs benchmark: top AI agent fixed 79% of 100 live Kubernetes incidents

Grading the system, not the report

Tymfi Labs has published a benchmark for AI SRE agents whose central design choice is where the score comes from. According to a company post on dev.to, each incident begins with a real fault injected into a running microservices deployment on Kubernetes — a shop with a frontend, cart, checkout, payments, Kafka, Postgres and feature-flag services. The agent is then paged with a symptom and nothing else, for example a report that 30% of search requests are timing out. It works from a locked-down terminal offering kubectl and standard Linux tools, with no internet access and the same command budget and time limit as every other model.

A pass is decided by examining the live system, not the agent's write-up. The examiner verifies that the user journey works end to end, that the fix holds across a hold window, that the service survives a restart, and that nothing else changed along the way. Certain shortcuts fail the attempt outright however healthy things look afterwards: deleting the namespace, scaling a service to zero replicas, disabling monitoring, removing health checks, or tampering with the examiner itself.

The post illustrates the gap with a trap scenario. A search deployment sits behind an autoscaler capped at a single replica while its CPU runs at 99%. An agent that ignores the autoscaler and simply scales the deployment to four replicas will see all replicas ready, report success, and then be scaled back down roughly forty seconds later. That attempt fails.

Edition 2 results

The current edition spans 100 incidents across eight layers, with one attempt per model per incident. Tymfi Labs reports these verified fix rates and per-fix costs:

• Gemini 4 Argon: 79% at $0.35 per fix
• Claude Opus 5.5: 77% at $0.26
• GPT-6.1 Sol: 75% at $0.11
• Claude Sonnet 5.5: 64% at $0.11
• Kimi K3: 60% at $0.44
• GLM-5.3: 54% at $0.26
• DeepSeek V4 Pro: 45% at $0.08

The top three sit within four points of one another — close enough, the post says, that the leaderboard shows a 95% range beside each pass rate given the single-attempt format. Costs diverge more sharply: GPT-6.1 Sol fixes an incident for roughly a third of what Gemini 4 Argon spends while scoring four points lower. Even the leader leaves about one incident in five broken.

The multi-fault finding

The result the company flags as most worrying comes from a pilot that is not yet on the public board. With two independent faults placed on the same checkout journey, Claude Opus 5.5 fixed three of ten incidents. In six of the seven misses, the agent repaired one fault, saw the error rate fall, and stopped — leaving the journey broken while producing a report that looks correct. Tymfi Labs argues this is precisely the failure mode a transcript-based grader would never catch.

Safeguards and stated limits

To keep live-fault scenarios honest, the company says each scenario is proven five times before shipping: broken when injected, fixed by a known reference fix, and restored exactly afterwards. Every score traces to a hash of the scenario version that ran, so results cannot silently drift when a scenario changes, and the environment is restored after each attempt so the next one starts from identical state.

The stated limits matter too. The benchmark covers Kubernetes only, in a single microservices environment, so agents that excel on another stack or on bare VMs are not measured. It is a pass@1 format. And it tests fixing, not the rest of SRE practice — paging etiquette, postmortems and customer communications are out of scope. Multi-fault incidents are built and validated, but arrive in a later edition.

Running an agent against it

The benchmark ships as an MCP server, so any MCP client can connect; the post shows a one-line setup from Claude Code and says ChatGPT and Cursor work the same way. A run uses six tools covering benchmark selection, environment preparation, receiving the page, shell access inside the sandbox, report submission and results retrieval. Users can alternatively supply a model key directly, including any model available through OpenRouter. The first three runs are free, after which it costs $0.001 per second.

Why it matters

Most agent evaluations grade what the model says, which is a weak proxy for operations: production cares whether the system works, not whether the summary reads well. By verifying recovery on the live system — hold windows, restarts and a ban on destructive shortcuts — this benchmark measures the property operators actually need before handing an agent a pager. The numbers are self-reported by the benchmark's vendor and reflect a single environment, but the pattern is instructive: the best model still leaves roughly one incident in five broken, and layered faults cut performance to three out of ten. The threefold cost spread among near-equal performers also suggests reliability and efficiency are separate dials for agent builders. Future plans include reinforcement-learning training on the same incidents, with rewards verified by the live system.