Independent benchmark finds RTK's token savings rarely lower real AI coding bills

What the claims are

RTK, short for Rust Token Killer, compresses terminal output before an AI coding agent reads it, and has accumulated more than 79,000 GitHub stars. Its popularity owes much to widely shared figures: one X post claiming RTK could cut Claude Code tokens by up to 60% reached 313,000 views. But two independent tests, one by JetBrains on its SkillsBench suite and one published by Quesma after the story reached Hacker News's front page, found those savings largely fail to appear on actual invoices. Even RTK's own README carries a disclaimer noting that cutting up to 90% of bash output is not the same as cutting your bill by 90%.

How Quesma tested it

According to Quesma, RTK rewrites shell commands such as ls, git and find into terser versions of the same output, keeping file names and sizes while dropping owners and dates. The team tested on Terminal-Bench 2.1 rather than the newer 3.0 and 4.0 releases, reasoning that cost only matters on tasks agents can complete, and that agents pass most 2.1 tasks.

They ran Claude Code with Fable 5.0 and OpenCode with DeepSeek V4 Pro 0813 through OpenRouter, five attempts per task with and without RTK on identical model routes and timeouts. After removing four Fable tasks that produced refusals, the comparison covered 85 Fable tasks and 89 DeepSeek tasks — 1,740 attempts, at a cost of more than $1,500 in tokens. Versions used were RTK 0.45.0, Claude Code 2.1.220, OpenCode 1.18.25 and Harbor 0.20.

Mixed results at best

On total spend, Fable costs fell 5% with RTK while DeepSeek costs rose 5%. Pass rates dipped slightly in both cases, from 84% to 83% for Fable and 71% to 69% for DeepSeek. Measured as cost per successful pass, Fable came out 3% cheaper and DeepSeek 7% more expensive.

Weighting every task equally, so one expensive task cannot dominate the average, Fable was 1% more expensive with RTK — a difference Quesma calls indistinguishable from zero — while DeepSeek's average task cost rose 17%. Restricting the DeepSeek analysis to the 36 tasks where all ten attempts passed still showed an 18% increase.

Almost all of Fable's savings traced back to a single task, winning-avg-corewars, where RTK roughly halved the number of turns needed; across the remaining tasks, savings were under 1%. DeepSeek had the opposite outcome on that same task, taking more turns and costing more with RTK enabled.

The savings counter does not measure money

Quesma's central objection is to RTK's own reporting metric, rtk gain, which counts raw minus filtered output bytes divided by four — not billed tokens. Across 445 DeepSeek attempts, RTK reported 349.2 million tokens saved, an 89% reduction, while real costs went up.

The post describes a concrete failure mode: in the train-fasttext task, the model requested head -1 train.txt twice, and RTK credited about 120.5 million tokens saved each time by comparing those limited reads against the entire file. Those two calls made up 69% of the reported savings, even though the requested commands would never have returned the whole file. Treating rtk gain as money saved also assumes the rest of the attempt stays unchanged, when compression can alter the agent's behaviour in later turns.

That dynamic appears throughout the data. DeepSeek's average turn carried 7% less input with RTK, but attempts took 18% more turns overall, and more turns usually meant higher cost. Quesma frames this as another form of "tokenflation" and notes JetBrains saw the same pattern on SkillsBench: RTK added turns at low effort and did not lower cost at high effort.

Caching further blunts the benefit. After each turn the context is cached, so later reads of terminal output are billed as cache reads — one tenth of normal input price for Fable and one thirtieth for DeepSeek. In the DeepSeek runs, RTK cut terminal-output characters by 9% while prompt tokens rose 9%, with model output accounting for 56–57% of cost either way.

Coverage is also partial. RTK intercepts only shell commands, so Read, Grep and Glob tool calls bypass it entirely. It applied to 31% of Claude Code's terminal calls and 51% of OpenCode's, and about half of Claude Code's Bash calls already limited their own output with head, tail or wc.

Bugs can bite

Quesma also documented a failure loop: a DeepSeek git-multibranch attempt passed a flag that rtk find 0.45.0 did not support, so the plugin rewrote the command to rtk find, which errored with "Use find directly." The agent accumulated 339 consecutive errors over roughly 12 minutes before timing out, costing about nine times the matching baseline attempt. RTK fixed the bug in 0.46.0 after the benchmark runs.

Why it matters

RTK's reputation rests on numbers that measure removed output, not money spent, and Quesma explicitly declines to recommend it as a general cost-saving tool. Transcripts showed frontier models already using the terminal efficiently — only about 7% of Fable's context was terminal output — which suggests RTK helped older models more than current ones. For anyone budgeting AI coding costs, the takeaway is to validate claimed savings against your own bills and treat byte-reduction counters as marketing rather than accounting.