PennyWyze is an open-source CLI that finds the cheapest Claude tier that passes your tests

What PennyWyze does

Three engineers from OSLabs have released PennyWyze, an open-source command-line tool that answers a narrow but expensive question: which Anthropic model tier — Opus, Sonnet or Haiku — is the cheapest one that still meets your accuracy requirements for a specific production task?

According to the team's dev.to write-up, the tool takes two inputs: the exact prompt you already send to Claude in production, and a golden dataset of real inputs paired with answers you already know are correct, formatted as one JSON object per line. You set a pass rate, and PennyWyze evaluates every example against each tier through the real Anthropic API, grades the responses, and prices each call using the token counts the API reports back rather than estimates.

How an audit runs

The command is pennywyze audit --prompt prompt.md --dataset dataset.l --pass-rate 90. The output lists each model's accuracy and projected monthly cost at your actual volume, then names the cheapest tier that cleared your bar.

One run shared by the team illustrates the payoff: Opus scored 49 out of 50 at a projected $205.94 per month, Sonnet scored 48 out of 50 at $77.30, and Haiku scored 49 out of 50 at $26.26. The verdict was to switch to Haiku and save roughly $179.68 monthly — and the audit itself cost $0.15 to run. Notably, Sonnet scored lower than both alternatives while costing nearly three times as much as Haiku, meaning there was no accuracy gain to justify any tier above the cheapest one on that task.

The tool also stops calling a model early once it mathematically cannot reach the requested pass rate, instead of burning API budget to confirm a foregone conclusion. Installation is npm install -g pennywyze, plus an Anthropic API key in a .env file.

Strict grading, on purpose

Grading is exact-match after normalization. Both the model's answer and the expected answer have surrounding quotes, code fences, capitalization differences and trailing punctuation stripped, then are compared for exact equality. A response of Billing passes when billing is expected; I think the answer is billing fails. The team says this is deliberate — a model that buries the answer in a sentence when your application needs a bare category is a real production failure, not a grading technicality worth hiding with fuzzy matching.

That strictness also defines the current scope. Per the dev.to post, PennyWyze suits tasks with a single correct answer — ticket classification, field extraction, routing, intent detection — and does not yet handle open-ended generation such as drafting emails or summarizing documents. LLM-as-judge grading is on the roadmap.

Under the hood, the audit loop is built around two small, swappable contracts: a ModelProvider, which takes a prompt and a question and returns an answer plus its token cost, and a Scorer, which grades answers against expectations. The team notes that adding a new provider, perhaps a non-Anthropic tier, means writing one new file rather than touching the loop.

An idea they almost didn't build

The dev.to posts describe a winding origin. The team initially pitched a conversation memory layer to stop chat applications from re-sending their entire history with every message. But Mem0, Zep and Letta were already working on that problem, and Anthropic had begun shipping automatic context compaction. After abandoning the idea, a teammate returned the next day with a different angle on the underlying theme of token efficiency: audit which model you are using in the first place.

Building it surfaced some quirks worth noting for anyone integrating with Claude. Opus's adaptive thinking can place a thinking block before the text block in the API response, which briefly caused the grader to score an internal reasoning fragment instead of the answer; the fix was to search for the block where type === "text" rather than assuming position zero.

Why it matters

A lot of production AI work is classification and routing rather than open-ended reasoning, yet teams routinely default to the most expensive model when launching a feature and never revisit the decision. Verifying the choice normally requires building an entire evaluation harness — a dataset, a grading system, and a way to run the same task across multiple models — which most teams skip just to answer one cost question. PennyWyze packages that harness and prices the check at cents. As the team's own audit showed, the answer can be hundreds of dollars a month in savings on a single prompt.