Real-workload LLM benchmark: mid-tier model beats flagship priced 5.8x higher

A benchmark that put cost and correctness head to head

A developer who runs a bilingual BaZi calculator — a Chinese "Four Pillars" birth-chart app called auspiceoracle.com — published a benchmark on dev.to in which eight LLM candidates were tested against the app's real production workload before launch. Because every reading is an LLM call and the free tier functions as marketing spend, model choice was both a quality decision and a unit-economics decision. The headline finding: the flagship candidate, priced 5.8x above a mid-tier model per call at list price, was eliminated, while the mid-tier model survived at a fraction of the cost with output the developer could not tell apart.

What counted as correct

According to the dev.to post, generic leaderboards were useless here because the domain imposes constraints no public benchmark measures. The acceptance criteria were strict terminology accuracy (rendering 甲 as Yin Wood instead of Yang Wood is an error of category, not degree), respect for a closed vocabulary (the Ten Gods and fixed star names — a model that invents terms the engine never computed is a liability), and cost per reading.

How each candidate failed

The flagship preview model could not have its reasoning mode disabled: the API rejected the enable_thinking flag and restricted it to true. One streamed request produced 286 reasoning events before the first character a user could read, taking 2.1 seconds to reach the first reasoning token and 10.5 seconds to the first visible one. Those reasoning tokens bill on top of the already-higher list price, for prose the developer found indistinguishable. Excluded.

Three mid-tier models failed outright on domain accuracy, flipping elemental polarities in English output. The character-roleplay fine-tunes hallucinated the most of any group, inventing Ten Gods relationships that do not exist in the system — the models tuned for persona turned out to be the least safe choice for a persona-driven product. And two well-known open-weight models had been quietly delisted from the provider's international endpoint between the planning and testing phases, a reminder that a model dependency carries an end-of-life date you do not control.

What survived was a cheap, accurate small model for the free tier and a mid-tier model for paid users — with the twist that the mid-tier's previous generation matched its accuracy at lower cost, making it a natural fallback.

Failure handling is three problems, not one

The benchmark results were compiled into a routing layer where prices and permitted models live in one file: a route is a provider, a model and that model's list price, and each tier gets an ordered fallback chain. The chain deliberately ends with the primary model running on a backup API key rather than a different model, because account balance exhaustion takes down every model on the account at once.

The subtle part is knowing when moving to the next model helps at all. The post classifies every failure as retry, next or fatal: 401 and 403 responses are fatal, since a new model will not fix a bad key; 5xx errors and network faults are retried on the same route; and a 400 or 404 that mentions the model moves the chain forward. A single 429 can be two different errors — transient rate throttling, which should be retried, or quota exhaustion, which should skip ahead — and telling them apart requires inspecting the response body, in a provider-specific way.

When an entire chain fails, the app returns placeholder text marked with an ok: false flag, and that flag gates the database write. The reasoning: a fallback stub that gets cached would be shown to a paying user on every revisit, and the system would never retry because a cached reading already exists.

Metering what actually served the request

One reading involves up to seven parallel calls, and each emits a usage event with cost computed against the model that actually served it rather than the intended one, alongside a fell_back flag. A true fell_back value is the alert condition, since it means the workhorse model is degraded and unit margins have quietly shifted. The measured figures, at roughly 3.7k input and 0.4k output tokens per call: $0.0021 on the paid model and $0.0005 on the free-tier model, putting a two-call free reading near $0.001.

Why it matters

The app itself is niche, but the conclusions travel. Public leaderboards cannot see a domain's closed vocabulary, and the most capable general model can be both the worst performer and the most expensive on a specific workload. Cost levers can hide in model configuration — a reasoning mode that cannot be switched off multiplied a flagship's bill for no perceptible gain. The operational patterns are reusable anywhere an LLM is one component in a pipeline rather than the whole product: treat price as routing data, classify failures before retrying, end fallback chains on a different account, never persist fallback output, and meter the model that actually answered. In this app a deterministic engine computes the chart and the LLM is only allowed to phrase it — which is precisely why model selection could be settled by measurement rather than reputation.