Hybrid prompt and semantic caching cut LLM costs 63% in measured workload test

Two caches at different layers

A recent dev.to analysis tackled a question that recurs in production LLM work: when does prompt caching pay off, when does semantic caching, and where exactly is the break-even? The post's premise, echoing a HackerNoon argument it cites, is that LLM bills are an architecture problem rather than a prompt problem. Support bots, summarisation pipelines and code-review assistants receive identical or semantically similar inputs hundreds of times a day, and paying full price every time for the same computation is a design flaw, not a prompt-engineering shortfall.

The two techniques sit at different layers. Prompt caching, offered by providers such as Anthropic, reuses a repeated prefix — system prompt, few-shot examples, long retrieved documents — within requests. According to Anthropic's documentation as cited in the post, cached reads are billed at 0.1x the input token price and cache writes at 1.25x, with a minimum TTL of five minutes, which works out to roughly 90% off the cached portion of input. Semantic caching is something you build yourself, typically with Redis or GPTCache: incoming queries are embedded, and if a similar enough past question exists, the LLM call is skipped entirely and a stored answer is returned. That eliminates output token costs too, but introduces false positives — serving a wrong answer to a question that merely looked similar in embedding space.

The measured setup

Instead of trusting headline savings claims, the author reconstructed a reproducible workload: 1,200 queries built from 300 unique ones, repeated in four patterns — exact repeats, paraphrases, typo variants, and a 25% slice of genuinely unique queries. Assumed pricing was Haiku-class at $0.80 per million input tokens and $4.00 per million output tokens, with an average request of 800 input tokens (400 system prompt plus 400 user query) and 200 output tokens. The semantic cache ran on Redis with sentence-transformer embeddings at a similarity threshold of 0.92, swept between 0.90 and 0.98 to map the hit-rate versus false-positive curve. The post ships copy-paste Python blueprints for both the Anthropic prompt-cache path and the Redis semantic-cache path.

What the numbers showed

On the 1,200-query workload, the uncached baseline made 1,200 LLM calls. Prompt caching alone cut costs by roughly 38%, with savings depending on burst traffic landing inside the five-minute TTL. Semantic caching at threshold 0.92 delivered a 47% hit rate and a 2.1% false-positive rate — about 12 bad hits, most of which sampled review judged acceptable — for roughly 47% cost savings. Raising the threshold to 0.97 became conservative: 31% hit rate, 0.4% false positives, about 31% savings. The hybrid approach, where a semantic-cache miss falls through to a prompt-cached prefix, reached approximately 63%.

The post cross-checks this against a production case reported by martinkostov.me — about 67% savings in a similar hybrid configuration — and lands within a few points, attributing the gap to the unique-query share of the test workload. The author explicitly warns that the exact figure depends on traffic distribution and that readers should recompute it for their own logs.

The break-even framework

The post distils the decision into four measurements. First, repeat rate: pull 30 days of logs, normalise, and measure duplication. If identical plus paraphrased repeats fall below about 50%, semantic caching has limited headroom. Second, false-positive tolerance: if the expected cost of one wrong cached answer — churn risk, re-handling, review labour — exceeds the saving per cache hit, semantic caching runs at a loss; this trade-off, not a default value, should set the similarity threshold, and human-in-the-loop review of hits can shift it. Third, prompt-cache economics: because reads are 12.5x cheaper than writes, reusing a prefix even once inside the TTL already beats paying full price twice (1.25x plus 0.1x versus 2x), so bursty workloads almost always win while sparse trickle traffic can end up repeatedly paying the write overhead. Fourth, the two compose rather than compete — the hybrid beat either cache alone in the measurements.

The resulting rule of thumb: prompt caching is close to a default for burst-heavy traffic; adopt semantic caching only after confirming a repeat rate of at least 40% and understanding your false-positive tolerance.

Why it matters

This is one of the few write-ups that puts concrete, reproducible numbers on the trade-off between the two dominant LLM caching strategies, and it reframes the choice as measurable rather than ideological. For engineering teams, the practical takeaway is that prompt caching is cheap to adopt and almost always sensible under bursty traffic, while semantic caching is a business decision as much as a technical one — its break-even depends on how much a wrong answer costs you. The caveats matter too: this is a single-author dev.to post that ends with a services pitch, and the workload is synthetic by construction. The figures should be read as directional, and the framework — not the percentages — is the transferable part.