Semantic cache replay test finds one in three hits answered a different question

What the experiment measured

The DevOps Daily ran a replay test, described in a post on dev.to, designed to stress one specific weakness of semantic caching: questions that read almost identically but need different answers. The team built 24 such pairs — restarting versus reloading nginx, rotating the staging versus the production password, raising a memory limit versus a memory request — and wrote six phrasings of each side, for 288 questions in total. Because every question was labelled by construction, a wrong cache hit was a matter of fact rather than a judge model's opinion.

The results at a cosine similarity threshold of 0.80 with the bge-m3 embedding model: the cache answered 32 percent of questions from memory, with 62 correct hits and 30 wrong ones per average run. Average response time fell from 8.7 seconds to 6.6. Roughly one hit in three returned the answer to a different question.

The canonical failure in the post involves Git. A user asked how to undo a commit already pushed to main. The cache matched it against a stored question about undoing an unpushed commit and served git reset --soft HEAD~1. For a commit already on shared history, the correct command is git revert; resetting and pushing means force-pushing over history other people have pulled.

Why raising the threshold does not fix it

According to the post, tightening the threshold made things worse, not better. Between 0.88 and 0.92, the few hits that remained were wrong more often than right. Across six embedding models and thresholds from 0.50 to 0.99 in steps of 0.01, the best zero-wrong configuration any model achieved was a 0.7 percent hit rate. For four models, the only thresholds that never served a wrong answer never served anything at all, and one model produced a wrong answer even at 0.99.

The underlying problem is distribution overlap. With bge-m3, pairs of paraphrases that share an answer had a median similarity of 0.723, while designated near-miss pairs sat at a median of 0.646 — but the near-miss curve stretched up to 0.932. Around 23 percent of near-miss pairs scored higher than the median paraphrase pair. A threshold is a single line through both distributions, and no line separates them. The post attributes this to how embedding models are trained: they place questions on the same topic close together, and near-miss pairs are exactly the same topic, one word apart.

The numbers also do not transfer between models. At 0.80, e5-large-v2 answered 88 percent of questions from the cache, and 65 percent of those hits were wrong.

The obvious fix costs more than it saves

The DevOps Daily also tested the remedy most teams would reach for: a small model verifying each hit before it is served. It worked for accuracy, cutting wrong answers to under one per run at every threshold. But it left the cache no faster than having no cache at all, and more expensive at every threshold tested — eliminating the point of the cache.

The measured savings were modest anyway. On gpt-oss-120b at published prices, cost was $0.25 per 1,000 questions without a cache and $0.18 with one at 0.80. Average latency improved; the slowest 5 percent of responses barely moved.

The one place the authors consider a semantic cache likely safe is near-verbatim repeats, which mostly scored higher against their originals than any near-miss pair did. They note they did not replay those through the cache. The benchmark code is available as The-DevOps-Daily/semantic-cache-wrong-answers on GitHub.

Why it matters

The authors are explicit that this is a stress test, not an estimate of production error rates: every question's near-miss partner was deliberately in the stream, and there were no exact repeats. How many near-misses real traffic contains is answerable only from logs. But the finding that no threshold, on any of six models, cleanly separates lookalike questions is a structural property of embedding similarity, not an artefact of the setup. For AI assistants in operations, where the near-miss answers often differ in exactly the way that breaks things — reset versus revert, FLUSHDB versus FLUSHALL, cordon versus drain — a semantic cache silently returning the neighbour's answer is worse than no cache. Teams running one should measure wrong-hit rates on their own question pairs before trusting the hit-rate dashboard.