OmniCache: open-source Python proxy adds sub-millisecond semantic caching for LLM APIs

A new open-source project called OmniCache takes aim at LLM API bills with a reverse proxy that answers semantically similar prompts from a local cache. According to the dev.to post announcing it, the pure-Python proxy sits between client applications and model backends such as OpenAI, Anthropic, Google Gemini, or local runtimes like vLLM and Ollama, and can serve a cached completion in under a millisecond — with the author claiming cost reductions of up to 50 percent.

A cache that matches meaning, not prefixes

The core idea is semantic caching: incoming prompts are embedded and compared against previously served requests, so a question phrased differently from an earlier one can still hit the cache. The author says the embedder is a 512-dimensional feature projection that runs entirely in memory with no call to a hosted embedding service, and that cosine-similarity lookups complete in under 0.8 milliseconds. No external vector store such as Redis or Qdrant is required. The post describes the project as zero-dependency, though its setup instructions do install three packages: Starlette, Uvicorn and HTTPX.

Intent-aware similarity thresholds

Semantic caches have a known failure mode: a request that is merely similar to a cached one can receive an answer that breaks code or structured data. OmniCache's answer is what the author calls dynamic intent gating — similarity thresholds that adapt to the payload. Code requires a 0.98 similarity score, JSON schemas demand an exact 1.0 match, FAQ-style questions accept 0.92, and creative writing bypasses the cache altogether. Traditional Redis-based semantic caches, by contrast, typically apply one global threshold, which the author argues can corrupt structured output.

Built for agents, voice and self-hosting

The post lists several components beyond the core cache. A tool-loop accelerator caches idempotent tool calls — file reads, git status, grep — made by coding agents like Claude Code or Cursor, with the author reporting turnaround for repeated steps falling from roughly 15 seconds to 350 milliseconds. An adaptive routing step classifies prompt complexity in under 0.2 milliseconds and sends simple classification queries to cheaper economy models. A vision cache uses 64-bit perceptual hashing to deduplicate screenshots and documents, a masking layer tokenizes sensitive credentials before requests go upstream, and a streaming replayer plays cached completions back at around 65 tokens per second with sub-10-millisecond time to first token, so cached answers still render like live streams.

The stated audiences are infrastructure engineers trimming token volume without touching application code, autonomous coding agents stuck in repetitive file-inspection loops, real-time voice pipelines where every millisecond of lookup latency compounds, and self-hosted deployments that want data to stay on the machine.

How it compares, according to its author

The post positions OmniCache against three alternatives. LiteLLM and Portkey, it argues, depend on external vector databases that add 30 to 80 milliseconds of network latency. Provider-native prompt caching only matches exact prefixes longer than 1,024 tokens and does nothing to speed up output generation. And static-threshold semantic caches risk returning malformed JSON or code. These comparisons come from the project's own announcement rather than independent benchmarks.

Running it

Setup is deliberately small: clone the GitHub repository, install the three packages, and start the main script. Existing applications only need to repoint an OpenAI SDK client at the proxy's local address on port 8000, and a bundled dashboard reports cache metrics.

Why it matters

For teams whose workloads are repetitive — agent loops re-reading the same files, support bots fielding overlapping questions — a local cache that answers before an upstream call happens cuts both spend and latency, and keeping answers on the machine has privacy value as well. The caveats are worth stating plainly: every figure here, from the sub-millisecond lookups to the 50 percent savings, comes from the developer's own post, and real-world savings will depend heavily on how repetitive the traffic is. The intent-gating thresholds are the part most worth testing before production use, since a cutoff set too loose returns subtly wrong answers and one set too tight wastes the cache entirely.