deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

dev.to post details cross-request KV cache leak in Workers AI and its SGLang remediation

A dev.to write-up describes how a config mismatch let one request's KV cache pages leak to later requests on Workers AI, and the three-part SGLang fix: request-scoped keys, an epoch fence and a minimum TTL.

dev.to post details cross-request KV cache leak in Workers AI and its SGLang remediation

What the post reports

A post on dev.to, written by Mubashir Arham and originally published on FutureStack, describes a cross-request key-value cache exposure in a large language model served on Cloudflare's Workers AI platform. According to the post, the incident dates to late 2024 and involved a model it identifies as GLM-5.3, roughly 5.3 billion parameters, with token generation orchestrated by the SGLang runtime. The author frames the flaw as both a security risk and an operational hazard: when an inference service leaks state between independent client calls, tenant isolation is broken and cache behaviour becomes unpredictable.

How the stack is described

The post lays out the serving architecture in detail. The model is sharded across three V8 isolates, each holding a local attention KV cache. Cache pages are persisted in Workers KV, which the post says offers eventual consistency and strong read throughput but no built-in request-level isolation. Pages are addressed by a composite key combining a static model identifier, a per-request UUID, a layer index and a token index. Because the per-token latency budget at the edge is around two milliseconds, fetches are batched.

On the runtime side, SGLang's CacheHandle pre-allocates a buffer pool, maps KV pages to buffer slots through a hash table keyed by that composite string, and returns buffers to the pool once a request completes. The stated invariant is that no buffer may be reused before its request's lifecycle ends; the pool is shared across the isolate to reduce allocation cost.

How the leak happened

Workers AI, as described in the post, exposes two mutually exclusive SGLang modes. With sglang.kv_isolation set to true, labelled strict, every request receives a private pool and buffers are zero-filled before reuse. Set to false, labelled relaxed, buffers are recycled across requests to raise throughput.

The post says a deployment script combined kv_isolation = false with a KV-page TTL of zero, meaning pages never expire. In that configuration, buffers from a finished request could be handed to the next request before Workers KV had flushed the stale pages. An attacker, the author writes, could infer earlier tokens' embeddings by timing fetch latency, or in the worst case read raw KV payloads outright.

At the code level, the post pins the bug on the CacheHandle::acquire routine: when isolation was disabled, the pool lookup key dropped the request_id component, so two unrelated requests could resolve to the same buffer slot. The isolation contract failed silently.

The remediation

FutureStack's fix, according to the post, rests on three changes:

  1. Request-scoped keys. The request_id is always embedded in the cache key regardless of the isolation flag, with keys prefixed req:{request_id} to guarantee uniqueness across requests.
  2. A generation fence. Each request carries an epoch counter, and any slot whose epoch predates the current request is treated as stale and discarded. This allows fast invalidation without flushing the entire pool.
  3. A defensive TTL. KV page expiry is forced to at least 30 seconds, computed as the maximum of the user-supplied TTL and 30, so eventual consistency can settle before any buffer is reused.

The post closes with a Python blueprint implementing the corrected key schema and epoch-checked reads. It is worth noting that the sample imports what the author calls a hypothetical SDK, so it illustrates the pattern rather than shipping production code.

What to keep in mind

This account rests on a single blog post. No corroborating advisory from Cloudflare or the SGLang maintainers appears in the material available to this digest, and the post's own details invite caution: it is dated November 2026 while describing a late-2024 disclosure, and it attributes the GLM model to Meta, even though the GLM family is generally associated with Zhipu AI. Treat the specifics as one team's engineering account rather than a confirmed vulnerability record.

Why it matters

Cross-request cache leakage in multi-tenant LLM inference is a severe class of bug. KV caches hold per-token key and value tensors computed directly from user prompts, so anything that exposes them to another tenant can reveal prompt content without touching logs or network captures. The timing side channel described, inferring embeddings from fetch latency, shows that even partial exposure can be exploitable. The remediation pattern is also portable well beyond this incident: scope cache keys to the request, fence buffer reuse with an epoch or generation counter, and enforce a minimum TTL wherever an eventually consistent store backs a recycled pool. Finally, the trigger, a throughput-oriented flag combined with a zero TTL, is a reminder that isolation guarantees often live in configuration rather than code, and a deployment script can silently switch them off.

  • #ai
  • #security
  • #llm-inference
  • #kv-cache
  • #cloudflare-workers
  • #sglang

Related posts