Cloudflare's GLM-5.3 leaked customer data via KV cache in 155 Workers AI responses

Cloudflare has disclosed a cross-tenant data exposure in one of its Workers AI models, caused by a sizing mismatch between a model's context window and its KV cache pool, combined with a bug in the open-source SGLang inference engine. According to a post on the Cloudflare blog, 155 responses served by the GLM-5.3 model between August 27 and September 9, 2026 contained data from other customers' requests.

The issue was reported on September 8, 2026 by Gadi Evron of Knostic through Cloudflare's HackerOne program: very long prompts sent to the model sometimes returned text belonging to other users' sessions. Cloudflare reproduced the problem the following morning and took the model offline within 40 minutes of declaring an incident, restoring it later that day with a corrected configuration and a patched engine. The company confirmed that no other Workers AI model was affected, and says that after excluding its own testing and the researcher's traffic, the affected requests look like ordinary long-context use rather than deliberate exploitation of the bug.

Two overlapping failures

The first issue was a misconfiguration. Cloudflare deployed GLM-5.3 — a Z.ai text-generation model aimed at agentic coding workflows — with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs only had room for 681,280 tokens. Cloudflare knew about the mismatch before launch, but expected the engine to return an error for any prompt it could not handle, and its long-context benchmarks had returned successful responses.

The second was a bug in how SGLang handled over-length requests in disaggregated prefill/decode mode. A prompt that did not fit the pool was marked as rejected and swapped for a single-token placeholder, but instead of aborting and returning a client error, prefill computed the KV cache for that placeholder and shipped it to the decode node. The decode node received a completion signal and assumed the full prompt's cache had overwritten whatever was there before; in fact most of the previous state remained, and the model generated its answer while attending over stale cache pages that still held data from earlier requests.

Why the bug was possible

Workers AI runs GLM-5.3 with SGLang in a disaggregated configuration: prefill GPUs process the prompt and produce its KV cache, then hand it to decode GPUs over RDMA via Mooncake. The KV cache is a shared pool of fixed-size GPU memory pages, assigned when a request is admitted and released when it finishes. Pages are not zeroed between requests — the next request simply overwrites them — an approach Cloudflare describes as common and necessary for performance, but which makes the engine responsible for tracking page ownership across nodes.

Notably, the pool size is not a direct setting: SGLang derives it at startup from whatever GPU memory remains after loading model weights, and nothing forces it to agree with the configured context length. Cloudflare points to its testing gap here — it validated HTTP status codes and throughput at long contexts, but not the coherence of model output. It now checks KV pool size against context length directly on the running engine before any disaggregated deployment change ships.

Remediation and upstream fixes

Cloudflare applied two fixes. First, it adjusted the GLM-5.3 configuration to match its GLM-5.2 setup, freeing enough GPU memory that the prefill KV pool now holds roughly 2.04 million tokens — about twice the context length — so a valid prompt always fits and the rejection path is never exercised. Second, it patched the engine so that a request carrying a pending rejection is retired at the disaggregation admission queue, with its original error, before any allocation or KV transfer occurs; the decode node also verifies that the number of transferred tokens matches what it reserved.

Cloudflare proposed these changes upstream as SGLang pull request #38704. A fix had already landed in SGLang's main branch as #36988, merged September 6 for a related failure path but not yet released at the time, and maintainers also merged #38935, which rejects such requests earlier. The bug is addressed in SGLang v0.5.21. Cloudflare validated its fixes against the researcher's reproduction script and its own canary-marker tests before restoring the model, and published the report alongside a coordinated disclosure from SGLang.

Why it matters

This incident highlights a failure mode specific to shared LLM serving: a standard performance optimization — reusing non-zeroed KV cache pages — turned a configuration mismatch plus a rejection-path bug into a cross-tenant confidentiality breach. As multi-tenant inference platforms become the default way businesses access models, the lessons are concrete. Internal capacity invariants like pool size versus context length need runtime verification against the running engine, not inference from HTTP response codes. Testing needs to check output correctness at boundary conditions, not just throughput. And because SGLang is widely used, the upstream fixes matter well beyond Cloudflare's own infrastructure.