Cloudflare prototypes Zstandard cache compression to save petabytes of storage

Cloudflare has described a prototype, called Cache Transcoding, that compresses eligible cached assets with Zstandard before they are written to disk. According to the Cloudflare blog, initial testing shrank eligible assets to roughly a third of their original on-disk size on average, trading a small CPU increase for what the company estimates as petabytes of effective extra capacity and reduced data transfer between its data centers.

The system was built during a 1.1.1.1 Intern Program internship and runs inside Pingora, Cloudflare's proxy framework. It targets a long-standing inefficiency: Cloudflare has conventionally stored whatever bytes the origin sent, so an uncompressed response was cached uncompressed and hauled between facilities in that same form.

How the prototype works

On a cache miss, the proxy encodes the response body with zstd before writing it to disk, and the cache metadata records the compressed representation while preserving the original content length. The object stays compressed for as long as it lives in the cache, and it also travels in that form between the upper and lower tiers of Tiered Cache. Decoding is reserved for the hop that faces the client.

An encoding marker on the stored object prevents double compression: a tier receiving an object from another tier can tell it is already zstd-encoded and keeps it that way.

What gets compressed

Eligibility is deliberately narrow. Media such as images, video and fonts is generally compressed already — in Cloudflare's traffic sample, that slice accounted for 21.4% of requests but 63.3% of bytes, so recompressing it would spend CPU for no benefit. Compressible text such as HTML, JSON, CSS and JavaScript made up 67.3% of requests and 22.3% of bytes, and roughly 71% of it arrived with no Content-Encoding set.

The prototype transcodes only 200 OK responses where Content-Encoding is unset, the Content-Type is compressible text, and a known Content-Length is at least 4 KiB. Range requests, slice subrequests, precompressed responses, active upstream compression, unknown-length bodies and binary content are all skipped. Cloudflare says the 4 KiB floor eliminated large numbers of tiny objects while excluding only about 1% of otherwise eligible bytes.

The CPU and storage trade

In Cloudflare's controlled test corpus, eligible assets compressed about 2.834x. Encoding cost roughly 4.31 nanoseconds per byte — about 232 MB/s — and is paid once when an asset enters the cache. Decoding cost roughly 1.56 nanoseconds per byte, about 641 MB/s, and happens on every serve. Because assets are served far more often than they are filled, the asymmetry works in the system's favor, and Cloudflare's model kept the extra CPU to a few percent under the traffic and reuse assumptions it tested.

The team initially considered restricting transcoding to popular content, since popular assets get reused more often, but found that decoding on every serve meant the restriction cut storage savings without a matching drop in CPU. The simpler policy — transcoding all eligible text at or above 4 KiB — delivered almost the entire measured storage gain while keeping CPU within budget. Both the compression level and the size threshold are tunable parameters rather than fixed limits.

Why Zstandard

Zstandard is a lossless compression algorithm developed by Yann Collet at Facebook and open sourced in 2016. Cloudflare's earlier browser compression testing found it compressed data 42% faster than Brotli at nearly the same file size, and produced files 11.3% smaller than gzip at comparable speed. The prototype uses level 3, which Cloudflare says provides most of the compression benefit without making cache fills CPU-bound; now that the initial CPU budget is understood, higher levels may be worth testing.

How it was validated

Correctness testing covered cache hits, misses, single-hop fills and Tiered Cache fills, with each request correlated across request logs, Prometheus metrics and Jaeger traces. A performance campaign sent more than a million requests across 10 cache servers, half with Tiered Cache disabled and half with it enabled.

Why it matters

Cloudflare frames the work against a backdrop of sharply rising RAM and hard disk prices over the past year. Compressed storage raises cache density on hardware that already exists, so fewer useful objects get evicted because an uncompressed form hogged space, and the smaller representations also ease backbone traffic between data centers. The economics are the point: a one-time encoding cost at fill keeps returning savings every time an asset is reused — a trade that looks increasingly attractive when storage, not compute, is the expensive resource.