Cloudflare reclaims 100TB of RAM by reworking consistent hashing in Pingora

What happened

Cloudflare has published an engineering deep-dive explaining how it reclaimed more than 100TB of RAM across its global network by making small changes to a single algorithm: the consistent hashing implementation inside one of its Pingora-based services. According to the Cloudflare blog, the savings come on top of a separate 100TB reduction the company's DNS team achieved the month before.

The company frames its scale plainly: thousands of servers worldwide, petabytes of RAM and millions of CPU cores, with every service expected to run on every node. Resources that vast are still finite, and at that scale even one-percent-at-a-time improvements are worth celebrating. The change described here returned over 100TB of memory to the fleet.

The ticket that started it

The work began with a performance ticket filed by an engineer named Ivan, who reported that pingora-ketama, used inside Cloudflare's Pingora Backend Router, was consuming far more memory than expected. The Backend Router — abbreviated PBR — is Cloudflare's internal load-balancing service, and pingora-ketama is its open-source library for consistent hashing. The structures associated with that library were the source of the bloat.

How consistent hashing works

Consistent hashing distributes tasks across servers in a way that avoids large-scale reshuffling when servers join or leave. Cloudflare uses it in PBR to route cacheable requests to servers by URL, which keeps a single copy of each file per data center and provides a stable way to locate it.

The mechanism rests on a property of hash functions: they accept any input but produce a single unsigned integer — 32, 64 or 128 bits depending on the function. The blog skips the usual ring visualisation and treats the output space as a number line. Servers and tasks are both placed on that line by hashing representative values, such as IP addresses for servers and cache keys for tasks. Assigning a task is then simply a matter of finding the first server to its left; the range belonging to the last server wraps back around to zero, which is where the ring imagery originates.

The imbalance problem

Because hashes are essentially random numbers, the ranges servers cover come out unequal, and the fraction of requests a server handles is proportional to the size of its range. The post works through the statistics for the share of the line one of N servers covers: the expected value is 1/N, and the standard deviation is (1/N) times the square root of (N−1)/(N+1).

For 100 servers, the expected share is 1% and the standard deviation is roughly 0.99% of the total. Scaling the deviation by the mean gives the coefficient of variation, which at N=100 works out to about 99%. In concrete terms, some servers can end up handling about twice their fair share of requests while others do almost nothing.

Within consistent hashing's constraints, the remedy is more hashes. Each server is represented by multiple points on the line rather than one, and the many small segments average each other out — the law of large numbers at work. NGINX hardcodes 160 hashes per server as its baseline, and Pingora adopts the same value as its default. With 160 points per server, the 100-server coefficient of variation falls from about 99% to about 8%, which the post describes as a significant improvement.

The tension the post explores

Better balance is not free: every additional point per server must be stored in the ketama structures Ivan's ticket flagged, and PBR has to run on every node of a fleet measured in thousands of machines. The blog carries on from there, asking what happens as the number of hashes grows further, and promises lessons in both the probability behind the tuning and the Rust implementation that made the memory savings possible. The headline result is that small changes to this one algorithm shrank the service's footprint enough to hand back more than 100TB globally.

Why it matters

Consistent hashing underpins a large share of distributed systems — CDN caches, key-value stores, sharded databases — and the 160-point default inherited from NGINX is widespread. Few operators audit what that constant actually costs in memory when multiplied across an entire fleet, and Cloudflare's numbers show it can be enormous.

The savings are also concrete capacity rather than an abstraction: 100TB returned to the network is headroom for new services without new hardware, and arriving shortly after a comparable DNS-team reduction suggests a repeatable practice of hunting down structural waste.

Perhaps most transferable is the method: a performance ticket, a first-principles model of expected versus actual load, and a targeted algorithm change — engineering that generalises well beyond one edge network.