Cloudflare did not buy another 100 terabytes of RAM. It found that capacity hiding inside a data structure.

The company says a pair of changes to the consistent-hashing code used by an internal Pingora service recovered more than 100TB of memory across its global fleet. One change squeezed each routing point from eight bytes to six. The other cut the number of points by 90 percent after the team proved that most of them were no longer buying useful accuracy.

Neither idea is exotic. The interesting part is what happens when ordinary layout choices are multiplied across many hash rings, thousands of machines, and every cacheable request on a large network.

At Internet scale, a six-byte record can become a fleet-level capacity project.

Why the ring got enormous

Cloudflare's Pingora Backend Router uses consistent hashing to send a cacheable URL to a stable machine inside a data center. Stability matters because moving a request to a different machine can mean missing the cached copy and going back to the origin. Add or remove a server, and a good hash ring moves only the slice of traffic that needs to move.

A basic ring gives each server one point in a fixed integer space. Requests go to the next server point on that line, wrapping around at the end. That is simple, but random spacing can leave one server with a much larger share of the line than another.

Implementations smooth the imbalance by giving every server many virtual points. Pingora inherited a default of 160 points per server, then multiplied that count by a weight so machines with more storage could receive proportionally more traffic. Cloudflare also needed separate rings for combinations of caching features and compliance constraints. The result was not one tidy circle. It was dozens of large rings, some consuming roughly 6GB of process memory.

That is the first useful lesson: reasonable local defaults can become unreasonable when dimensions multiply. A point count that looks cheap in one ring becomes expensive across weighted servers, feature combinations, and the full deployment fleet.

Six bytes, not eight

The first improvement was a data-layout audit. Each ring point stored a 32-bit hash and a 32-bit index into a separate server array. But the router was not going to coordinate more than 65,535 servers at once, so the index only needed 16 bits.

struct Point {
    hash: u32,
    index: u32,
}

// The compact representation stores the same information.
struct PointV2([u8; 6]);
Changing the index type alone did not help because Rust padded the structure for alignment. Packing the fields into a six-byte array did.

A naive Rust structure containing a u32 and a u16 still occupies eight bytes. Its size must respect the alignment of its largest field, so the compiler inserts two bytes of padding. Cloudflare avoided #[repr(packed)] and its awkward unaligned references. Instead, the new point stores six raw bytes and exposes getters that reconstruct the integer values.

That saved 25 percent of the memory used by the point array. It is an old systems-programming move with a modern reminder attached: a language's memory-safety guarantees do not make its default layout free. When a structure exists millions or billions of times, checking size_of is part of performance engineering.

The bigger win came from deleting points

The team then questioned the 160-point baseline and the much larger weighted counts built on top of it. More virtual points make the work distribution smoother, but the returns diminish. Cloudflare derived an exact expression for the coefficient of variation when each of N servers receives k points:

CV(k) = sqrt((N - 1) / (N * k + 1))
Each large reduction in routing error requires roughly an order-of-magnitude increase in points.

In the example from Cloudflare's write-up, a server weight turns the 160-point baseline into 100,000 hashes per server. The final 90,000 points improve the predicted error by only about 0.7 percent. Worse, the implementation uses 32-bit hashes, so collisions become increasingly relevant at very high point counts. Beyond a certain point, adding entries consumes memory while introducing another source of noise.

With the math and simulation lined up, Cloudflare reduced the generated hash count by 90 percent without an appreciable loss of routing accuracy. This was the larger architectural win. The compact record made every point cheaper; the model showed that most points did not need to exist.

  • Measure representation: confirm what each entry costs in real memory, including alignment.
  • Model the objective: connect the number of entries to the actual balance error the system cares about.
  • Check finite limits: a 32-bit hash space does not behave like an infinitely precise mathematical circle.
  • Delete before tuning: the cheapest structure is still the one the service does not have to store.

A memory win can still become an outage

Changing the ring changes where some cache keys land. If Cloudflare had replaced every ring globally in one step, a large share of cached objects could have appeared to vanish at once. Origins would receive the resulting misses, turning a memory optimization into an avoidable traffic event.

The migration therefore carried both versions for a while. The old ring and the smaller ring lived side by side, and a stable decision based on the request hash chose which one would route each request. That supplied a clean rollback path without redeploying the service.

Cloudflare also separated two rollout controls: how much traffic used the new ring, and which data centers were allowed to use it. The team started in small validation locations, expanded to larger groups, and watched backend-selection traces, ring-version counters, connection errors, process memory, startup time, cache behavior, and origin traffic. Only after the new path reached 100 percent did it remove the old rings and realize the full memory drop.

This is the part that turns a clever optimization into production engineering. The code change may be six bytes wide, but the safety case includes cache locality, rollback, observability, and geographic blast radius.

The result is reusable

The implementation is now in Cloudflare's open-source pingora-ketama crate behind a v2 feature. It includes the compact point representation, a faster sorting path, and a configurable base point count. The original v1 ring remains available, and callers can construct both and choose between them per request while migrating.

That compatibility matters. A new algorithm is easier to adopt when teams do not have to bet the cache in one move. The library exposes the improved machinery, but it also preserves the old behavior long enough for users to prove equivalence under their own traffic.

The takeaway

The headline is 100TB, but the transferable pattern is smaller. Start with the profiler. Inspect the representation. Ask whether the algorithm's default still matches the scale and error budget of the system that now depends on it. Then treat the rollout as part of the optimization, not as the ceremony after it.

Cloudflare's hash rings grew because each decision made sense in isolation: more points for smoother balance, weights for different machines, separate rings for different capabilities, and a straightforward Rust structure for every point. The waste appeared in the multiplication.

Finding it required both low-level and high-level thinking. Two bytes disappeared from a record. Ninety percent of the records disappeared from the ring. A dual-path migration kept those disappearances from reaching the origin as a traffic spike. That is how a tidy bit of math becomes 100TB of usable infrastructure.

Sources