GitHub's Project Rubicon rebuilds Git storage on reference compare-and-swap, tested at 5,000 agents

What GitHub announced

On 5 November 2026 GitHub published Project Rubicon, the storage design behind its rebuilt Git architecture, according to a dev.to write-up of the company's engineering post. In internal testing the system handled 5,000 concurrent agents working against a single repository while sustaining 7,200 pushes per second. Those figures come from a limited private preview, with a broader rollout expected early next year and six further optimizations still on the list.

The design leans on a property of Git itself: objects are immutable and named by content hash, so two writers can store the same object at the same moment without coordinating. The only part of a push that genuinely needs agreement is the reference update — the pointer saying which commit a branch names. GitHub's stated principle is that reference updates are the sole thing requiring consensus, and that record is tiny compared with the pack carrying the content.

Splitting packs from references

Rubicon divides storage into two layers. Pack data — the compressed Git content — goes to Azure Blob Storage, while reference state lives in Ref-CAS, a replicated log service GitHub built itself. A virtual replica handling a push indexes the incoming pack, uploads it to Blob Storage first, and only then commits one transaction to Ref-CAS that lists the pack in the manifest and moves the references. Because a reference is never committed before the objects it points at are durable, recovery reduces to replaying pointers to data that already exists.

Above the data plane, Spokes — GitHub's existing routing, load-balancing and tenant-partitioning layer — is unchanged. Each repository maps to a small group of virtual replicas whose local disks are pure cache and can be rebuilt at any time. Since correctness never depends on which replica serves a request, GitHub can add replicas to a hot repository without a migration.

Two designs that failed first

The dev.to article documents two earlier attempts that shaped Ref-CAS. The first kept a write-ahead log for every reference in a repository, indexed by a single blob, with each update written conditionally against that blob. It was correct but capped throughput in the mid-hundreds of writes per second, because only one conditional write to a blob can be in flight at a time.

The second attempt split references across segment blobs by hash so that writes to different segments stopped waiting on each other. That lifted the write ceiling but forced every read to check each segment, leaving latency bounded by the slowest one. Since reads outnumber writes by orders of magnitude on busy repositories, the trade went the wrong way.

Ref-CAS is the third answer: effectively a small consensus group dedicated to reference state. A transaction locks only the references it touches, verifies each still holds the value the writer expects, and becomes visible once a majority of the log's replicas record it. Because agents mostly push to their own branches, most updates replicate in parallel and never contend.

Reads cost one version check

Every Ref-CAS commit carries a log sequence number, and the sequence number of the latest commit serves as the repository version. A read begins by sending the version the replica already holds; if it is current, the reply carries no reference data at all. An unchanged repository therefore costs a single comparison — significant because GitHub shows one repository's daily requests growing roughly 4x between March and May 2026, driven by CI runs, deployment systems and other agents fetching the new commit, and the design plans for at least two orders of magnitude more scale than today's busiest repositories carry.

Compaction moved to background nodes

Every push adds a pack, and each extra pack makes lookups slightly more expensive, so the pack count cannot grow without limit. The legacy design ran compaction on the same replicas that served traffic. Rubicon moves that work to dedicated background nodes that read and write durable storage directly, coordinated through leases in Blob Storage so only one node touches a shard. A node collects new packs into a segment, seals it at a limit set by count, size or age, merges the packs, and checks that the output contains exactly the same objects as the inputs before uploading, updating the manifest and removing the old packs in a second transaction. That verification step carries the design's weight: publishing a pack that dropped a single object would corrupt reads for every replica.

Why it matters

Agentic coding workflows push far more frequently than human developers, and each push drags read amplification behind it. Rubicon is GitHub's answer to that workload: a consensus-based log for the small mutable part, cheap content-addressed storage for everything else, and version-checked reads that make an unchanged repository nearly free to serve. The published numbers are internal and preview-stage, but the architecture — separating what must be agreed from what need not be — is a pattern any system facing high-frequency writes plus read-heavy fan-out can borrow.