GitHub's Project Rubicon rebuilds Git storage for thousands of concurrent AI agent pushes

A Git storage layer built for thousands of agents

GitHub has published details of a rebuilt Git storage architecture that it announced at GitHub Universe, and the headline numbers are aimed squarely at AI-driven development. According to the GitHub Blog, internal testing showed the system supporting 5,000 concurrent agents working on a single repository and absorbing 7,200 pushes per second to it, while read traffic continued to be served with consistently low latency.

The company calls the design Project Rubicon, a name chosen, GitHub says, because agentic development is a threshold the industry has already crossed. Work began in May 2026, shortly after GitHub moved 99.99999% of repositories into Azure, and the new layer was built underneath the platform while it kept running. GitHub positions it as the foundation for supporting agentic development for hundreds of millions of developers. For users, nothing about the interface changes: pushes, fetches and branches behave the same way, with the same protections in place.

Agent traffic changes what a busy repository looks like

GitHub says it thinks about scale along two axes. Horizontal scale, meaning many developers and agents across many repositories, is already handled by battle-tested techniques such as tenant partitioning, traffic fairness enforcement and load-aware routing, all of which carry forward. Vertical scale, meaning many writers on one repository, is where agents change the picture, and most of the new requirements came from that axis.

Today, the busiest repositories on GitHub.com receive at most dozens of writes per second, and typically only in short bursts. GitHub expects parallel, round-the-clock agent work to turn those bursts into a sustained rate within the next year. Each push also multiplies into reads: CI systems run against the new commit, deployment tooling pulls it, and other agents fetch it for context. According to the post, daily requests to one repository grew roughly fourfold between March and May and have continued climbing exponentially since.

Rather than planning for the next increment, GitHub set a target of at least two orders of magnitude more scale. That translated into concrete requirements: thousands of writes per second on a single repository; multiple writer nodes, because indexing incoming packs and checking object connectivity are the most CPU-expensive parts of a write and a single node becomes compute-bound quickly; minimal commit latency, since an agent waits on each push before taking its next step; ordered changes per reference with fast linearization across references; cheap reads when nothing has changed, which is the common case; and components that scale independently.

Coordinate only on reference updates

The architecture rests on an observation GitHub laid out earlier in the series: very little of a push actually needs to be ordered. Git objects are immutable and named by their contents, so two writers can store objects at the same time without agreeing on anything. The only part requiring agreement is the reference update, which is tiny by comparison. Two principles follow from that: minimize coordination, and decouple storage from compute.

Inside Rubicon: virtual replicas, Blob Storage and Ref-CAS

At the edge, little changes. Spokes still handles routing, load balancing, the internal API the rest of GitHub uses to read and write repository data, and tenant placement and fairness. The data plane underneath Spokes is what is new.

Spokes routes each request to one of a small group of virtual replicas assigned to a repository. Virtual replicas are disposable compute that serve reads, writes or both, and their local disk is only a cache that can be rebuilt at any time. Routing prefers the same replicas to keep caches warm, but correctness does not depend on which replica serves a request, and busy repositories simply get more of them. A separate fleet of compaction nodes merges packs and removes unreachable objects in the background, never on a replica, so replica CPU and memory stay dedicated to serving requests.

Durable state lives in two places. Git objects, stored as packs along with read-acceleration data, sit in Azure Blob Storage. References and the pack manifest, the list of packs that make up a repository, live in Ref-CAS, a replicated log service named for Reference Compare-and-Swap.

A write is a two-step affair: the virtual replica indexes the incoming pack and uploads it to Blob Storage, then commits a single transaction to Ref-CAS that lists the pack in the manifest and updates the references. A read has the replica check its reference version with Ref-CAS, pull any packs it needs from Blob Storage into its cache, and serve from there. Compaction nodes later merge packs so reads stay fast.

Why it matters

Hosted Git has historically been sized for human cadence: dozens of writes per second in bursts. If agents keep writing code in parallel around the clock, the single busy repository, not the overall fleet, becomes the bottleneck, and GitHub has re-architected its storage specifically for that world. The design also concedes that concurrency is now largely machine-generated, and that pushes amplify into reads, which matters to anyone operating CI systems or mirrors that consume these repositories. Most notably for developers, the change is meant to be invisible: Git remains Git, while the substrate underneath gains two orders of magnitude of headroom.