GitHub reports two September incidents: database saturation and silent Copilot review failures

Two incidents, two very different failure modes

GitHub has published its availability report for September 2026, disclosing two incidents that degraded performance across its services: a roughly 88-minute disruption on September 13 triggered by a background cleanup job exhausting a shared database, and a 52-minute failure on September 28 in which some Copilot code reviews completed without producing any output. The report, which also covers August because no separate edition ran that month, frames both events alongside wider reliability work.

A cleanup job saturated a shared database

The first incident began at 09:16 UTC on September 13 and ran for one hour and 28 minutes. According to GitHub, a background data-cleanup job included a brake meant to slow its writes whenever database replicas lagged — but the brake never watched load accumulating on the primary server. The job therefore kept writing until the primary hit its connection limit, and requests across dependent services then failed or slowed.

GitHub identifies two factors that made things worse. The database calls had no quick timeout, so request handlers hung on a stalled database instead of failing fast, and a retry loop around token creation kept resending writes that were already failing, holding the database at saturation. Automated monitoring caught the incident; responders cut internal load, paused the job, and error rates returned to normal.

The follow-up work includes throttling background processing against actual database capacity, cutting redundant retries during failures, bounding the blast radius of slow database requests, and hardening service startup so brief connectivity problems do not undermine recovery.

Copilot code review reported success with no results

The second incident started at 21:16 UTC on September 28 and lasted 52 minutes. Copilot code review delegates reviews to a separate service that returns two copies of the results: a limited version without review comments and a full version containing the completed review. Reviews are posted only from the full copy. A change to that service began delivering only the limited copy for some customers, which meant the review could not be assembled — yet the job was reported as successfully finished.

Because nothing monitored for missing results, the problem surfaced only when some reviews sat incomplete for more than 30 minutes and tripped an alert to first responders. GitHub reverted the change, posted to its status page while the revert cleared its checks, and told customers to request the affected reviews again. Planned fixes include delivery safeguards across the review pipeline, automatic recovery for failed reviews, earlier notification of problems, and tighter monitoring and rollout controls.

Azure migration advances but misses its deadline

The report also recaps the ongoing migration to Azure's Central US region. By October 7, GitHub says, 120 of 160 production services were deployed there and 13 of 51 primary MySQL databases had moved. Azure was serving 65.4% of read traffic for the core application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic — percentages that measure distinct workloads and should not be summed.

The migration hit friction. GitHub paused Central US deployments and traffic increases on September 11 because capacity was spread unevenly across availability zones, resuming permanent ramps on September 18 after temporary relief, with full capacity targeted for the end of October. The company also missed its aggressive September 30 goal of deploying every production service in the region. Separately, during a planned database failover, an automated emergency process disconnected Central US database copies and disrupted customers in that region; GitHub has disabled that process during planned handoffs while it improves recovery.

There was measurable progress elsewhere. Twenty-four authentication-core tables moved off GitHub's oldest shared database, lifting roughly one million queries per second from its replicas. The cluster behind issues and pull requests shed about 980 million unnecessary edit-history rows, freeing an estimated 1.8 terabytes, and weekly deadlocks fell roughly 77% from their June peak.

Actions headroom and overload controls

For GitHub Actions, production changes cut measured Redis operations by about 28% and relative CPU by about 30%. A separate infrastructure cache reduced Redis bandwidth by 84%, average response time by 66%, median response time for the primary inventory request by 96%, and queued requests by 95%.

Roughly half of sampled pull request traffic can now keep operating in a reduced mode when a shared dependency fails, and database traffic routing scales automatically under load — one group grew from 14 to 19 instances during a September 23 spike in pull request activity. A new overload control lets responders drop lower-priority requests to protect key API and web workflows before the system saturates.

Why it matters

The two September failures are textbook distributed-systems lessons: a safety mechanism watching the wrong signal, missing timeouts, a retry storm, and a success path with no monitoring to notice empty output. GitHub's report is unusually candid about the fixes and the setbacks, including a missed migration deadline and a failover that hurt customers. For any team operating shared databases or async pipelines, the remedies — capacity-aware background jobs, fail-fast calls, and alerts on jobs that succeed but produce nothing — are directly applicable.