deniz.in

Markets

Weather

Loading weather

· via GitHub Blog

GitHub reports five August incidents led by 10-hour Actions outage amid Azure migration

GitHub's August 2026 availability report counts five incidents, details a 10-hour-42-minute Actions outage and an authentication failure, and logs Azure migration milestones.

GitHub reports five August incidents led by 10-hour Actions outage amid Azure migration

GitHub recorded five incidents that degraded performance across its services in August 2026, according to its monthly availability report on the GitHub Blog. The company described August as a difficult month for availability and said it uses incidents to reshuffle priorities, while acknowledging that risk cannot be fully eliminated even as it invests in architectural improvements and its move to Azure. The report gives detailed postmortems for two incidents and summarizes remediation across databases, Actions and monitoring.

The August 6 Actions outage

The longest incident described began at 15:22 UTC on August 6 and lasted 10 hours and 42 minutes. A routine deployment to an internal GitHub Actions service that converts incoming events into workflow jobs briefly reduced pod capacity at one site. GitHub rolled the deployment back to confirm its contents were not the cause; the issue was the temporary capacity loss itself, which pushed the remaining sites past their limits as traffic shifted to them.

The affected services were already running close to their capacity and concurrency limits, so the smaller pod count was enough to exhaust headroom. Service mesh sidecars hit CPU throttling and out-of-memory restarts, and the failures cascaded into cache, DNS and API errors across multiple clusters. At the worst point, a large share of Actions workflow runs failed to start or complete.

A latent bug in the job-assignment path slowed recovery: runners were handed jobs that had already been revoked, then kept retrying them instead of picking up valid work, compounding the backlog. GitHub declared a public incident, expanded capacity for the saturated services and throttled incoming webhook-triggered work. It then shipped fixes so runners stopped acquiring invalid jobs, drained the queues and raised internal rate limits that were slowing recovery. Some self-hosted runners had to be recovered manually, and certain events from the incident window could not be replayed automatically.

Follow-up work includes adding headroom and autoscaling for the service mesh ingress and the affected Actions services, changing deployments so rollouts no longer cut capacity, monitoring the saturation and database-proxy conditions that preceded the incident, and automatic recovery for self-hosted Actions Runner Controller runners in upcoming runner and ARC releases.

The August 17 authentication incident

A second incident began at 13:40 UTC on August 17 and ran for 7 hours and 35 minutes. A new traffic peak pushed one datacenter's load balancers past their limits, and a service-mesh sidecar hit its concurrency limit without scaling up. As requests backed up, several load-balancer nodes exhausted their network flow limits, degrading the shared gateway authentication path and causing widespread authentication latency and failures across services routing through that datacenter. The report also points to a latent client retry bug that sharply amplified traffic to an internal authentication service.

Azure migration milestones

On August 11, GitHub ran a production MySQL primary from Azure for the first time, with minimal client-observed write impact and no customer impact during the transition. It repeated the exercise with two more primaries on August 27 and has further failovers scheduled, increasing in complexity. Read traffic also hit new highs: reads from migrated services peaked at 60.4%, reads from GitHub's monolith peaked at 64.3% in Azure, and Git reads reached 54%.

Database, Actions and Git capacity work

The 24-table authentication-core cohort moved off mysql1, GitHub's oldest shared database, removing roughly one million queries per second from its replicas. Separate query-hygiene changes removed another 120,000 queries per second and eliminated about 59,000 seconds of wasted database work per hour.

For Actions, job-routing changes moved 33% of jobs from a constrained production cluster to spare capacity, cutting peak cache CPU utilization from 98% to 80% and buying an estimated three months of headroom. GitHub frames this as a stopgap, noting the August outage reinforced the need for more durable capacity and isolation. Pull request isolation work continued, with authenticated reads for the first production cohort reaching 100% after unauthenticated traffic was already being served.

Git overload protection served 6.4% more traffic while improving 95th-percentile duration by 24% and maximum delay by 78%. New load-shedding levers at the edge were in fact used to mitigate the August incidents.

Monitoring and detection

Pull request monitoring now tracks merge, review and comment failures separately, so heavy read volume can no longer hide a failing write path. On August 21, automated high-impact incident detection started combining customer-support signals with service telemetry, and API monitoring was recalibrated and validated over 30 days to cut noise. Coming next: more database primary moves, continued migration of services and traffic to Azure, more capacity-management automation and auto-scaling, and broader dependency-failure handling across the pull request experience. The ordering GitHub says guides all of it: availability first, then capacity, then features.

Why it matters

GitHub is load-bearing infrastructure for a large share of software development, and Actions outages directly block builds and deployments. The August 6 postmortem is a textbook case of services running too close to their limits being tipped over by a routine rollout, with latent bugs amplifying the damage during recovery. The report also offers unusual visibility into what it takes to migrate a platform of this scale to Azure while usage keeps growing, which is useful context the next time a workflow fails to start.

  • #github
  • #github-actions
  • #outage
  • #azure
  • #reliability