· via Hacker News – Front Page (native)
Microsoft blames November Exchange Online outage on software defect introduced in July fix
Microsoft's preliminary post-incident report says a July update to Microsoft 365 gateway servers carried a latent defect that left more than 80 percent of them unresponsive.

A latent defect from a July update
According to The Register, Microsoft has attributed last week's major Microsoft 365 email outage to a software defect introduced by an update it shipped in July 2026. In a preliminary post-incident report published on November 15, the company said the update targeted the network gateway servers that front Microsoft 365 services, and that it passed load testing and was deployed without any visible impact at the time.
The problem, Microsoft acknowledged, was that its load testing concentrated on lower-spec machines with fewer CPU cores. The defect triggered lock contention inside an internal Windows process, exhausting a machine's CPU and memory, and the contention grows worse on servers with more cores and newer hardware, precisely the machines that populate a production gateway fleet.
Two unremarkable events aligned
The defect stayed dormant until November 12. At 0107 UTC that day, Teams failed over to the primary gateway that serves Exchange Online, producing a load surge that Microsoft described as not unusual on its own. At the same time, close to 14 percent of the gateway's server capacity was out of service. Microsoft said that capacity was already being restored and that the service still had adequate headroom.
Together, the two conditions pushed the gateway into the defect. What followed was a textbook cascading failure: a server would stop responding, traffic would shift to the next healthy endpoint, that endpoint would hit the same resource exhaustion and fall over, and the cycle repeated. Within two hours the defect had been exercised across the fleet, and at the lowest point more than 80 percent of gateway servers were unresponsive.
Why Exchange Online bore the brunt
Microsoft said services such as Teams, OneDrive, SharePoint and the Office apps were able to fail away from the faulty gateway infrastructure. Exchange Online could not, for scale reasons, which is why Outlook users saw the longest disruption.
Patching the patch
Recovery was awkward. Bringing unresponsive servers back simply caused them to be overwhelmed and fail again, and Microsoft's automated recovery processes were not designed to restore service from such a broad unresponsive state. A clean rollback was off the table because the defective change had been running in production for months. Instead, engineers built a fresh update that restored the affected Windows code path to its previously known good state, then applied it to servers in batches until service gradually returned.
The timeline in the report shows how long that took. Microsoft's connectivity anomaly detection flagged rising failures at 1603 UTC on November 12, and the first support case arrived at 1633 UTC. Over roughly the next fifteen hours the team paused all ongoing and planned deployments and configuration changes, added capacity and rerouted traffic, without success. At 0030 UTC on November 13 they identified connection negotiation as the failing operation, with competing processes preventing connections from being handled. The mitigation was identified and applied at 0720 UTC, the service was confirmed healthy at 1610 UTC, and the final customer update went out at 0415 UTC on November 14. A final post-incident report is due on November 20.
Why it matters
As The Register points out, the uncomfortable part of this incident is that a change which sat dormant for months caused global disruption partly because load testing did not reflect hardware scenarios entirely within Microsoft's own control. For organisations that depend on Exchange Online, it is a reminder that a single latent defect, combined with an ordinary load shift and a routine capacity shortfall, can defeat both capacity planning and automated recovery in a hyperscale cloud. And when the defective change is too old to roll back, recovery becomes an improvised engineering effort rather than a routine operation, which is exactly what stretched this outage across the better part of two days.
- #microsoft-365
- #exchange-online
- #outage
- #cloud
- #reliability