deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

A latent Microsoft bug and one FAA router: how two cascading failures unfolded

A dev.to postmortem traces how a dormant Microsoft 365 gateway defect and a misbehaving router in Arkansas cascaded into a day-long email outage and a nationwide ground stop.

A latent Microsoft bug and one FAA router: how two cascading failures unfolded

On November 12, Microsoft 365's email and collaboration services failed for roughly a day across dozens of countries. The following afternoon, the FAA held every departing flight in the United States for more than four hours. Officials have found no connection between the two events and no evidence of a cyberattack in either, but an engineering postmortem published on dev.to works through both incidents stage by stage and argues they share a familiar anatomy: a small local fault that the surrounding system amplified into a sweeping outage.

A bug that slept for three months

According to Microsoft's preliminary post-incident report, as covered by GeekWire, The Register and BleepingComputer, the email outage traces back to a change shipped in July to the networking gateway servers that front Microsoft 365. That change carried a defect which, under heavy load, triggered lock contention: threads piling up on a shared lock while consuming CPU and memory without doing useful work. Microsoft said the contention lived inside an internal Windows process and grew worse on newer machines with more cores.

The defect sat dormant in production for over three months because everyday traffic never pushed the gateways into the regime where contention runs away. The safety margin eroded shortly before the failure: about 14% of gateway capacity was already offline from a separate earlier problem, and roughly 15 hours beforehand Teams traffic had been shifted onto the Exchange gateway pool, per GeekWire. Microsoft says neither condition alone would have caused an outage; the combination tipped the gateways into the defect. The company also acknowledged that its load testing had relied on weaker hardware with fewer cores than production machines.

The cascade

When the first gateways locked up and stopped responding, automatic rerouting did its job and moved their traffic onto the remaining healthy servers, which then crossed the same threshold themselves. By the time the spread stopped, more than 80% of gateway servers were unresponsive. Microsoft resolved the outage by deploying an update that reverted the defective change.

Recovery brought its own problem: restarted servers were immediately swamped by every backed-up client reconnecting at once, the classic thundering herd pattern that GeekWire's summary names explicitly. The isolation meant to confine a problem to a single region also leaked, and the preliminary report does not yet explain why. Exchange Online and Outlook bore the brunt, with Teams, Copilot and SharePoint also affected; Morningstar DBRS estimated insured losses at $300 million to $500 million. Microsoft's final report is due November 20.

One router, national impact

The FAA event began between 3:30 and 3:36 p.m. Eastern on November 13, when controllers noticed inconsistent radar and radio feeds. At 3:54 p.m. the agency issued a nationwide ground stop, lifted around 8:15 p.m. Instability reached 17 of the 22 Air Route Traffic Control Centers that handle high-altitude traffic, plus several approach facilities, and FedScoop noted it was only the second nationwide stop since 2001.

The trigger was routine infrastructure work. L3Harris, which operates the FAA's telecommunications network, was replacing copper circuits with fiber at a site in El Dorado, Arkansas, and installed a new router there, a Juniper unit replacing an older Cisco device according to AeroTime. HPE, which owns Juniper, said the device behaved exactly as the installer configured it. The router then began emitting roughly 1,000 connection attempts every four to five seconds across the shared network, according to the FAA. Legacy equipment elsewhere could not absorb the flood, and radar and communications feeds to most en-route centers degraded. Whether the cause was a misconfiguration or a faulty unit remains undetermined.

Diagnosis was slow. FAA Administrator Bryan Bedford said the agency spent about 45 minutes hunting for the cause before halting flights, partly because this kind of maintenance is recorded by hand on paper, FedScoop reported. Isolating the El Dorado router, plus a second unit at a site in Amarillo, Texas, stabilized the network, after which the FAA disabled the suspect function on about 60 similar routers fleet-wide. The FBI is examining the router at an L3Harris lab as a precaution rather than as evidence of an attack, and the FAA's root-cause review is ongoing.

Why it matters

The dev.to analysis maps both incidents onto one pattern: a change introduces a trigger, degraded capacity or a shared medium hides the precondition, and the system's own protective instincts, failover and propagation, become the amplifier. The defenses it recommends are unglamorous: load-test with capacity deliberately removed, alert on shrinking headroom rather than only on errors, shed load instead of migrating all of it, use retry backoff with jitter, keep searchable change records, and actually test isolation boundaries. Both explanations are still preliminary, so details may shift. The broader lesson stands regardless: the next outage tends to land on a day when something else is already broken, and these defenses are far cheaper to put in place before that day than during it.

  • #microsoft-365
  • #faa
  • #outage
  • #reliability
  • #incident-analysis

Related posts