False smoke alarm shut cooling and armed sprinklers at Jane Street's NYC datacenter

Michael Bacarella, a former member of Jane Street's systems team, has published a first-person account of the night a false smoke alarm nearly flooded the trading firm's only New York City datacenter. Construction dust tripped a smoke detector, which automatically switched off the machine room's cooling and pressurized its overhead sprinklers — starting a race to unplug every server by hand before the heat itself opened the sprinkler heads. The post appeared on Bacarella's personal blog and was featured on Hacker News's front page.

Why the machines lived in an office tower

According to the account, Jane Street long resisted moving its computers away from the trading floor, housed at 1 New York Plaza in lower Manhattan. What began as a small machine room grew into a substantial facility: roughly ten rows of server cabinets, ten cabinets deep, an online UPS, computer room air conditioning (CRAC) units, oversized network switches and a below-ground diesel backup generator — all inside an otherwise ordinary office skyscraper.

Bacarella gives two reasons for the arrangement. One was fear of intellectual property theft: the firm worried that servers removed from the building could expose its trading strategies. The other was control: engineers wanted the option of crossing the hall and physically cutting power to runaway automated trading systems. He credits the firm's risk culture partly to a partner who had served on a nuclear submarine, whose guiding rule was to distrust automation unless a mechanically connected, human-reachable shutoff existed.

How a false alarm armed the sprinklers

The trigger was mundane: contractors cutting into materials in a prep room next to the datacenter raised enough dust to set off the smoke detector.

As Bacarella explains, the sprinkler setup works in stages. The pipes above the room normally hold no water, which avoids problems from slow leaks developing overhead. When the smoke alarm fires, water is pumped into those pipes under pressure — but nothing sprays yet. Each sprinkler head opens individually once the air around it becomes hot enough, a threshold Bacarella puts at around 150°F. The dangerous wrinkle is a separate interlock: because circulating air would feed a real fire, the smoke alarm is wired to shut down the CRAC units. In a fully loaded server room with no cooling, the temperature climbs steadily — directly toward the point at which the freshly pressurized heads would open.

Unplugging a datacenter by hand

Alerted by a call from the firm's Hong Kong office — where someone had noticed server temperature readings rising across many machines at once — Bacarella, who lived a nine-minute walk from the building, ran there, rode the elevator up 33 floors and began pulling power cords, working down each row and unplugging cabinet after cabinet. He watched the temperature readout climb through 141°F and past 143°F before it finally flattened at roughly 143.6°F, just short of the trigger point.

The irony, he notes, is that the machine room sat beside the trading floor precisely so someone could always cut power in an emergency — though nobody had pictured doing it to protect the hardware from the fire-suppression system itself.

Aftermath and the monitoring gap

Because New York's major markets had already closed for the day, downtime was minimal and systems were running again the next morning. Bacarella describes the post-incident reviews as calm and constructive rather than punitive. The outcome: the sprinklers would stay, procedures and oversight would be strengthened, and servers would continue to be hosted in the office building next to the trading floor. That stance held, he writes, until Hurricane Sandy arrived — a sequel the post leaves for another day.

Detection was nearly as lucky as the response. No individual server was configured to alert simply for running a bit hot, since that happens routinely. The problem was spotted only because someone happened to see many machines reporting the same rising condition simultaneously. Bacarella writes that the next morning he set out to build a monitoring tool whose single job was to catch exactly that pattern: a simultaneous temperature rise across many servers indicating a systemic cooling failure.

Why it matters

The incident is a compact study in how sensible, isolated safety mechanisms can combine into a destructive chain. A smoke detector, a staged sprinkler design and an HVAC shutdown interlock each behave reasonably on their own, yet together they turned nuisance dust into a countdown toward drowning a datacenter. It also makes the case for last-resort human overrides that are physically reachable in time — the outcome hinged on one engineer living minutes away — and for fleet-wide monitoring, since correlated anomalies across many hosts reveal failures that per-host thresholds will never flag. For anyone running critical equipment in repurposed office space, the lesson is that the most dangerous failure modes are the ones that cross system boundaries.