Zinc whiskers under raised floors can take down multi-megawatt AI clusters

A maintenance check that stopped training runs

According to a technical write-up on dev.to, a scheduled inspection beneath the raised floor of a 2-megawatt AI training cluster ended with hardware failures across the cluster roughly ninety minutes later. The facility described runs 128 racks of liquid-cooled servers, each holding eight NVIDIA H100 GPUs drawing up to 700 watts per accelerator. When technicians lifted the access tiles, they noticed a metallic sheen on the zinc-plated steel pedestals and stringers supporting the floor. Once airflow and compute load were restored, nodes began logging intermittent short-circuit events on PCIe backplanes and power distribution boards. The symptoms, non-reproducible ECC errors, GPU resets and abrupt job terminations, disrupted multi-week large language model training runs.

Particle analysis of the failed boards, as reported in the article, found zinc filaments between 0.5 and 2 millimeters long with diameters of 1 to 5 micrometers. The whiskers had grown on the electroplated underfloor hardware over several years, driven by compressive stress from floor loading and humidity fluctuations, and were mobilized when the plenum was disturbed.

How zinc whiskers form

Legacy raised-floor systems commonly rely on galvanized or zinc-electroplated steel panels and stringers, with coatings 5 to 15 micrometers thick. Plating, cutting and stamping leave residual compressive stress in the zinc layer, and over time that stress drives the spontaneous extrusion of whiskers: single-crystal filaments that grow perpendicular to the surface from defects and grain boundaries. The dev.to piece notes that nucleation can begin within months of installation, but filaments generally need years to reach problematic lengths.

Underfloor conditions accelerate the process. Relative humidity of 40 to 60 percent facilitates zinc ion migration, vibration from CRAC units, PDUs and server fans repeatedly works the stressed lattice, and temperature cycling between roughly 18 °C and 32 °C adds micro-stress through differential expansion between the coating and the steel substrate. The article cites growth rates that can exceed 0.1 millimeters per year and densities of several hundred whiskers per square centimeter on heavily affected panels. Electroplated coatings, common in cost-optimized legacy panels, are said to whisker more readily than hot-dip galvanized ones because of their finer grain structure and plating additives that create nucleation sites. In floors that have been in service for 15 to 25 years, filaments frequently pass the 1-millimeter mark at which they can bridge adjacent conductors on circuit boards or inside power distribution units.

Why AI workloads make it worse

The article's central argument is that AI infrastructure has amplified an old failure mode. Rack densities of 60 to 120 kW per cabinet, achieved when liquid-assisted rear-door heat exchangers supplement air cooling, force cooling systems to move far more air through the plenum. The write-up cites CRAC and CRAH units exceeding 3,000 CFM each against 1,200 to 1,800 CFM in legacy designs, with underfloor pressure differentials rising from a historical 0.02 to 0.05 inches of water column to 0.08 to 0.12 inches beneath perforated tiles serving AI rows.

In the incident described, each GPU server pulled roughly 200 CFM through its intake fans, producing localized velocities above 400 feet per minute at perforated tile outlets. Those shear forces stripped further whiskers from the zinc surfaces and, helped by the pressure differentials, pushed sub-millimeter particles through cable cutouts and into the front intakes of adjacent racks. Inside the servers, the conductive filaments bridged fine-pitch traces on GPU daughter cards and voltage regulator modules, creating transient low-resistance paths that tripped protection circuits. Failures clustered during peak training epochs, when fan speeds and power draw were highest, and post-incident borescope inspections showed visible depletion zones on zinc-coated supports near the highest-velocity tiles.

Remediation and persistence

Recovery reportedly required systematic replacement of affected boards, installation of conductive-particle filters on all underfloor supply paths, and substitution of zinc-plated components with powder-coated or stainless-steel alternatives. Airborne-particle counters placed at rack intakes recorded elevated metallic particulate counts for weeks after the initial event, underlining that the contamination mechanism persists until the source material is fully addressed.

Why it matters

AI buildouts are frequently housed in facilities whose raised-floor systems were never designed for modern airflow volumes, and this failure mode stays invisible until maintenance or a load spike mobilizes years of accumulated whisker growth. The symptoms look like ordinary hardware flakiness, which makes root-cause analysis slow and expensive while training runs keep dying. The practical takeaways from the write-up are concrete: inspect plenum hardware for whisker growth, monitor metallic particulate at rack intakes, treat underfloor maintenance as a contamination event, and prefer non-whisker-forming coatings when retrofitting floors that serve high-density AI rows.