Microsoft publishes rack-level specs for Maia 300 inference chip, targets 2027 production

Microsoft frames Maia 300 as a rack, not a chip

Microsoft has published rack-level performance numbers for Maia 300, the third generation of its in-house AI inference accelerator, according to a dev.to report covering the company's 12 October 2026 announcement. Scott Guthrie, executive vice president for Cloud + AI, described the product as a "complete intelligent system" aimed at long-context reasoning, latency-sensitive conversation and sustained multi-step agentic workloads. Microsoft said Maia 300 production deployments begin in 2027, when the Maia family also becomes the primary inference platform for its own products and scales toward multi-gigawatt deployments.

A multi-chiplet package with 288 GB of HBM4

The accelerator splits its functions across dies: compute chiplets are built on TSMC's N2P process, while memory-controller and I/O functions sit on separate dies chosen for bandwidth, connectivity and cost. Microsoft's Azure hardware team argues the separation lets each function use the process technology best suited to its role.

Memory is the headline at the package level. Eight HBM4 stacks deliver 288 GB of capacity and 18 TB/s of peak bandwidth inside a 950 W TDP. Against Maia 200, that works out to 2.5x the memory bandwidth, 33% more capacity and roughly twice the HBM bandwidth per watt, per the report. The package also moves to a locality-aware memory organisation across chiplets, so data sits near the compute that consumes it, and supports enhanced MXFP4 and sub-3-bit storage formats for inference.

SiliconANGLE, cited in the report, notes the capacity figure matches NVIDIA's Rubin accelerator while bandwidth lands about 20% lower — two vendors converging on capacity but diverging on bandwidth per chip.

Ninety-six accelerators in a 167 kW rack

The rack is the unit Microsoft chose to publish. Each holds 96 accelerators across 24 four-accelerator trays, with host processors, memory, storage and cooling folded into each tray — the change that doubles accelerator density over Maia 200, since separate host trays disappear from the floor plan. The rack carries 27.6 TB of HBM (27.7 TB in the deep dive's rack table, a minor inconsistency the report flags) and 1.7 PB/s of aggregate memory bandwidth within a 167 kW envelope.

Microsoft's comparison table spans pod sizes from 48 to 96 racks, with rack HBM bandwidth from 341 to 1,728 TB/s, HBM capacity from 10.4 to 27.7 TB, FP4 performance from 485 to 1,324 peta-operations per second, and rack power from 64 kW to 167 kW. Read together, rack HBM bandwidth grows by a factor of 5.1 between generations while rack power grows by 2.6, or about 1.95x the bandwidth per watt. Liquid cooling carries more than 80% of the rack's thermal load, and the design connects either to facility water or through a heat exchanger for air-cooled buildings — a choice that determines which facilities can host the capacity at all.

Ethernet superpods and a programmable bandwidth split

Scale-up networking is where Maia 300 departs furthest from a conventional GPU cluster. Microsoft's AI network controller runs a custom AI transport over Ethernet in a two-tier topology: a half-switched electrical arrangement inside the rack, where each accelerator links directly to a tray partner and through switches to the other 94, and an optical tier that joins up to 128 racks into a 12,288-accelerator superpod.

The split between in-rack and cross-rack bandwidth is not fixed at design time; job packing sets it at run time, and an accelerator can drive up to 750 GB/s to a peer elsewhere in the pod. Tensor parallelism can stay in-rack while expert and pipeline parallelism span racks, and schedulers can absorb a failed tray using hot-spare capacity in a neighbouring rack. The report characterises this as a scheduling capability expressed as a network design: capacity that would once sit stranded inside a rack boundary becomes usable by jobs packed elsewhere, raising effective utilisation of the same silicon.

The performance claims and their baselines

Microsoft says Maia 300 should deliver 40% more tokens per megawatt than "leading GPU-based systems" — an unnamed cross-vendor baseline — and up to 7x the rack-scale token throughput of Maia 200 in a 96-accelerator configuration. Maia 200, already running production workloads including Microsoft Copilot across the United States, Europe and Australia, is credited with up to 1.7x more tokens per watt than leading GPU infrastructure for built-in AI in Teams, Copilot and Cowork. GitHub Copilot and Microsoft Foundry follow on Maia 200 by the end of 2026, alongside OpenAI models, Microsoft's own MAI models and leading open-source model families.

Why it matters

The two halves of the efficiency claim carry different weight. The generational ratios — 7x rack throughput, roughly 2x bandwidth per watt — are checkable against Microsoft's own published Maia 200 figures, while the 40% tokens-per-megawatt claim rests on an unnamed GPU baseline that will be hard to verify independently. The framing itself is the signal: Microsoft is competing on tokens per megawatt and accelerators per rack, the metrics that actually decide datacenter capacity. For operators, power envelopes and cooling options now determine where capacity can land. For NVIDIA, a first-party hyperscaler rival is putting efficiency claims directly against its hardware at exactly the moment when power, not silicon supply, is becoming the binding constraint on AI inference.