Microsoft's Maia 300 deep dive: chiplets, HBM4 and 12,288-chip Ethernet superpods

A third-generation chip tuned for inference economics

Microsoft has published an architectural deep dive into Maia 300, the third generation of its in-house AI accelerator. According to the Azure blog post, which surfaced on the front page of Hacker News, the chip is the fullest expression of a design philosophy the company sums up as delivering more useful tokens per watt and more useful tokens per dollar. Rather than treating the accelerator as a standalone part, Microsoft describes a complete platform co-designed across software, silicon, rack infrastructure, networking and datacenter facilities, aimed at cloud-scale agentic inference: workloads built from multi-step execution, long-context decoding and tool use that change the arithmetic intensity of serving models.

Software manages the data, not just the math

Central to the design is SDLA, or Software Defined Locally Accessed Dataflow Architecture, introduced with Maia 200. Microsoft's argument is that storing, moving and converting data has become as costly as computing on it. SDLA therefore separates control-path programming from data-path execution, giving software explicit control over data placement, movement, conversion and synchronization across local memories, DMA engines, compute engines and the network. Maia 300 adapts the model to decode-heavy, long-context work in three ways: memory shifts from a uniform (UMA) organization to a locality-aware non-uniform (NUMA) one across chiplets, new narrow-precision formats gain hardware-accelerated type conversion, and the scale-up domain grows sharply in size.

Chiplets and a bandwidth-heavy memory hierarchy

Maia 300 is a multi-chiplet package. Compute sits on TSMC's N2P node, which Microsoft says offers the best performance per watt, while memory-controller and I/O functions move to separate dies chosen for bandwidth, connectivity and cost. Compared with Maia 200, the company claims this split yields 36% more compute FLOPs, 2.5 times the HBM bandwidth and 14% more scale-up bandwidth. Each accelerator pairs software-managed on-die SRAM with eight HBM4 stacks — 288 GB of capacity and 18 TB/s of bandwidth per chip — amounting to 33% more capacity and roughly twice the HBM bandwidth per watt versus the previous generation. On precision, Maia 300 introduces enhanced MXFP4 and sub-3-bit storage formats, shrinking footprints so larger models, longer contexts and more concurrent sessions fit within the same power envelope. The vector engine also gains ISA-level optimizations for inference kernels such as softmax and low-precision data casting.

One Ethernet fabric from tray to superpod

The rack doubles to 96 accelerators, and 128 racks join into a superpod of up to 12,288 accelerators over a two-tier network running a custom AI transport built on Ethernet. Within the rack, a half-switched topology links accelerator pairs directly inside a tray while an electrical switched fabric binds the full rack into a pod; a second optical tier of switches provides reach across racks. Microsoft presents a single communication model to software: any accelerator can initiate transfers to any other accelerator in the superpod with identical semantics, without involving the host CPU or a discrete scale-out NIC.

The fabric defines three latency domains — directly connected pairs, one switch hop within a pod, and three hops across pods — so latency-sensitive tensor-parallel traffic can stay in the lowest-latency domains. Notably, the split between in-rack and cross-rack bandwidth is not fixed at design time but becomes a runtime parameter set by job packing; Microsoft says an accelerator can drive up to 750 GB/s to any peer when only a subset of chips crosses racks. Jobs can also straddle rack boundaries — a 64-accelerator job can run as four 16-accelerator shards spread across four racks — reducing stranded capacity. Prefill and decode workers can be sized and placed independently, with KV state transferred directly over the scale-up fabric rather than through a host-mediated path.

Why it matters

Maia 300 is further evidence that hyperscalers now design custom silicon around inference rather than training, and that the binding constraints are memory bandwidth, data movement and power — not raw FLOPs. The NUMA-across-chiplets layout, software-managed dataflow and sub-3-bit formats represent a bet that meticulous data placement beats brute-force compute within a datacenter's fixed power and cooling envelope. The unified Ethernet-based fabric is also a competitive statement against proprietary interconnects, showing standard networking stretched into scale-up territory. For operators and anyone tracking the economics behind Copilot, GitHub Copilot and Microsoft Foundry — the workloads Microsoft says shaped the design — the post offers a rare, quantified look at how cloud AI capacity is actually being engineered.