SkyPilot scheduler claims 27% more usable GPUs from Grace Blackwell NVL72 racks

The claim

A blog post from SkyPilot, which reached the Hacker News front page, argues that NVIDIA's Grace Blackwell NVL72 racks only pay off when the scheduler treats the rack — not the individual server — as the unit of placement. The company says its platform yields 27% more usable GPUs from identical physical racks by keeping multi-node jobs inside a single rack's NVLink domain.

Not all interconnects are equal

For years, NVLink existed only inside a single server. Any job spanning multiple machines had to cross InfiniBand or a cloud analogue such as AWS's EFA or Google's GPUDirect-TCPX, and that boundary lands exactly where it hurts. Two communication patterns sit on the critical path of modern training: the all-reduce steps of tensor parallelism and the all-to-all exchanges of expert parallelism.

Tensor parallelism shards a layer's weight matrices across GPUs. The arrangement popularized by Megatron-LM — splitting the first matrix multiply column-wise and the second row-wise — trims synchronization to one all-reduce per transformer block, but every layer still triggers two all-reduces during the forward pass, and these cannot fully overlap with computation. Hugging Face measured a 43% drop in training throughput after moving that traffic from NVLink to InfiniBand, a figure SkyPilot cites.

Expert parallelism compounds the problem for mixture-of-experts models: a router dispatches tokens to experts scattered across GPUs, then combines the results, and the volume of all-to-all traffic grows as models route each token to more experts. A 64-way expert-parallel group on conventional 8-GPU servers must span many machines, pushing all-to-all traffic through InfiniBand, where SkyPilot says unoptimized communication can consume up to 60% of training time and per-GPU bandwidth falls 18x.

What NVL72 changes

NVL72 stretches NVLink across a whole rack: all 72 GPUs join one fabric with 130 TB/s of aggregate internal bandwidth. Each GPU carries 18 NVLink ports delivering 100 GB/s of bidirectional bandwidth apiece — 1.8 TB/s per GPU — mapped onto 18 NVSwitch chips. The topology is fully non-blocking, so any GPU in the rack can reach any other at full bandwidth with uniform latency.

The practical consequence is that large parallelism groups fit inside one rack. A 64-way expert-parallel group stays entirely within the NVLink domain instead of fragmenting across eight machines. For inference, SkyPilot cites figures showing GB200 NVL72 serving DeepSeek-R1 at interactive speeds with 32-way expert parallelism delivering up to 4.4x the per-GPU throughput of 8-GPU B200 servers, which top out at 8-way. A vLLM benchmark SkyPilot references makes the memory side concrete: splitting a model tensor-parallel across two GPUs multiplied available KV-cache blocks by 13.9x and throughput by 3.9x.

Scheduling is the hard part

The catch is that NVL72's advantage evaporates the moment a job spills across racks, because inter-rack traffic still rides InfiniBand. And rack locality is fragile in operation: hardware failures take nodes offline, while jobs finishing at different times scatter free capacity across the cluster.

Reliability data makes this concrete. SkyPilot points to a supercluster Meta operates at gigawatt scale, where — after substantial reliability engineering — a 4-GPU node still saw a mean time between interruptions of roughly 490 days. Across thousands of nodes, interruptions are routine.

SkyPilot's counterintuitive advice is to avoid allocating entire 18-node racks. Leaving a couple of spare nodes in each rack yields more usable capacity than packing every node, because slice sizing determines whether a job survives a GPU failure without surrendering rack locality. The SkyPilot Platform combines topology-aware placement, in-place recovery from hardware failures, and automatic defragmentation of shared clusters; the 27% headline comes from the company's own analysis of that approach.

Why it matters

Frontier training budgets are dominated by GPU cost, and NVL72 racks are expensive, scarce capital. Extracting 27% more usable capacity from hardware you already own is equivalent to acquiring roughly a quarter more GPUs — or deferring that purchase entirely. The post also reframes a hardware story as a software one: the value of rack-scale fabrics is captured or destroyed by the scheduler, not just by the silicon. The obvious caveat is that the number comes from a vendor selling the scheduler in question, but the interconnect physics behind it rest on measurements from Hugging Face, Meta, NVIDIA and vLLM that SkyPilot assembles into a coherent argument.