Serving GLM-5.3 at 607.8 tok/s: NVIDIA splits the 753B model across two DGX Stations

What the setup does

The NVIDIA Developer Blog has published a tutorial, currently on the Hacker News front page, showing how to serve Z.ai's GLM-5.3 on local hardware once the model outgrows a single machine. The recipe pairs two DGX Stations, each with a GB300 GPU, over a direct ConnectX-8 RDMA link and runs vLLM with tensor parallelism set to 2, splitting the model's tensors across both GPUs. Both machines compute every request, but only Station A (rank 0) exposes the OpenAI-compatible API; Station B runs a worker with no API server of its own. The walkthrough reports a headline figure of 607.8 tokens per second from that one endpoint, measured as part of a sequence that covers serving, sending requests and testing behavior under concurrent load.

Why one GPU no longer fits

The model is NVIDIA's GLM-5.3 NVFP4 build, a quantized version of Z.ai's mixture-of-experts model with 753 billion parameters in total and 40 billion activated per token. Because routing can send any given token to any expert, sparse activation trims the compute per token but not the memory footprint: every weight has to stay resident. At roughly 464 GB, the checkpoint comfortably exceeds the 252 GB of HBM3e on one GB300, so the tutorial spreads it across two Stations and turns CPU weight offload off entirely, keeping all weights in GPU memory.

Wiring and pre-flight checks

The two Stations are first cabled and validated with NVIDIA's Connect Two DGX Stations playbook, which includes a GPU-memory RDMA test on both direct rails. You then pick one rail, cx8r1 in the post's example, note its RDMA device (mlx5_1) and each machine's IPv4 address on that rail, and record the values plus the node rank in an environment file that later commands source. Before anything launches, the guide runs a battery of checks: link status, MTU-9000 jumbo-frame pings across the rail, and time synchronization. According to the post, these exist to catch cabling and addressing problems early, before they turn into opaque NCCL timeouts; a healthy link reports 400000 Mb/s.

Pinned software, copied weights

Tensor parallelism requires both ranks to run identical software, so the guide pins the vLLM container image by digest (v0.30.0 on arm64) and the checkpoint (nvidia/GLM-5.3-NVFP4) by revision. The roughly 464 GB download takes about an hour at 1 Gb/s or around six minutes at 10 Gb/s, after which it is rsynced over SSH to the second Station so each machine holds a full copy on disk. vLLM performs the actual partitioning when the model loads, dividing the weights across the two GPUs without quantizing anything further.

What you need before starting

The prerequisites go beyond the hardware: two GB300 DGX Stations running Linux, SSH and sudo access with the same username on both, Docker and the NVIDIA Container Toolkit (preinstalled in the DGX software stack), outbound access to the container registry, Hugging Face and package repositories, and at least 464 GB of free disk per machine for the checkpoint, container image and temporary files. On the licensing side, users must accept the NVIDIA Open Model Agreement and GLM-5.3's original terms, and SPEED-Bench is governed by a separate NVIDIA evaluation dataset license. Benchmark prompts come from the gated cais/hle Hugging Face dataset, so a read-access token is required. A companion video covers the same workflow from preparation through inference, benchmarking and shutdown.

Why it matters

Open frontier-class models have crossed the threshold where a single accelerator, even one with 252 GB of HBM3e, cannot hold them. The usual answers are CPU offload, which taxes latency, or a multi-node cluster, which taxes budgets. This tutorial demonstrates a third path: two workstation-class machines, one direct RDMA link, and stock vLLM serving a 753B-parameter model at high throughput behind a single OpenAI-compatible endpoint. For teams self-hosting large mixture-of-experts models, the value is less the benchmark number than the reproducible baseline underneath it, with a pinned image, pinned model revision and explicit network validation, and an approach that should transfer to other checkpoints that no longer fit on one GPU.