CUDA 13.4 adds enforceable GPU memory limits for shared Kubernetes workloads

NVIDIA shipped CUDA 13.4 on 27 October 2026 with a capability that shared-GPU setups have lacked: a memory boundary the application cannot see but the platform can enforce. According to a dev.to write-up of NVIDIA's technical blog, the release also comes with an open-source Kubernetes operator that applies per-container GPU shares on every node, and KAI Scheduler v0.18.0 is the first scheduler to integrate the mechanism.

The problem: whole GPUs for small jobs

The post frames the issue as simple arithmetic. A pod asks for one GPU, the scheduler grants one GPU, and a batch job that needs 8 GB of memory ends up holding an 80 GB device while the remaining 72 GB sit idle. NVIDIA's hardware already offered three ways to share a device: Multi-Instance GPU (MIG) splits it into hardware-isolated instances, time-slicing lets processes take turns, and Multi-Process Service (MPS) runs processes concurrently on shared streaming multiprocessors, a mode NVIDIA calls sm-sharing. The gap was enforcement: time-slicing imposed no memory limit at all.

A boundary that behaves like a smaller GPU

CUDA 13.4 adds memory enforcement on top of these sharing modes. A workload requests a memory size and CUDA holds it there, so an allocation above that size fails exactly as it would on a physically smaller GPU. Workloads can instead declare a range: a guaranteed minimum plus a maximum they can grow into when memory is free. Each workload sees only its fraction of the device's memory, making the limit transparent to the application, and each container also selects a compute mode, with time-slicing as the default and sm-sharing available.

The scheduler writes, the node enforces

Kubernetes cannot reach CUDA's enforcement on its own. NVIDIA's answer is gpu-fractioning, an open-source operator hosted under the KAI Scheduler organization and deployed alongside the NVIDIA GPU Operator. It has three jobs: set up and run MPS on every GPU node, apply each container's memory limits and compute mode before its process starts, and measure memory and compute usage per fraction.

The contract between scheduler and node travels in four per-container Pod annotations under the nvidia.com prefix, covering the guaranteed GPU memory request, the maximum memory limit, the compute mode, and which GPUs the container was assigned. NVIDIA's stated rule is that any Kubernetes scheduler able to write those annotations can enforce GPU fractions with CUDA. The scheduler keeps two obligations of its own: choose a GPU with enough free memory, and never mix compute modes on one GPU. NVIDIA's position, as relayed by the dev.to post, is that this enforcement work belongs on the node rather than in whichever scheduler placed the workload.

KAI Scheduler v0.18.0 is the first integration. It exposes a GPU-sharing mode called NvFractions that is enabled with a single Helm flag and installs the operator as a dependency.

Two ways to split compute

With n workloads sharing a GPU, each gets roughly 1/n of the compute in either mode, but the scheduling differs. Time-slicing is serial: workloads take turns on the whole GPU, unused time is shared among the others, and each workload runs in its own error domain, giving full fault isolation. SM-sharing is parallel: streaming multiprocessors are divided between workloads that run concurrently, with hardware allocating and releasing SMs as needed, and a single active workload can use them all. The trade-off is that SM-sharing workloads share one error domain, and NVIDIA's guidance is to reserve the mode for workloads you trust to run together. Different GPUs on the same node can run different modes.

Worked examples

NVIDIA ships two configurations that cover the common shapes. An inference server requests a guaranteed 40Gi on each of four GPUs, a 60Gi limit it can grow into when neighbours are idle, and sm-sharing. A development or batch job asks for half of each of two GPUs (a fraction of 0.5 with a limit of 0.8) using time-slicing, so a crash stays local to that workload. Requests apply per device, and when the limit is omitted it equals the request, capping the workload at exactly what it asked for.

Why it matters

Until now, splitting a GPU across tenants or inference services meant either static partitioning with MIG or unenforced time-slicing, where one greedy allocation could disturb everything sharing the device. CUDA 13.4 settles the memory accounting question at the node level, which makes shared inference and multi-tenant GPU clusters materially more practical. The change is deliberately narrow: memory bandwidth, cache behaviour and failure domains are unchanged, so teams adopting it should verify the guaranteed request and burst limit per workload and treat SM-sharing as a utilisation gain paid for with isolation. The scheduler-agnostic annotation contract is the part most likely to matter beyond this release, since any scheduler can adopt it without NVIDIA building bespoke support.