virtio-nvgpu gives KVM guests near-native NVIDIA GPU access by forwarding driver ioctls

nestrilabs has released virtio-nvgpu, an experimental virtio device that gives KVM virtual machines near-native access to an NVIDIA GPU on the host. According to the project's GitHub repository, which reached the Hacker News front page, a guest renders within 2% of bare-metal frame times on the same machine, at the same CPU cost.

Forwarding ioctls instead of translating APIs

The mechanism is where translation happens. Approaches such as Venus serialize individual Vulkan or OpenGL calls in the guest and replay them on the host. virtio-nvgpu instead works at the kernel driver ABI level, forwarding the ioctls that NVIDIA's kernel driver handles between guest and host.

The guest runs NVIDIA's own user-mode driver stack — Vulkan, NVENC and the rest — unmodified, against the same physical card. Because those libraries build command buffers locally in guest memory, and that memory maps to the host's, individual draw calls never cross the boundary. Over 813,691 benchmark frames the backend served only 13,792 messages, a crossing about once every 59 frames, nearly all of it device setup.

The measurements

The numbers come from an RTX 3060 running driver 595.99.02, comparing a guest against the same host on bare metal under an identical headless Vulkan load. For frames the host takes 39 ms, 9.9 ms or 2.0 ms to draw, the guest lands within 2% of bare metal — the 39 ms frame was 0.4% faster than bare metal, which the project attributes to noise. Overhead appears only below about 2 ms per frame: +7.1% at 0.5 ms and +40.8% at 0.05 ms, where a wake alone costs roughly 0.02 ms. Since games draw frames far heavier than 2 ms, the project treats the fast path as the normal case.

CPU usage over a 12-second unpaced run near 100 fps was 0.40 s for the host and 0.37 s for the guest. Inside the guest, nvidia-smi reports the card's real power and memory, and vulkaninfo exits cleanly. In an end-to-end demo, a Wayland client presented through a guest compositor, frames were encoded on the same GPU through Vulkan Video, and ffmpeg decoded the resulting 618 H.264 frames without error.

Four guests on one card

Four guests running the same load on one RTX 3060 reached 25.84, 26.49, 25.57 and 25.79 fps — 103.7 fps combined, against 102.9 for a single guest — with median frame times between 39.164 and 39.168 ms, matching to four decimal places. All four rendered and encoded H.264 simultaneously at 60 Hz without hitting an NVENC session limit. The project ran four; it did not establish that as the limit.

Built for headless streaming

The stated target is streaming from a VM with no monitor: a game renders, a guest-side Wayland compositor composites the windows, CUDA zero-copy imports the composed frame, and NVENC encodes it, so only a compressed H.264 or H.265 bitstream of roughly 100 KB per frame leaves the machine.

The repository explains why existing options fall short for that pipeline. Venus-style API translation costs 1–3 ms of serialization per frame on draw-call-heavy workloads — 6–18% of a 16.6 ms budget at 60 fps — and burns host CPU, while host-owned buffers leave no practical path to guest-side encoding. DRM native context lets Intel and AMD guests build command buffers locally, but no equivalent exists for NVIDIA. VFIO passthrough is native but hands the entire GPU to one VM, which multi-tenant hosting often cannot accept.

Structure, licenses and limits

The repository holds four components across three license zones. driver/ is a GPL-2.0 guest kernel module that registers /dev/nvidia* devices and forwards ioctls and mmaps over the virtqueue. device/ is an Apache-2.0 Rust crate with no virtual machine monitor in its dependency list; a VMM adopts it by implementing a small set of traits. protocol/ carries dual-licensed ABI definitions shared by both halves, and the layout follows chromeos/virtio-media. isolate/ is a design note for a sandboxed per-guest helper that would hold the real device file descriptors; it is unwritten, and the backend holds them itself today inside the VMM's process.

The limits are stated plainly: nothing heavier than vkcube at 720p has been tried, two cards have been used but only one benchmarked, CUDA is forwarded but untested past enumeration, and the multi-tenant isolation envelope is unbuilt. ABI profiles cover driver versions 535.129.03, 580.178.04 and 595.71.05, matched by range, and anything older than the first is refused.

Why it matters

NVIDIA has been the missing vendor in Linux GPU virtualization: Intel and AMD guests can build command buffers locally, NVIDIA guests could not. virtio-nvgpu closes that gap with an architecture whose measured overhead is near zero at the frame sizes games actually produce, and that shares one card across several guests without losing throughput. If the results hold beyond a 720p demo, GPU-hosted game streaming and multi-tenant rendering get cheaper, because a card no longer needs to be pinned to a single VM. The unwritten sandbox and untested CUDA keep it experimental — but the performance claim is measured, not asserted.