Developer shows first working live migration of bhyve guests on FreeBSD 15.1

A first working prototype

A developer who publishes as gyptazy has released a first working implementation of live migration for bhyve, FreeBSD's built-in hypervisor, demonstrated on FreeBSD 15.1. According to the blog post, which surfaced on the Hacker News front page, a running amd64 guest can be moved from one host to another over TCP while it keeps executing, pausing only for the final stop-and-copy step.

The implementation rests on bhyve's experimental snapshot support and adds two new pieces: dirty-page tracking inside the vmm kernel module, and a migration protocol shared between bhyve and bhyvectl. The code and its test tooling are being published as a work-in-progress RFC — on the developer's own Git server and on GitHub — so that testing, discussion and improvement can happen in the open rather than in the abstract.

How the migration works

Migration begins with a pre-copy phase on the source host. bhyve first transmits the guest's populated memory, then repeatedly asks for a bitmap of pages modified since the previous scan and sends only those. Once the outstanding dirty set is small enough to fit within a configured downtime target — or stops shrinking, or hits a configured round limit — the guest and its devices are paused, the last dirty pages go out, and the snapshot state covering vCPUs, kernel state and emulated devices is transferred.

The destination host checks basic VM geometry (protocol version, page size, memory layout and vCPU count), receives the memory and state, restores it, and signals that it is ready. Only then does the source send a GO message, which acts as the commit point: before GO, a failed migration should resume the guest on the original host, and the destination will not start a guest that never receives that commit. A warm, stop-and-copy-only mode is also available, useful both as a fallback and as a comparison baseline. One design detail worth noting: the migration socket is opened by bhyvectl and handed to bhyve, which lets bhyve stay confined in its Capsicum sandbox.

What has been tested so far

Testing described in the post includes repeated back-and-forth migrations under a memory-rewrite workload that verifies every page and flags missing or stale data, plus a disk workload that writes and fsyncs files with checksum verification, re-checked after a cold reboot to exercise the device pause, drain and restore path. Failure cases — destination refusal, an unreachable target, and a target vanishing mid-migration during both pre-copy and stop-and-copy — have also been exercised.

The protocol handles multiple vCPUs: the handshake carries the guest's vCPU count, the destination rejects a mismatch, and all vCPUs are paused and resumed together. In practice the test bed has mostly run a one-vCPU guest, so multi-vCPU stress and race testing remains outstanding.

In a two-host environment, repeated live migrations completed successfully, with observed downtime of roughly 138–171 ms for a slowly dirtying 512 MiB guest. A rapidly dirtying guest may never converge during pre-copy, in which case the migration still completes correctly through stop-and-copy with a longer pause. The developer is explicit that these are engineering results, not a promise of production support. Scope is currently amd64-only, the work depends on the experimental BHYVE_SNAPSHOT support and snapshot-capable emulated devices, and PCI passthrough cannot yet be migrated.

What stands between this and the FreeBSD tree

The post lists a set of engineering requirements before upstreaming can responsibly be evaluated. The kernel-side dirty-page algorithm must be proven never to lose a write while dirty bits are examined and cleared, backed by concurrency and multi-vCPU stress coverage and a reviewable locking and memory-ordering argument. The compatibility contract must grow beyond matching memory size and vCPU count to cover guest identity, CPU capabilities, bhyve/vmm and snapshot format versions, device topology and shared-storage identity.

The transport also needs hardening: today's TCP path carries no authentication or encryption and speaks a first-version native protocol, while a stable protocol needs explicit wire formats, bounded message handling, timeouts, version negotiation and a real security story. Beyond that, the commit boundary around device side effects needs review so restored devices cannot cause externally visible work before the source commits; storage and network handoff — shared-storage coherency and fencing, plus tap/bridge handling, MAC/IP movement and traffic convergence — needs specification; an in-tree automated test strategy with a documented support matrix is required; and the work must be split into independently reviewable patches covering dirty-page tracking, snapshot API refactoring, protocol work, CLI changes and tests.

Why it matters

Live migration is a foundational capability for enterprise virtualization: it is what makes host maintenance, hardware replacement, load balancing and failure-domain management possible without disrupting workloads. Until now bhyve has had no working implementation — the post presents this as a first working state — which has been a real constraint for FreeBSD-based hosting.

The post names two concrete beneficiaries. Sylve, a project aiming to deliver a Proxmox-like virtualization platform on FreeBSD, treats migration as a prerequisite for clustered operation rather than a convenience. And BoxyBSD, the developer's service offering free BSD-based VPS instances, currently faces guest outages for routine host work without a migration path. The stated priorities going forward are correctness, compatibility validation, transport safety and repeatable tests — performance features come later.