Docker Engine 29 quietly breaks old API clients as routine upgrades roll it out

A post on dev.to by Oleksandr Kuryzhev, republished from his own blog, describes a failure mode that is catching production Docker hosts off guard: unattended upgrades install a new docker-ce package overnight, the daemon restarts, and by morning a reverse proxy can no longer discover containers. The proxy logs report that its client API version is too old, while docker ps still works from the shell. The daemon looks healthy; the breakage sits in the clients around it.

According to the post, Docker Engine 29 shipped on 10 November 2025 and the release line has since reached 29.9.3, with 29.x now offered through Docker's apt repository for supported Ubuntu releases. Hosts still on 28.x are being swept up by routine patching, and anything on a floating package channel receives the new major automatically.

What changes in Engine 29

The post, which draws on Docker's release notes and engine documentation, lists four changes worth planning around, and advises verifying each against the notes for the exact patch release being targeted because details shift between 29.x versions:

• The daemon now enforces a minimum API version, with a documented floor of 1.44, and rejects clients that speak anything older. That hits aging reverse proxies, monitoring agents, CI runners and homegrown scripts built on pinned Docker SDKs.
• Fresh installs default to the containerd image store. Existing hosts are documented to keep their current graph driver, but anyone who rebuilds hosts from images or automation and expects the classic overlay2 layout is exposed.
• An experimental nftables firewall backend is available as an opt-in alternative to iptables. Only hosts that enable it are affected, but the post flags it as worth knowing about if a security team is pushing nftables.
• Live restore does not cover major upgrades. Its documented scope is patch-level daemon upgrades, so containers should be expected to restart across the jump to 29.

Hosts that run only docker compose and the CLI from the same package set are largely unaffected, since client and daemon move together. The post also notes compose files should already have dropped the legacy top-level version key.

How to prepare

The post's short version: pin the package, inventory every client that talks to /var/run/docker.sock, and upgrade a canary host first. In practice that means:

• Inventory socket clients, meaning anything that mounts the socket or uses DOCKER_HOST: proxies, log shippers, CI agents and deploy tools. Check each tool's changelog for its Docker API support.
• Record a baseline of server version, API version, storage driver and the running container list so the outcome can be diffed afterwards rather than guessed.
• Drain the host from the load balancer before installing anything, because the package restarts dockerd and live restore will not carry containers across a major version.
• Stage configuration in advance, writing the target daemon.json and any systemd drop-in so the restart during installation picks them up.
• Install an explicit version string rather than "latest", then hold docker-ce, docker-ce-cli and containerd.io with apt-mark.

On configuration, the post recommends making the storage layer explicit, setting storage-driver to overlay2 and the containerd-snapshotter feature flag to false, so a rebuilt host behaves like the one it replaces and the image store changes only deliberately. For clients that cannot be upgraded before the maintenance window, a systemd drop-in setting DOCKER_MIN_API_VERSION=1.24 keeps pre-1.44 clients connecting. The author frames this strictly as a temporary bridge along a deprecated path, with a removal date on the ticket.

Verify and roll back

Verification is a diff against the baseline: the server should report 29.x, the storage driver should match what was recorded, and the container list should be identical. One subtlety the post stresses is that the daemon returns API rejections to the client and may not log them at the default log level, so an empty search of the daemon journal is not proof. Check each socket client's own logs for "too old" errors, and watch the proxy's discovered-backend count and monitoring agents' container metrics for drops.

Rollback means unholding the packages and installing the previous pinned 28.x string with --allow-downgrades, then holding again. The post cautions that downgrades across major versions are not guaranteed to be clean for every on-disk state, so a VM or volume snapshot should come first, while acknowledging that snapshotting a large /var/lib/docker costs real time and storage. Where that is too expensive, the canary-first approach is the safety net.

One gotcha after the switch: the classic graph driver and the containerd image store keep images separately, so enabling containerd-snapshotter on an existing host shows an empty docker images list until images are pulled or loaded again.

Why it matters

The failure signature here is inverted from the usual one. docker ps works, health checks pass and the daemon is fine, while discovery, metrics collection and CI jobs quietly degrade because they speak an API version the daemon no longer accepts. The trigger is typically a routine patch rather than a planned migration. Pinning package versions, keeping an inventory of everything that touches the Docker socket, and upgrading a canary host first turn an overnight surprise into a scheduled, reversible change.