Docker's graphdriver store re-pushes zstd image layers labelled as gzip

The bug

A report published on dev.to describes a distribution bug in Docker's graphdriver image store: when a daemon using that store pushes an image it previously pulled with zstd-compressed layers, the manifest it writes labels those layers with the gzip media type application/vnd.docker.image.rootfs.diff.tar.gzip. The layer blobs are reused unchanged, so they remain zstd-compressed — only the label is wrong. The registry accepts the push, and docker run of the relabelled image still works, because Docker and containerd detect compression from the blob's leading magic bytes rather than from the media type.

The store in question is overlay2. According to the report, that covers every Docker 28 install, any Docker 29 daemon upgraded from 28 (which keeps overlay2 and logs "using prior storage driver: overlay2" at startup), and any daemon with containerd-snapshotter switched off. Fresh Docker 29 installs default to the containerd image store, which handles the same pushes correctly. docker info tells the two apart: "overlay2" means the graphdriver store, "overlayfs" the containerd one.

How it was reproduced

The author reproduced the behaviour on Docker CE 29.9.1 and 29.9.2 using docker-in-docker daemons, a local registry, and a single-layer zstd image constructed with curl and the zstd CLI. A graphdriver daemon pulled that image, retagged it and pushed it back: the blob digest was identical, but the new manifest declared gzip. A containerd-store daemon performing the same pull-tag-push sequence preserved the original tar+zstd media type. The relabelled blob still began with the zstd magic bytes 28 b5 2f fd.

Where it breaks downstream

Docker's own workflow never fails, which is what makes the bug easy to miss. The cost appears in tools that trust the label, and usually later, on a different machine. The report demonstrates this with BuildKit: a build FROM the relabelled image with compression=zstd and force-compression=true fails with "failed to get compression blob "zstd": not found". Per moby/buildkit#7308, force-compression treats a layer's media type as its actual compression, so a zstd blob labelled gzip is assumed to be gzip needing conversion, and the lookup for its zstd variant finds nothing.

Without force-compression the build succeeds — but the new image inherits the two wrong labels, so whoever builds on it next receives them too.

There is also a cache trap. After one build from the mislabelled image, even builds from the correctly labelled original failed on the same builder, because the layer had been cached under the gzip label. Only a docker buildx prune -af on that builder restored correct behaviour for the clean base image.

Root cause and status

According to the report, when the graphdriver store pushes a layer the registry already holds, or can mount from another repository, it uploads nothing and only writes the manifest entry — and the media type on both of those code paths is hard-coded to Docker's gzip type in daemon/internal/distribution/push_v2.go. That is correct for originally-gzip layers and silently wrong for zstd layers pulled from a registry; nothing validates it. The containerd store keeps the layer's original descriptor, media type included, which is why it pushes correctly. The issue is tracked upstream as moby/moby#53999, labelled help wanted, with no fix in a released version at the time of writing.

Workarounds

The report recommends pushing from a containerd-store daemon: either a fresh Docker 29 install, or setting "features": {"containerd-snapshotter": true} in /etc/docker/daemon.json followed by a restart. Images stored under the old store do not appear in the new one, so they need re-pulling. Alternatives are rebuilding pulled zstd images rather than re-pushing them from a graphdriver daemon, and pruning the builder cache after hitting the BuildKit error. A companion script in the report audits a registry by fetching each image's manifest and the first bytes of every layer, then comparing label against content; run against Docker Hub's alpine:3.24 and nginx:alpine, every layer checked out fine.

Why it matters

This is a quiet corruption of distribution metadata, not a crash. Because Docker itself sniffs the bytes, mislabelled manifests can spread through registries — via retagging, mirroring or promotion between repositories — until a tool that trusts media types hits them and fails somewhere else. With Docker 28 deployments and upgraded 29 daemons widespread, plenty of registries can mint these manifests today. The broader lesson the report draws: a label is a claim, not a fact, and metadata recorded next to data can drift from it — so verify content when it is cheap, and test the step after the one that was reported, which is where this bug actually bit.