systemd's mstack mounts single-layer directories writable, silently overwriting read-only files

What mstack does

systemd 260, released in March, added a new way to describe overlayfs mounts. Instead of assembling a long mount -t overlay -o lowerdir=... command line, a user creates a directory suffixed .mstack, drops symlinks named layer@0, layer@1 and so on pointing at the read-only layers, adds an rw/ subdirectory for the writable top, and calls systemd-mstack --mount. According to a hands-on write-up on dev.to, systemd 261 — released in June, with 261.3 being the build the author tested from a current Arch Linux image — extended the mechanism to systemd-nspawn --mstack= and to a service-level RootMStack= directive. The stated aim is ergonomics: mount stacks become shareable, inspectable directories instead of shell-script incantations.

The write-through surprise

The finding that anchors the write-up is blunt. The author built a directory containing two files, mounted it with mstack, wrote one line through the mount, and watched that line permanently overwrite a file that had been marked read-only. The command exited 0, with no error text and no warning. The post's title states the mechanism plainly: mstack mounts a single-layer directory writable by default. The practical consequence is that the mental model most people bring to overlays — lower layers are immutable, writes land in a separate upper layer — does not hold here, and the permissions on the source file offer no protection.

Confusing and silent failure modes

Two other rough edges surfaced during testing, which the author ran inside Docker because the sandbox had no systemd running as PID 1.

First, mstack fails confusingly inside a plain Docker container. The kernel will not stack a new overlayfs mount on directories that already sit on overlayfs, which is exactly what a container's root filesystem is under Docker's default overlay2 driver. The hand-written mount command reports this constraint in plain language ("filesystem on upper not supported as upperdir"); mstack instead prints (layerfd)' failed with exit status 1 followed by an "Invalid argument" complaint, naming an internal function rather than the actual limit. The limitation is not mstack's fault — the workaround, confirmed by the author on an ext-family host directory, is to place the layers on tmpfs or a bind-mounted host directory.

Second, some failures produce no output at all. A dangling layer@ symlink makes the inspection command exit 1 with no message; only --mount surfaces the "No such file or directory" error, and only because mounting forces the tool to actually open the target. A stray file or directory inside a .mstack that matches neither layer@* nor rw behaves the same way: exit 1, nothing printed. For anyone generating these directories with scripts, a template that leaves a stray artifact behind yields a bare failure with nothing to act on.

Speed, storage and one oddity

The documentation's version-sort claim held up: layers named layer@1 through layer@3 and layer@10 through layer@12 sorted numerically rather than lexicographically, verified through -- output. Timing five mounts each on tmpfs, mstack ran roughly a millisecond slower than the hand-written command (4–5ms versus 3–4ms) — the cost of resolving symlinks and building the option string, and noise at this scale.

There is no hidden duplication either. A 100MB file placed inside a layer was fully visible through the mount, while the temporary staging directory mstack created measured 40 bytes — worth stating, the author notes, since a self-describing abstraction is exactly the sort of thing that might smuggle in a copy step.

One observation went unexplained: /proc/self/mountinfo showed two identical lowerdir+= entries pointing at the same temporary path for a genuine two-layer stack. The mount worked and both layers' files were present, and the author preferred to say the cause was unknown rather than guess.

Why it matters

mstack pushes overlay assembly toward something declarative and inspectable, which is a real convenience for container and immutable-root setups. But its current defaults and diagnostics assume an attentive operator. Write-through behavior that silently defeats read-only markings can destroy source data on a stack the user believed was safe to poke at; message-free exit codes make scripted pipelines brittle; and the Docker-first tester — most people's first instinct — hits a kernel limitation dressed up as a tool bug. Until the diagnostics improve, treat every mstack mount as writable, verify what a write through it will actually touch, and do not expect a performance win: the feature's value is ergonomics, not speed.