cgroup v2 io.max caps backup I/O so nightly jobs stop starving database latency

The failure mode

A guide published on dev.to walks through a scenario many operators will recognize: the nightly backup starts at 01:00, and by 01:05 the database's p99 query latency has jumped from 4 ms to 90 ms and the on-call alert fires. According to the post, the backup is not corrupt, slow or CPU-hungry. It is simply writing around 400 MB/s to the same NVMe device the database lives on, and the database is left waiting in the block layer.

Why ionice no longer helps

The first instinct, ionice -c3, has been ineffective on most systems for years, the author argues. I/O priority only means something if the block scheduler acts on it, and CFQ, the scheduler that made ionice famous, was removed in Linux 5.0. NVMe drives commonly run the none or mq-deadline schedulers, neither of which schedules on ioprio, so the priority value is recorded and never read. Even under BFQ, which does honor priorities, ioprio is proportional rather than a ceiling: an idle-priority backup still consumes every spare IOPS the database is not using, and on a fast device that is enough to ruin tail latency.

Three knobs, three mechanisms

The core of the post is a breakdown of the cgroup v2 io controller, which the author stresses is several mechanisms with different guarantees:

• io.max, backed by blk-throttle, imposes a hard ceiling on bytes or operations per second per device. It is not work-conserving, so unused capacity is wasted.
• io.weight, backed by iocost or BFQ, gives a proportional share of I/O time among siblings, and is only active if iocost is enabled or BFQ is the scheduler.
• io.latency, backed by blk-iolatency, protects a group's latency target by throttling peers; it only compares siblings and works by observing, then reacting.

Two design details explain most surprises, per the post. First, io.max is enforced on block-layer requests, not on write(2) calls: buffered writes land in the page cache and become device I/O later during writeback, so a process can dirty hundreds of megabytes before the throttle ever sees them. Second, the kernel documentation explicitly allows temporary bursts. The practical conclusion is that io.max caps sustained device traffic rather than per-syscall speed, which is acceptable for a backup as long as you measure the right thing.

Wiring it into systemd

The recipe starts with verification: confirm that /sys/fs/cgroup is the unified hierarchy (cgroup2fs), that io appears in cgroup.controllers, and gather the MAJ:MIN device numbers with lsblk. Settings should reference the originating physical device rather than a partition or loop device, as systemd.resource-control(5) specifies.

The author then creates a backup.slice rather than limiting the service directly, so everything the job spawns inherits the same budget and the limit can be raised later without editing the unit. The actual caps go in a drop-in: IOReadBandwidthMax and IOWriteBandwidthMax of 200M and 100M respectively on the device, plus IOWriteIOPSMax of 2K.

Three details are flagged before copying that config. Systemd suffixes are base 1000, so 100M means 100,000,000 bytes per second, roughly 4.6% lower than intended if you were thinking in MiB. Bandwidth and IOPS caps apply simultaneously, so the IOPS limit bites first for small random writes while the bandwidth limit bites first for large sequential ones, which is usually what a backup needs. And resolving a path such as /srv to a device does not work through RAID or most volume managers, so name the physical device explicitly.

Proving the limit is real

After a daemon-reload, the ground truth is /sys/fs/cgroup/backup.slice/io.max. If the limit line is absent, the controller is not attached to the slice and no amount of restarting the service will help. The io.stat file provides live per-device counters: read it before and after a run, and wbytes divided by elapsed seconds should land near the cap rather than far above it. For a controlled test, the post runs fio with direct I/O inside the same slice via systemd-run and expects roughly the capped rate; if you instead see uncapped throughput, the limit is not attached to that unit.

The traps

Buffered writes lie, the author warns: dd without oflag=direct or conv=fsync measures the speed of copying into page cache and can happily report 3 GB/s while the cgroup is capped at 100 MB/s, because the bytes have not reached the device yet. Always test with direct I/O or forced writeback. The post also notes that cgroup writeback attribution requires filesystem support, though the published text cuts off mid-explanation at that point.

Why it matters

Bulk workloads and latency-sensitive services routinely share a single NVMe device, the default layout on small servers and many cloud VMs, and a backup window paging the on-call engineer is a recurring, self-inflicted incident. The post's value is that it pairs a working io.max configuration with an explanation of why earlier attempts fail: ionice is ignored by modern schedulers, page-cache buffering hides the throttle, and base-1000 suffixes shift the numbers. It also clarifies when to reach for io.max versus io.weight or io.latency: a hard ceiling for background jobs, and proportional share or latency protection when the goal is fair sharing rather than containment.