When df -h shows free space but writes fail: a field guide to inode exhaustion

A write failure that looked like a full disk

According to a post on dev.to, a developer spent roughly two days chasing a write failure that behaved like a full disk but was not one. A small Python worker that dropped a JSON sidecar file next to each run started failing with OSError: [Errno 28] No space left on device, even though df -h reported plenty of free space on the root filesystem and /tmp looked equally calm. A manual touch in the home directory even succeeded, which the author took as evidence that the worker's path was somehow special.

That successful probe was the trap. As the post explains, a filesystem can accept a file in one directory and refuse a tiny write in another, because storage blocks and inodes are separate limits. The author spent about twelve hours treating the symptom as a space problem before questioning that assumption.

Cleanup that made things worse

The first round of fixes followed familiar operator instincts: truncating logs, deleting large files surfaced by du -sh, and restarting the process in case a leaked temp file was involved. None of it helped. The situation then deteriorated because of a well-meaning "safe delete" helper that copied each file into a .trash directory before unlinking it. For tiny JSON files, copying costs almost nothing in bytes but consumes an extra inode while both copies exist, so tidying pushed the filesystem closer to its inode ceiling.

Python tooling compounded the problem. __pycache__ trees, pytest cache directories and editor swap files never appear large in du -sh output, because they consist of thousands of small entries rather than one big log — exactly the profile that drains inodes.

The command that answered the question

Around hour twenty, the author finally ran df -i, which reports inode usage instead of block usage, and found the mount effectively out of inodes. Once the real constraint was visible, ordinary tooling identified the offenders:

• Counting files per directory with find piped through awk, sort and uniq -c revealed the noisiest folders, which turned out to be caches and sidecars, not the logs being truncated.
• An extension histogram showed which file types dominated.
• Searching for known cache directory names such as __pycache__, .pytest_cache and .mypy_cache pinpointed classic tiny-file nests.

A lab and a diagnostic script

The post ships two artifacts. The first is a demonstration script that writes 20,000 tiny JSON files into a throwaway directory so readers can watch df -i climb before deleting the lab. It refuses to run without an explicit --i-understand flag and is labeled as a local demonstration rather than a production test.

The second is a read-only diagnostic script the author now runs before blaming hardware. It checks four distinct ceilings in a fixed order: block usage via df -h, inode usage via df -i, a touch probe to see whether a file can actually be created at the failing path, and directory density measured by entry count, plus open-file totals through lsof and ulimit -n. The script prints observations and attempts no repairs.

The author also flags a cousin of the bug: a directory with so many entries that ls appears to hang is not filesystem-wide inode exhaustion, but it stems from the same mistake of tracking bytes when file entries are the actual constraint.

Why it matters

ENOSPC does not only mean blocks have run out. Workloads that generate many tiny files — sidecars, caches, per-request artifacts — can exhaust inodes long before they fill a disk, and cleanup routines that copy before deleting can accelerate the failure. The practical lesson from the dev.to post is to pair df -h with df -i at the start of any mysterious write failure, since the two commands answer different questions, and to profile directories by file count rather than by size alone.