Post-mortem: trusting Python's garbage collector with temp files filled CI disks

A nightly build runs out of disk

The story opens the way many infrastructure incidents do. According to a post-mortem published on dev.to by the team behind Flude, a C++ analysis engine, a scheduled nightly build failed on a CI runner with the blunt errno message "No space left on device". Inspecting the machine turned up the culprit: thousands of leftover directories inside /tmp, every one of them carrying the ude_xml_ prefix the engine used for its own scratch space.

Where the leak came from

Flude's engine drives Doxygen in an unusual way: it hands the tool source code, receives a large XML dump in return, and parses that dump. Each run created a scratch directory via tempfile.mkdtemp() and placed a generated Doxyfile inside it, where Doxygen would write hundreds of megabytes of XML. The plan was to remove the folder at the very end of the pipeline with shutil.rmtree(). The flaw was structural: if parsing raised an exception, the pipeline stopped before the cleanup call executed, and a directory full of XML was stranded in /tmp indefinitely.

The team tried wrapping things in try/finally, but the pipeline's logic was spread across dozens of classes and the leak survived. Reports arrived from teams building very large projects, and because the failure depended on whether an exception had occurred, it looked flaky rather than systematic.

Delegating cleanup to the garbage collector

The next attempt took the apparently pythonic route: replace mkdtemp with tempfile.TemporaryDirectory, whose finalizer deletes the folder once the object is garbage collected. To keep each directory alive exactly as long as the engine needed it, the team stored references to these objects in a list held by a manager class.

Complaints stopped for a few days, the issue was closed, and then the CI runners crashed again. Two details had been missed.

First, for large C++ projects the engine parsed dozens of modules one after another, each getting its own directory. The parsers were full of circular references — an AST node pointing back at its parent class while the class kept a list of children — so objects were not released promptly and instead waited for a full garbage collection cycle. The manager's reference list kept every TemporaryDirectory reachable, and hundreds of them piled up in memory along with the disk space they occupied.

Second, when the collector finally did run the finalizers, it performed one enormous synchronous deletion, blocking the main execution thread for minutes. Other services on the same server then started failing with timeouts. The supposed fix had converted a slow disk leak into periodic multi-minute stalls.

Deterministic cleanup with context managers

The team's conclusion was blunt: relying on finalizers to release heavy system resources is an anti-pattern, because you cannot control when the garbage collector runs and therefore cannot control the disk I/O load it generates.

They rebuilt the pipeline so that the lifecycle of every temporary folder is governed by with blocks instead. Creation and cleanup now happen in the same place, and any component that needs the files reads them inside the block, which guarantees the folder is deleted on exit — including exits via exception. As a guard against regressions, a janitor step inspects /tmp before each build, force-deletes any orphaned directories it finds, and logs a warning that the original caller never ran cleanup.

Why it matters

This is one of the most common ways CI disks fill up: a scratch directory survives an exception, nothing reports it, and the damage accumulates one build at a time until a nightly job dies. TemporaryDirectory is convenient precisely because it hides cleanup inside a finalizer, but garbage collection timing is an implementation detail, and del is the wrong place for anything with real I/O cost.

The lesson generalises beyond Python: make resource lifetimes explicit with context managers, keep creation and release in the same scope, and add a pre-flight sweep for leftovers. And remember that deferred cleanup can be worse than no cleanup at all — a finalizer that turns into minutes of synchronous file deletion lands at the least convenient moment, on the main thread.