CircleCI outage: scheduled Temporal cleanup filled Postgres log volume, halted pipelines for 12 hours

A scheduled deletion, not a load spike

CircleCI stopped processing pipelines and workflows for roughly 12 hours on October 9, 2026, after the Postgres instance behind its Temporal queueing layer ran out of disk space. According to a dev.to writeup of CircleCI's October 12 incident report, the trigger was not a traffic surge but a cleanup job the company had scheduled days earlier: Temporal's routine deletion of completed workflow history, queued up by the very incident CircleCI believed it had already fixed on October 6.

The setup behind the failure

Per dev.to, CircleCI's Temporal deployment relies on a multi-AZ Postgres instance whose write-ahead log lives on a dedicated 1TB volume provisioned at 3,000 IOPS — ceilings AWS will not raise. If Postgres writes WAL entries faster than they can be archived to backup storage, free space on the volume falls; when it reaches zero, the database stops accepting writes entirely.

The October 6 outage began when ordinary table maintenance produced WAL writes that outran archiving. CircleCI's remedy returned the transaction log to its dedicated volume and adjusted Temporal's cleanup settings to cut a major source of writes. What it missed: Temporal purges finished workflow history a few days after completion, meaning the unusually heavy October 6 workload was already booked for deletion on October 9.

How the day unfolded

At 08:00 on October 9 the deletion job started working through that backlog. Available space on the log volume slid from 0.8TB to 0.1TB by 10:10; monitoring alerted at 10:28 and an incident was declared three minutes later. By 11:15 the volume was full, Postgres flipped to read-only, and Temporal processing halted.

Reapplying the October 6 workaround — relocating the log onto the data volume — restored the database by 12:59, and the queue backlog that had accumulated since 11:15 drained by 14:40.

A second failure after recovery

At 15:15 a slower problem emerged. Routing-cleanup tables that ordinarily carry tens of thousands of rows awaiting deletion had swelled to tens of millions, forcing every Temporal operation to scan past them. Manual vacuum jobs made matters worse by competing with live queries for CPU and were cancelled at 17:45. By 20:48, engineers recognized the database had shifted from a WAL bottleneck to CPU thrashing, aggravated by a retry storm in which upstream systems resubmitted work that had already failed. Pausing all new and retried jobs was what finally allowed Postgres to catch up.

CircleCI's response

According to dev.to's summary, CircleCI has since split Temporal into multiple sharded instances with independent databases, removed the fixed-size log volume, added alerting on WAL archive lag and write rate — which its postmortem says would have caught the problem hours earlier — and changed how its systems retry against an unhealthy Temporal instance.

Why it matters

The instructive part for teams running large Postgres or workflow-engine deployments is the shape of the failure: the mitigation for incident one was scoped to the write pattern just diagnosed, while a second, delayed trigger sat on the calendar. Nothing in the October 6 symptoms pointed at the deletion job that would fire three days later. The dev.to author frames the general lesson: a postmortem fix is not verified until you have checked it against every other job scheduled to touch the same resource in the following days.

Two practical checks follow. First, alert on WAL archive lag and write rate directly rather than only on free disk space, since those signals surface the problem much earlier. Second, confirm whether retention or cleanup jobs run on a fixed schedule regardless of system load, so they cannot land during the worst possible hour. Retry behavior deserves an audit too — upstream retries added load on top of a genuine outage, so backoff logic that senses a struggling downstream dependency can shorten recovery considerably.