Postgres recovery_prefetch linked to hours-long write outages via replica IOPS amplification

A field report published on ardentperf.com, titled "Postgres recovery_prefetch Considered Harmful and Causing Outages" and currently on the Hacker News front page, argues that the recovery_prefetch setting — designed to speed up WAL replay on standby servers — can instead cause real production outages, and includes a step-by-step reproduction that reliably takes down writes on a synchronous-replication cluster.

Two failure mechanisms

According to the report, the problem has two parts. The first was reported to the PostgreSQL hackers mailing list in March 2025 by Michael Zhilin of Postgres Pro: the prefetching code does not account for blocks that were already loaded into shared buffers from full-page images (FPIs) in the WAL, so the startup process issues fadvise prefetch calls for data that never needed to be read from disk. Zhilin's customer had a physical standby lagging hours behind on replay on modern, powerful bare-metal servers, with CPU concentrated in kernel space; the incident was resolved by disabling recovery_prefetch.

The second mechanism, which the report says had not previously been diagnosed, is IOPS amplification. When the primary performs large sequential I/O, recovery_prefetch converts that work into 8 KB random reads on the replica. For the same number of bytes read, this can amplify read IOPS as much as 16x on the standby — enough to exhaust a modest cloud disk's IOPS budget and leave replicas significantly lagging after any burst of write activity.

How lag turns into an outage

Amplification alone only slows replay. It becomes an outage when combined with synchronous replication, where the primary cannot accept writes unless at least one healthy standby is reachable.

The report walks through the standby's core state machine in WaitForWALToBecomeAvailable (src/backend/access/transam/xlogrecovery.c): a standby that falls back to restoring WAL from archive or pg_wal will not return to streaming until it has caught up. So if the walreceiver connection drops just as a write burst finishes, the replica must grind through replay at amplified IOPS while the primary blocks every write waiting for it to reconnect. The recipe is simply "enough bad luck": a network glitch that interrupts streaming replication at the same time as a burst of writes — and VACUUM, the author notes, is quite capable of producing that burst.

The reproduction

The author describes the outage as trivial to reproduce once you know the steps and has published a reproduction setup on GitHub. On small servers (2 CPUs, 8 GB memory, a 500-IOPS cloud disk) running Postgres with synchronous replication to a standby:

• initialize pgbench at scale 640, about 10 GB of data
• disable autovacuum on pgbench_accounts so it does not interfere
• run pgbench at a fixed 5 TPS
• checkpoint and update 5% of the table, generating a burst of writes, then wait for it to finish
• kill the walreceiver process to emulate a network or system hang

The result: TPS drops to zero and all writes hang for over 30 minutes — potentially hours with a larger pgbench table. During the catch-up window, the primary's disk saw roughly 300 IOPS for around five minutes, while the replica sat pinned at its 500-IOPS limit for about 40 minutes, reading approximately the same number of bytes in total.

With recovery_prefetch disabled (full-page writes still enabled), there is still a pause during the UPDATE itself, but no burst of read IOPS follows: the standby applies the 5% update in under ten minutes and stays caught up.

Warnings that predate the outages

The concern is not new. In February 2021, while the feature was still in development — about a year and a half before it shipped — Stephen Frost warned on the PostgreSQL mailing list that issuing unnecessary duplicate prefetches was arguably a concern even with FPWs off, though he concluded the benefits of prefetching would almost certainly outweigh it and did not object to the feature being committed.

Why it matters

Full-page writes are the only way community Postgres protects against torn writes on operating-system restarts, and the report says the majority of production installations enable them. Meanwhile, synchronous replication is increasingly the standard durability option on Kubernetes and commodity hardware. In that common configuration, the report argues, recovery_prefetch's benefits are dubious while its failure mode — a total write outage from an ordinary network blip — is severe. The author is leaning toward disabling the setting across the board pending community discussion, and separately suggests running two standbys with quorum commit (ANY 1), since a single walreceiver connection is a fragile dependency for the whole cluster's write availability.