deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Postgres as a message queue turns stalled workers into database outages

A dev.to essay argues that running job queues inside Postgres fuses app and database failures into one event, citing a benchmark of roughly 660 messages per second versus RabbitMQ's 25,000.

Postgres as a message queue turns stalled workers into database outages

A dev.to essay argues that the increasingly common habit of running an application's job queue directly inside Postgres quietly merges two systems into a single failure domain, and that the cost of that merge becomes visible during an outage. The piece walks through why the pattern is attractive, what it does to the database underneath, and where its limits sit.

The appeal, and the hidden cost

The article takes aim at the advice that Postgres can handle messaging alongside its other duties. The usual implementation leans on SELECT ... FOR UPDATE SKIP LOCKED, which lets workers reserve jobs from a table with one query and zero new infrastructure: no broker, no separate ops burden. That convenience is precisely the problem, according to the author, because the queue ends up embedded in the component your customers depend on most and that is the hardest to replace.

MVCC makes queues expensive

Postgres uses multiversion concurrency control, which writes a new row version on every update. A queue is essentially a table in constant flux: jobs get claimed, completed, and retried, and each transition is another update. Brandur Leach, formerly a staff engineer at Stripe, documented the consequences in his article on transactionally staged job drains in Postgres, describing table bloat, index fragmentation, and an autovacuum process that cannot keep pace. Gunnar Morling reached a similar conclusion in a November 3, 2025 analysis titled "'You Don't Need Kafka, Just Use Postgres' Considered Harmful", arguing that long-running consumer transactions inevitably produce MVCC bloat and write-ahead log pile-up until vacuum loses the race against the rate of change. In other words, the queue slowly degrades the database it lives in.

The throughput ceiling

The dev.to post also cites benchmark figures from May 15, 2023, in which a Postgres-backed queue topped out at roughly 660 messages per second with 1KB payloads and a P95 publish latency of 38ms. RabbitMQ, in the same test environment, processed 25,000 messages per second, close to forty times the throughput. Managed services stretch the gap further: standard Amazon SQS offers very high throughput with little practical ceiling, while FIFO SQS provides exactly-once processing at up to 3,000 messages per second, or 30,000 when batched. The author's point is that teams avoiding operational complexity are choosing a tool with a hard ceiling while a boring cloud queue scales well past them.

The failure mode that pages you at 3am

The essay's central warning concerns failure coupling. It references an AWS Architecture Blog post from December 17, 2021, which described how a stalled delivery process places backpressure on the database, creating a feedback loop of escalating failures. When the queue and the database are the same system, the consequences compound: the queue going down takes the application with it, the thrashing database prevents the queue from recovering, and every worker retry makes both worse. A second, quieter pressure point is connections. Postgres spawns an OS process per connection, and many worker nodes continuously polling, claiming, and updating jobs generate connection churn that can exhaust the pool, forcing user-facing queries to compete with job runners for slots.

When the pattern is reasonable

The author stops short of ruling the approach out entirely. Minor internal tasks and occasional background jobs are reasonable fits, and SKIP LOCKED remains an elegant primitive at low volume. The trap is assuming that something demonstrated at low volume will behave identically under production traffic. The essay offers a blunt diagnostic: if your queue failing would take your database down with it, you have not built a shortcut, you have built a single point of failure.

Why it matters

The piece reframes a popular architectural simplification as a risk decision rather than a free win. Coupling a queue to the primary database ties job throughput to database health, turns worker stalls into customer-facing incidents, and adds update-heavy churn that fights the storage engine itself. For teams whose job volume is creeping upward, the practical takeaway is to measure before assuming parity: dedicated brokers and managed queues cost some operational overhead, but they decouple failure domains, remove vacuum pressure, and offer throughput that Postgres-as-a-queue simply does not deliver.

  • #postgres
  • #message-queues
  • #database
  • #architecture
  • #aws

Related posts