deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Inngest outage: restarted services needed the database the incident had taken down

Inngest reported an October 20 outage where restarted event ingestion services could not finish booting because startup required the same database connection the incident had broken.

Inngest outage: restarted services needed the database the incident had taken down

What happened

Inngest, a platform for event-driven function execution, suffered two disruption windows on October 20, 2026: 13:00 to 13:15 UTC and 15:20 to 15:35 UTC. According to the company's incident report, as summarized in a writeup on dev.to, event submissions could fail or time out, function scheduling ran late, and execution and checkpoint requests were slower than normal during both windows, with event ingestion fully unavailable for parts of each.

Inngest traced both windows to a single underlying problem: a loss of connectivity between its application infrastructure and its database. Requests to the database stalled, and every operation that depended on it, including scheduling, execution and checkpointing, slowed down along with them.

How the recovery path made things worse

The detail that turned a brief disruption into a pair of outages came next. During the incident, Inngest's event ingestion services exhausted their available memory and restarted themselves, which is exactly the behavior you would want from a resilient system under normal conditions. The catch was that the startup sequence for those services also required a successful database connection to complete.

That connection was still broken. The restarted instances therefore could not finish booting, and event ingestion stayed down for as long as the connectivity problem persisted, even though the service layer had already attempted to self-heal. Inngest's own analysis, per the dev.to writeup, identifies two resilience gaps that reinforced each other: memory exhaustion under load, and a recovery procedure that depended on the exact resource the incident had disabled.

What the report leaves open

Inngest's writeup does not say what caused the database connectivity disruption in the first place, only how the response to it extended the impact. As the dev.to article notes, that omission makes it impossible to tell whether the trigger sat with Inngest, with its cloud provider, or somewhere on the network path between them.

This is also the third incident report Inngest has published in October 2026, according to the writeup, which references earlier cross-data-center and connection-pooler failures reported for October 9 and 10. The recurring pattern across all three, the author argues, is a mechanism intended to provide resilience that instead becomes the single path an outage has to cross before it can end.

Why it matters

The incident is a compact illustration of a failure mode that appears in many distributed systems: a recovery mechanism that quietly shares a dependency with the failure it is supposed to absorb. An automatic restart is only as good as what it needs in order to finish restarting, and if that requirement is the same component the outage has already taken down, the restart does not shorten the incident, it extends it.

A few checks, adapted from the writeup, are worth running against your own infrastructure:

  • List every service that restarts automatically on resource exhaustion, and inspect what its startup sequence requires before it reports itself healthy.
  • For each of those dependencies, ask whether an outage severe enough to trigger the restart could also take the dependency down.
  • Where a service cannot start without a specific downstream connection, consider a degraded-start mode that lets it come up and buffer work locally instead of blocking entirely.
  • Treat "the service restarted" and "the service recovered" as separate signals in monitoring, because this incident shows the two can diverge for the full duration of an outage.
  • #incident-report
  • #reliability
  • #distributed-systems
  • #databases
  • #cloud

Related posts