Uber assigns error ownership across call chains to stop cross-service retry storms

The problem Uber is solving

Uber has published an engineering write-up, surfaced on the Hacker News front page, describing how it defends production systems against retry storms: cascades in which automatic retries pile onto an already-struggling service and turn a local outage into a stack-wide incident. According to the post, such storms have historically damaged both operations and brand trust, and Uber's existing defenses, per-service retry tuning and retry budgets, share a weakness: they are configured manually and have no visibility into how retries amplify across deep dependency chains and fan-out patterns.

The deeper gap, Uber argues, is that retry behavior is not context-aware. A system can limit how many retries occur but cannot judge whether a specific retry is worthwhile, because callers cannot reliably distinguish an error a dependency generated from one it merely propagated. The company considered translating downstream error codes upstream but rejected the idea: at Uber's scale of fan-in, fan-out and constantly evolving call graphs, it would not keep up. The chosen fix instead lives in shared infrastructure.

How retries multiply

The post spells out the arithmetic. Take a chain of seven services, each receiving the same number of requests in steady state. If the service at depth three begins failing and every hop retries once, load doubles per level: that node and everything beneath it end up serving eight times the baseline traffic. Generalized, a node at depth d sees R to the power of d times the baseline, where R is the per-hop retry factor. One sick node now floods six.

A 10% retry budget reshapes the formula to (1+B) to the power of d, capping the deepest services in the same example near 1.33 times baseline, a significant improvement. But budgets assume errors are independent and retries likely to succeed. Uber's own numbers show a caller perceiving roughly 99% availability from a callee actually running at 90%, yet the post stresses that during overload, with bad database hosts, database saturation or sharding issues, errors are correlated, so retries rarely recover anything while adding load. Retrying harder against a degrading service accelerates the failure.

Error ownership

Uber's answer is error ownership, framed as symptom versus cause. If a service makes outbound calls to fulfill a request and one of them fails, causing the service to return an error, that error is only a symptom; the cause sits downstream. If no outbound failed and the service still errors, the service owns the error. Uber's Service Dependency Analysis Solution correlates inbound failures with outbound failures per request and applies decision logic that claims or refutes ownership, expressing the verdict in error-claim headers.

Callers then adapt retry behavior. A claimed error, where the callee admits the fault is its own, can be retried, because the callee is the right target for a second attempt. An error a caller merely propagated, marked unclaimed, should not be, since retrying an intermediary just replays the same downstream failure. Where claim headers are missing, because a downstream service lacks the dependency analysis or adequate context propagation, the first node to notice unclaims the error; retries still occur at the edge nearest the failure, but the disturbance stops climbing the stack.

The post also covers mixed scenarios in a decision matrix and flags coincidental errors: a node can fail internally, for instance from an overloaded cache that the dependency analysis does not track, at the very moment its fail-open dependencies are also struggling, which is why per-request correlation is needed rather than static topology alone.

Why it matters

Retry storms are not an Uber curiosity; they are a default failure mode of any microservice architecture that retries automatically. The transferable lesson is architectural: push the decision of when to retry into shared platform infrastructure, make it conditional on where an error actually originated, and accept that under severe degradation the correct retry policy is often to stop. For teams that have already tuned per-service budgets and still watched cascades form, the ownership model offers a concrete next step.