Gateway logs: 12% of LLM API calls return no answer, and retries multiply the cost

A production LLM gateway operator has published telemetry showing that a meaningful share of API traffic simply never produces an answer. According to a post on dev.to by AltRouter, the gateway logged 5,087 chat completion requests between 28 June and 10 September 2026, and 622 of them — 12.2% — ended without any response reaching the caller. The author's point is not that a provider went down, but that this failure rate is ordinary background conditions for anyone running LLM clients at scale, and it is absent from almost every pricing calculation.

What actually fails

The breakdown in the post is dominated by five outcomes. The largest, at 6.88% of all calls, is HTTP 503 — no provider capacity available for the requested model at that moment. The second largest, 2.40%, is HTTP 400: a malformed request, an unsupported parameter, or a role the endpoint does not accept, meaning the fault sits with the caller's own code. Client-closed connections (499) account for 1.51%, timeouts (504) for 0.94%, and out-of-credit errors (402) for 0.28%.

That split matters because the retry pattern most codebases copy — catch the error, sleep, retry three times — does not distinguish between the two families. A 503 has a genuine chance of succeeding on a later attempt. A 400 will fail identically every time. Of the 622 failures logged, 137 were terminal in that sense (400, 402 and 404 responses). Retried three times each, the post calculates, that is 411 requests sent that could never have produced output.

Where the meter actually runs

The cost asymmetry between failure types is the counterintuitive part of the data. A rejected request is essentially free: a 400 never reaches the model, and a 503 means generation never started, so no tokens are billed in either case. What you lose is latency.

Timeouts and cancellations are different. A 504 means the model was generating and the caller simply stopped waiting; a 499 means the user aborted or a client deadline fired after the answer had begun streaming back. The tokens exist regardless. Per the post, those two categories cover 2.45% of all calls, and they are the only failures during which billing was accruing.

The bill then compounds on retry, because a second attempt pays for the entire input again — the full prompt from the system message down, not just the remainder. Citing gpt-5.6's official rate of $2.50 per million input tokens, the author notes their median 1,290-token prompt costs roughly a third of a cent per attempt. That is trivial in isolation, but a job carrying a 40,000-token context that retries twice under load has purchased that context three times to obtain a single answer. The metric the post recommends tracking is therefore attempts per answer, not calls.

The recommended fixes

The post proposes changes centred on classification rather than volume. First, retry only status 429 and 5xx responses; anything else in the 4xx range signals a defect in the request itself, and waiting will not repair malformed JSON. Second, cap the retry budget in tokens rather than in attempt counts — three retries on a 500-token prompt and three on a 40,000-token prompt are identical lines of code but an eighty-fold difference in spend. A short code sample in the post implements both rules. A third, cheaper suggestion is to enable streaming, so that a timeout leaves a partial answer that can be shown or salvaged instead of nothing at all for the same spend.

A hole in the metering

The author is explicit that the dataset has a blind spot. The gateway records zero tokens and zero charge for all 622 failed events, because it only meters the usage block the upstream provider returns — and a request that dies mid-generation never returns one. The frequency of failure is therefore well documented, while the true upstream cost of abandoned generations is unknown. The author suspects most operators' logs share the same gap.

Why it matters

Most LLM cost modelling starts from a clean assumption: one request, one answer, multiply by price. This dataset shows that in production roughly one call in eight returns nothing, that naive retry logic wastes hundreds of requests on errors it can never fix, and that the most expensive failures are the ones where generation had already begun. For teams budgeting LLM spend or setting reliability targets, the practical takeaways are to classify errors before retrying, to bound retries by token spend rather than attempt count, and to instrument attempts per answer — because any cost model built on call counts alone is quietly wrong in both directions.