Default nginx proxy_buffering raised SSE first-token latency from 0.7s to 3.4s

A developer's Claude-powered chat app streamed smoothly on a laptop and then stuttered badly in production. According to a detailed write-up on dev.to, the culprit was nginx's default proxy_buffering behavior, which quietly turned a token-by-token server-sent events (SSE) stream into a batch delivery — and the obvious fix initially did nothing because of a second nginx rule hiding in an included file.

What the setup looked like

The app in question is a small internal chat tool: a FastAPI service with an endpoint that calls the Anthropic API in streaming mode and re-emits each text delta as an SSE event. Locally, uvicorn served it on port 8000 and streaming worked, with the first word appearing in under a second. In production, nginx terminated TLS and proxied the /api/ path to uvicorn — and there the same app sat on a blinking cursor for three to nine seconds before dumping a whole paragraph at once.

The measurements

The author replayed 200 prompts from their own chat history, one at a time, and timed not the first byte of the HTTP response but the first parsed SSE data event containing actual text, since headers can arrive early while the body remains stuck. They also counted how many separate network reads each response body arrived in, which proved to be the most revealing metric: a healthy stream arrives in dozens of reads, a buffered one in a handful or just one.

The results were stark. Direct to uvicorn, median time to first token was 0.71 seconds with a p95 of 1.32 seconds, and no response arrived as a single burst. Behind nginx with default settings, median time to first token jumped to 3.4 seconds, the p95 hit 9.6 seconds, and 58 of the 200 responses arrived in one single burst. The single-burst cases were short answers whose entire SSE body fit inside nginx's buffers, so the user saw nothing until the model finished. Longer answers came through in four or five lumps, which the author initially mistook for bad Wi-Fi.

Why buffering breaks token streams

As the post explains, nginx reads the upstream response into memory buffers and forwards it to the client in buffer-sized chunks. For ordinary web traffic this is a sensible default: nginx can pull a response from a slow app server quickly, free the worker, and trickle bytes to a slow client at its own pace. But a token stream is hundreds of tiny writes whose entire purpose is to show the first word immediately, and coalescing them is exactly the wrong behavior.

The fix that didn't apply

The first attempted fix was a dedicated prefix location block for the streaming endpoint with proxy_buffering off, proxy_cache off and gzip off. After a reload, the replay numbers were unchanged.

The reason: nginx location matching. nginx first finds the longest matching prefix location and remembers it, then checks regex locations in the order they appear in the config — and if any regex matches, the regex wins. An older regex location matching ^/api/, added months earlier to set a CORS header and living in an included file the author had forgotten about, was capturing the request before the new prefix block could serve it.

The debugging technique was simple and effective: add a distinguishing response header to each location block, request the endpoint with curl, and see which header comes back. When the expected header didn't appear, the request was clearly being handled elsewhere. Running nginx -T to dump the merged configuration and grepping for location revealed the forgotten regex block.

The reliable fix

The final solution has two layers. First, the streaming location uses the ^~ modifier, which tells nginx that if this is the longest matching prefix it should not check regexes at all. The block also sets proxy_http_version 1.1, clears the Connection header, and disables buffering, caching and gzip. Second, the application itself sends X-Accel-Buffering: no in the streaming response, a header nginx honors to disable buffering for that response — unless told to ignore it via proxy_ignore_headers. The author favors the header because it lives next to the streaming code and survives future nginx config rewrites.

One gotcha the post highlights: once the regex block stops matching the streaming path, everything it was doing for that path stops too, so the CORS header had to be copied into the new block.

After the real fix, time to first token landed at 0.76 seconds median and 1.41 seconds p95, with zero single-burst responses — roughly 50 milliseconds of added latency attributable to nginx.

Why it matters

Streaming responses are now the default UX for AI chat products, and time to first token is the latency users actually feel. Most deployments sit behind a reverse proxy whose defaults were tuned for serving pages, not token streams. Anyone shipping an SSE or streaming LLM endpoint through nginx should suspect buffering first, measure latency at the parsed-event level rather than the first byte, and consider the response-header fix so the setting travels with the code. The story is also a compact lesson in nginx location precedence: a regex added in an include file can silently override a more specific prefix block, and nginx -T plus per-block debug headers is the fastest way to find out.