PyPI traces two weeks of download 502s to a Fastly canary bug and its own CDN config flaws

PyPI has published a detailed incident report explaining roughly two weeks of intermittent 502 and 503 errors that broke package downloads from files.pythonhosted.org in August 2026. The report, posted on the PyPI blog on September 8, traces the failures to two unrelated causes: a misconfigured cache node inside Fastly's network exposed by a canary deployment, and several long-standing bugs in PyPI's own Fastly configuration that only surfaced during the investigation. Both are fixed, and downloads were back to full function on August 28.

How PyPI serves package files

According to the report, files.pythonhosted.org is a Fastly CDN service in front of three origins. Uploaded files are written first to Amazon S3, which holds the durable master copy, and a background job syncs them to Backblaze B2. B2 is the preferred read path because Fastly and Backblaze have an arrangement that makes egress free. If B2 fails, Fastly is meant to fall back to S3, which PyPI keeps in the Glacier Instant Retrieval storage class at a higher retrieval cost. Package files never change and carry a cache lifetime of roughly 11.5 years, so cached requests never touch an origin at all. A third backend, Conveyor, handles non-package requests such as predictable URLs and legacy redirects.

That failover design only works if the edge notices B2 has failed, and one of the bugs described below is that it did not always do so.

One bad Fastly cache node

The trouble started on August 15 at a single Seattle-area cache node, per Fastly's own account, a date corroborated by a user report to PyPI's support tracker. Fastly operates a canary cohort — a slice of its fleet running new caching and routing software ahead of wider rollout — and PyPI's traffic had been part of it for years. A partial rollback during a canary deployment left the caching configuration at the Seattle point of presence reverted while the routing configuration in front of it was not, and the mismatch made the routing layer return 502s for any traffic reaching that node.

Two user reports of persistent 502s arrived on August 17. A third, on August 18, included a reproduction showing 88 recorded 502s across six hours and 32 unrelated packages, including the small .whl.metadata range requests installers use to read PEP 658 metadata — while PyPI's own logs showed nothing significant at that point. On August 19, another report isolated the fault to a specific node, cache-pae2080020, using x-served-by response headers: every request routed there failed for more than 19 hours. Fastly's network operations then removed a routing override that was sending a slice of PyPI traffic to the affected location, and recovery was observed on August 20. Fastly patched the underlying canary bug on August 28 and has excluded all Python Software Foundation traffic, including PyPI, from the canary program; PyPI says it would like to rejoin eventually, given clearer controls and notifications.

Bugs in PyPI's own configuration

While investigating, PyPI's team uncovered its own bugs producing the same symptoms: elevated 502s, plus a few 501s that looked like 502s from the outside. The most significant was in origin fallback. The retry against the S3 archive only ran when B2 answered with an error status; if B2 did not answer at all — a timeout, refused connection or TLS failure — Fastly synthesized a 503 and skipped the retry entirely. A fix merged on August 18 added the missing archive attempt.

Another cluster of problems involved segmented caching, which PyPI uses to avoid pulling a whole large wheel into cache just to serve a partial-content Range request. That feature supports less range syntax than HTTP in general and returned a synthetic 501 for suffix ranges such as bytes=-1024 — exactly what some installers send to read wheel metadata without downloading the entire file. PyPI exempted suffix and multi-range requests on August 21 after a spike traced to a single client sending logically invalid ranges, and adjusted the handling again on August 24, after two broken parallel downloaders generated 41,315 errors of the same class in a single day. The report calls a 501 for a malformed client request the wrong status class and says a Fastly support ticket is open, after which the workaround can likely be reverted.

A final bug, fixed on August 28, was one of ordering: a segmented-caching exemption check ran before the request URL was normalized, so a .metadata request carrying a query string escaped the exemption — and a bad segmented-caching response could then be cached and served to every subsequent request for the same file.

Why it matters

PyPI is the default install source for the Python ecosystem, so edge errors surface as random, unreproducible failures in CI pipelines and deployments, far from the real fault. The report shows how much invisible plumbing — fallback ordering, range handling, URL normalization — decides whether a backend hiccup is absorbed or becomes user-visible, and how a dependency you cannot patch can still be your outage. It is also a case study in transparency: PyPI credits user reports with pinpointing a vendor-side fault to a single machine, and Fastly, in turn, shared the root cause.