One in three Common Crawl index lookups gets no answer, five months of monitoring shows

Five months of daily monitoring suggest that the Common Crawl index fails to answer roughly a third of the lookups sent to it, with failures clustering in the early UTC morning. The figures come from a developer writing on dev.to, who has run a small Cloudflare Worker since 15 May that asks the index, once a day, whether about a dozen URLs from a personal site appear in the current crawl.

What the numbers show

The check is deliberately simple, according to the post: fetch the list of crawl collections, run one CDX lookup per URL, and store a row with the answer. Over 173 snapshots and 1,889 individual lookups, 605 — 32 percent — received no answer at all. The reported breakdown: 344 HTTP 503 responses, 103 HTTP 504s, 80 HTTP 521s (the status Cloudflare returns when an origin is unreachable), 52 requests cut off by the worker's own timeout, plus 18 HTTP 502s, seven 500s and a single 520. In 11 of the 173 snapshots, every lookup failed.

Failures cluster around 04:00 UTC

The scheduled job fires at 04:00 UTC, and the data splits along that line. Lookups made between 03:00 and 06:00 UTC failed 491 times out of 1,339, about 37 percent. Lookups at any other hour failed 114 times out of 550, roughly 21 percent.

The author is careful about this split: most off-hour snapshots were manual reruns started only after a morning run had already failed, when service may have been recovering. Even allowing for that bias, the direction has held across five months, and October has been the worst month so far, with 79 of 132 lookups failing.

Retries helped, but not for outages

On 10 and 11 October the job did not even reach the URL lookups. The request for the collection list returned an HTTP 521, and that call — unlike the URL lookups, which had retries since August — had none, so the whole stage failed and wrote nothing. The developer added three attempts with 10- and 20-second waits. On 12 October the collection list came through; all nine URL lookups then returned 521.

Afternoon reruns of the same code against the same URLs fared far better: 12 of 12 lookups answered on 10 October, and 10 of 12 on 11 October. While the post was being written, around 06:10 UTC, a plain curl request got a 200 for the collection list and a 504 for one URL lookup.

The takeaway in the post is that retries spaced a few seconds apart help when a single request drops, but do nothing for an index that is unavailable for hours. The next planned change is a second attempt hours later rather than another attempt seconds later.

Errors are not the same as absence

The second lesson concerns how failures are recorded. A failed lookup is stored as an error, never as a negative result, and the inclusion rate is computed only over URLs that actually got an answer; when too few respond, the rate stays empty instead of showing a number. On 11 October, that discipline was the difference between saying that 7 of the 10 URLs that answered are in the crawl and implying that only 7 of all 12 URLs are. Only the first statement is true. Had errors been folded into a negative result, October would have looked like the site was dropping out of Common Crawl when the index was simply unavailable.

Why it matters

Common Crawl is a foundational dataset for web research and downstream tooling, and its index is often treated as a slow but dependable public endpoint. This evidence comes from a single monitor probing about a dozen URLs, and the author openly notes the selection effects in the data, but the failure rate has been consistent for five months and is at its worst in October. Anyone polling the index on a schedule should follow the post's closing advice: check how many lookups actually came back before interpreting what they said, plan retries in hours rather than seconds, and keep a missing answer strictly separate from a negative one. Otherwise an availability problem quietly turns into a wrong conclusion about the data.