Robots.txt survey of 30 major sites finds ClaudeBot blocked more often than GPTBot

A survey published on dev.to has fetched the robots.txt files of 30 well-known websites and found that blocking AI crawlers is now the default among major publishers — and that Anthropic's ClaudeBot gets a noticeably colder reception than OpenAI's GPTBot.

What the survey found

The author checked the robots.txt files of 30 prominent sites on 1 November 2026, according to the dev.to post. Two of them did not respond usefully: npr.org timed out after 15 seconds, and stackoverflow.com returned HTTP 418 to the automated client, so the results cover 28 sites.

Of those 28, 25 block ClaudeBot, Anthropic's training crawler, while 17 block GPTBot, OpenAI's counterpart. Other training-focused bots sit between those poles: ByteDance's Bytespider is blocked by 24 sites, Common Crawl's CCBot and Apple's Applebot-Extended by 23 each, PerplexityBot by 21, Meta's meta-externalagent by 20, Amazonbot by 18, and Google-Extended — used for Gemini training — by 16.

Search crawlers are treated more gently

The data shows a clear split between crawlers that gather training data and those behind AI search products. OpenAI's OAI-SearchBot, which governs whether a page can surface in ChatGPT search, is blocked by only 8 of the 28 sites, and ChatGPT-User, which fetches pages on a person's behalf, by 10. Anthropic's equivalents, Claude-SearchBot and Claude-User, are each blocked by 16.

Six sites in the sample shut out every AI search crawler: nytimes.com, cnn.com, reddit.com, yelp.com, imdb.com and amazon.com. Six leave them all open: wikipedia.org, medium.com, github.com, etsy.com, tripadvisor.com and webmd.com.

Where GPTBot is allowed and ClaudeBot is not

Eight sites give the two training crawlers different answers, admitting GPTBot while keeping ClaudeBot out: theguardian.com, washingtonpost.com, wsj.com, theatlantic.com, theverge.com, wired.com, arstechnica.com and allrecipes.com.

The survey's author points out that most of these outlets have publicly announced content licensing deals with OpenAI, but cautions that the deals were not individually verified, so the overlap should be read as a pattern rather than proof of causation.

The Washington Post file illustrates the mechanics. Its robots.txt disallows ClaudeBot across the whole site, with one carve-out for /creativegroup/, while GPTBot has no group of its own and falls under the default User-agent: * rule, which only closes a handful of print and email paths.

llms.txt remains rare

Despite ongoing discussion of llms.txt as a machine-readable way for sites to describe themselves to AI systems, only two of the 28 sites — github.com and etsy.com — actually publish one.

Method and caveats

The data was collected with AI Crawler Checker, an Apify Actor that costs $0.003 per site; the dev.to post also serves as a walkthrough for the tool. It parses robots.txt according to RFC 9309 — the longest matching rule takes precedence, Allow wins over Disallow on a tie, and the * and $ wildcards are supported — and returns an allowed-or-blocked verdict for 16 crawlers. The author suggests scheduling weekly runs and comparing the blocked-bot lists to catch policy changes.

Two caveats are worth keeping in mind. First, robots.txt is an advisory convention rather than an enforcement mechanism: the major vendors — OpenAI, Anthropic, Google, Apple and Perplexity — say their crawlers respect it, but the survey captures what each site requests, not what any crawler actually does. Second, this is a one-day snapshot of 28 large sites, not a representative sample of the web.

Why it matters

The findings suggest robots.txt has quietly become a negotiating surface between publishers and AI companies. Rather than blocking everything or nothing, major sites are making crawler-by-crawler decisions that line up with commercial relationships — most visibly in the eight sites that admit GPTBot while barring ClaudeBot.

For anyone who publishes, the practical lesson is granularity. A blanket Disallow: / under User-agent: * also shuts out every AI crawler that has no group of its own, which is how reddit.com ends up blocking all of them. Sites that want to be cited in ChatGPT, Perplexity or Claude answers can leave the search and user crawlers open while still excluding training crawlers — the approach theguardian.com takes for OpenAI. And with llms.txt still nearly absent, robots.txt remains the main lever most publishers have.