Audit of 30 top websites finds only one blocking GPTBot while the most AI-cited sites skip llms.txt

What was tested

A hands-on audit published on dev.to by a developer writing as dongdongdev set out to test a piece of received wisdom: that the web is shutting AI crawlers out. Against a backdrop of viral robots.txt blocklists and default-on AI blocking of the kind Cloudflare has popularised, the author checked 30 well-known domains directly, collecting the data on November 11, 2026.

The method is deliberately simple and reproducible with curl: fetch each site's robots.txt and note whether it explicitly disallows OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot or Common Crawl's CCBot; request llms.txt and record whether it returns a valid response; validate sitemap.xml; and check the homepage for JSON-LD structured data. All requests used an ordinary browser user agent.

The sample was split into two groups of 15. Group A covers properties that AI assistants cite heavily, including Wikipedia, GitHub, Stack Overflow, MDN, Python.org, Kubernetes, Stripe, AWS documentation, Microsoft Learn and the Go and Rust sites. Group B is commercial SaaS: Notion, Linear, Atlassian, Salesforce, Zoho, Wix, Typeform, Calendly, Loom, Miro, Basecamp, Zendesk and GitLab, among others.

Explicit bot blocks are rare

Across all 30 sites, exactly one — Loom — explicitly blocks GPTBot in robots.txt. Not a single site in the sample blocks ClaudeBot, PerplexityBot or CCBot. According to the author, the lockout story may be genuine at the margins of the web, among some news publishers and forums, but among the domains that dominate AI citations, none of the four crawlers is being turned away.

The most-cited sites lag on AI-ready signals

The sharper result comes from comparing the two groups on machine-readability. The commercial SaaS sites, which have spent the past two years investing in "AI visibility", come out clearly ahead on the technical basics, while the heavily cited properties trail:

• llms.txt present: 9 of 15 SaaS sites (60%) against 6 of 15 citation-heavy sites (40%)
• Homepage JSON-LD: 11 of 15 (73%) against 4 of 15 (27%)
• Valid sitemap.xml: 12 of 15 against 4 of 15

Wikipedia and Stack Overflow ship no llms.txt at all, the post notes, and most of the developer documentation sites carry no structured data on their homepages. The author's pointed question: if llms.txt and schema markup governed citations, Wikipedia — described in the post as the most-cited domain in AI answers — would barely register.

The author's interpretation: trust is earned elsewhere

The explanation offered is that citations are settled by third-party reputation rather than on-site configuration. Models cite Wikipedia because enormous numbers of outside references vouch for it, and they cite GitHub and Stack Overflow because developers continually link to and quote them across the wider web. llms.txt and structured data make a site easier to parse, the argument goes, but no markup on your own domain compensates for the absence of external discussion.

The post also points to an earlier experiment by the same author, which logged roughly 1,400 citation events from five models across 200 buyer-intent prompts. That run pointed in the same direction: extractable pages and consistent entity naming helped, but only once a brand already appeared in third-party sources the models trusted.

Caveats to weigh

The audit is small and informal, and it measures declared policy rather than enforcement. robots.txt is advisory, so a site can serve errors to a crawler's actual user agent without ever naming that crawler in the file; the same fetches made with GPTBot's own user agent might tell a different story. The sample is curated rather than random, and it reflects a single day. The author also discloses that Wikipedia and GitHub returned errors to some automated robots.txt fetches — their crawler permissions are publicly documented as permissive, and both were left out of the llms.txt percentages rather than guessed at. Finally, the author sells an AI-visibility audit tool built on these same checks, which readers may want to weigh against the advice on offer.

Why it matters

The post is a useful corrective to two assumptions that often travel together: that major sites are quietly walling off AI crawlers, and that adopting llms.txt and schema markup will win an AI share of citations. For teams that are not Wikipedia, the practical sequence it suggests is to keep robots.txt permissive (per the author, blocking GPTBot is the surest way to forfeit AI-driven traffic), invest in presence on third-party properties the models already trust — community answers, directories, comparison pages — and make pages cleanly extractable, treating llms.txt and JSON-LD as inexpensive hygiene rather than a strategy in themselves. One 30-site snapshot does not settle the debate, particularly for news publishers where blocking is more common, but it suggests that at the top of the web the AI-crawler door is far more open than the headlines imply.