deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Audit of 700 top sites' robots.txt finds AI crawler rules often block the wrong bots

A dev.to audit of 700 top-ranked sites found that 43.6% of OAI-SearchBot blocks came from blanket rules meant to keep training crawlers out, and published the full domain-by-domain dataset as open data.

Audit of 700 top sites' robots.txt finds AI crawler rules often block the wrong bots

An audit of the robots.txt files belonging to 700 of the web's highest-ranked sites has found that operators routinely block AI crawlers they never chose to target, and sometimes loosen restrictions they believed were airtight. According to the analysis published on dev.to, every file was matched against the AI crawler tokens documented by the crawlers' own operators, and the entire reading — domain by domain, with the exact line that decided each verdict — was released as open data, based on a scan performed on 9 November 2026.

What the audit found

Not every file cooperated: of the 700 sites, 407 robots.txt files could actually be retrieved, while 145 hosts refused the request outright. The author cautions that the sample consists of large, high-ranked sites, so the figures should not be read as a picture of small businesses.

Two numbers stand out. Among sites that block GPTBot, 80.3% did so with a rule naming GPTBot explicitly — a deliberate choice to keep pages out of a training corpus. Among sites that block OAI-SearchBot, which fetches a page at the moment someone asks a question in order to answer with it and link back, 43.6% never named it at all. Those blocks fell out of a User-agent: * rule aimed at every bot equally.

That is the core problem the post identifies: rules written to exclude training crawlers are also excluding answer engines that would have sent the site visitors.

Four ways robots.txt gets misread

The author traces most confusion to four readings of RFC 9309, the standard defining the robots.txt format, that are easy to get backwards.

First, the * group is a fallback, not something merged in. If any group names a specific crawler, the * group is ignored for that crawler entirely. A file that disallows /private/ for all bots but only /drafts/ for GPTBot therefore lets GPTBot into /private/. The inverse trap also bites: adding a GPTBot group to a file whose * group blocks half the site relaxes the policy for GPTBot unless those paths are repeated inside its own group.

Second, group selection is exact rather than a prefix match. A rule for Google-Extended does not govern Googlebot, and a rule for Googlebot does not govern Googlebot-Image. The author found that every parser handled this correctly, while human readers consistently misread a Google-Extended block as "I have blocked Google" — when it actually blocks only the product used for Gemini training and leaves the search index untouched.

Third, the longest matching pattern wins, and an Allow breaks a tie. Given Disallow: /blog/ and Allow: /blog/public/, the path /blog/public/x is allowed; when Allow and Disallow match at equal length, Allow wins. Line order plays no part, despite what many people assume.

Fourth, an empty Disallow means everything is allowed. A bare Disallow: is the documented way to express "no restrictions"; interpreted as a zero-length prefix, which every path would match, it would mean the opposite.

Open data and a provenance-aware checker

Alongside the audit, the author released a reference dataset of AI crawler user agents — the token as it appears in User-agent lines, who operates it, what it fetches pages for, and what its operator documents about robots.txt — available in JSON or CSV under a CC BY 4.0 licence, plus a free checker and rules generator.

One design choice distinguishes the checker: rather than returning a bare allowed-or-blocked verdict, it reports the deciding rule and its line number, since that is the only form of the answer a site owner can verify against their own file. The generator deliberately omits Googlebot and Bingbot, on the argument that a switch capable of de-indexing an entire site should not sit in a list alongside nine AI crawlers. The full 9 November reading, with per-suffix figures and deciding lines, is published openly, and the author invites anyone who re-runs it and gets different results to say so.

Why it matters

Robots.txt has become the primary lever publishers have for negotiating with AI companies, yet this audit suggests a large share of operators are pulling it without understanding its semantics. The consequences run in both directions: sites that would benefit from appearing in AI answers, which link back and send traffic, are silently excluded by blanket training-crawler blocks, while sites that name specific crawlers can accidentally reopen paths they closed for everyone else. As answer engines grow into a meaningful traffic source, an unexamined wildcard Disallow becomes a business decision made by accident — and with the dataset now public, that decision is finally auditable.

  • #robots-txt
  • #ai-crawlers
  • #web-crawlers
  • #open-data
  • #seo