Python script checks whether servers really serve AI crawlers that robots.txt allows

A developer writing on dev.to has shared a compact Python utility that addresses a quiet failure mode in AI crawler management: robots.txt states what a site permits, but says nothing about what the server actually delivers. A firewall or CDN bot rule can hand a 403 to a crawler that robots.txt explicitly welcomes, and nothing in the file will reveal it.

Two checks in one pass

According to the post, the script accepts a base URL and an optional path, then evaluates four AI crawlers: OpenAI's GPTBot, OpenAI's OAI-SearchBot, Anthropic's ClaudeBot and Perplexity's PerplexityBot. For each bot it does two things: it parses the site's robots.txt with the protego library to determine whether that bot may fetch the path, and it issues a live request to the path carrying the bot's real user agent string, recording the HTTP status and response size.

The output is a table in which any row gets flagged where robots.txt permits the bot but the response is not a 200, or where the bot is disallowed outright. The process exits with status 1 when anything is flagged, which makes it usable as a CI step.

The author validated it against a local server that allowed everything in robots.txt but rejected any request using the OAI-SearchBot user agent, the kind of behaviour a strict WAF rule produces. All four rows read allow on the robots side, yet OAI-SearchBot received a 403. That contradiction between the two layers is what the tool is designed to surface.

A parsing choice with a reason

Rather than Python's built-in urllib.robotparser, the script leans on protego. The post explains why: the standard library module evaluates directives sequentially and stops at the first match, so an early blanket Allow: / can mask a later, more specific disallow. Protego implements the longest-match semantics of RFC 9309, which the author considers closer to how real crawlers interpret the file.

What the result does and does not prove

The post is candid about limits. Requests originate from your own IP, and many CDNs authenticate genuine bots by IP range or reverse DNS, so a 200 from this test does not confirm the actual crawler receives one; a 403, by contrast, is treated as a meaningful warning. A 200 also does not guarantee readable content: if the article text only appears once JavaScript runs, a crawler that never executes JavaScript misses the content entirely, and the byte count should be compared against a normal browser fetch. The definitive record, the author notes, sits in server logs filtered by user agent and status code.

Handling a flagged row

For flagged rows the post recommends inspecting WAF or CDN bot-management rules for the user agent token, then admitting the crawlers you want via verified IP ranges or the provider's verified-bot category rather than relaxing rules for all traffic. Re-run the script afterwards and confirm in the logs once a real crawl takes place. One further detail: when robots.txt contains multiple named groups, each blocked path must be repeated in every group, because a crawler reads only the group matching its own name and ignores the wildcard group.

Why it matters

Many publishers deliberately open robots.txt to AI crawlers to stay visible in AI-generated answers and search products. This post demonstrates that permission alone guarantees nothing: the enforcement layer in front of the origin can silently overrule it. A short per-bot check turns an obscure debugging exercise into something a pipeline can verify on every deploy, and it makes a broader point worth remembering: robots.txt is advisory, the WAF is enforcement, and the two can disagree without either one reporting an error.