deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

robots.txt cannot block AI crawlers; edge and server rules actually can

RFC 9309 defines robots.txt as voluntary guidance rather than access control. A dev.to explainer maps which AI bots honor it and how to enforce a real block.

robots.txt cannot block AI crawlers; edge and server rules actually can

A dev.to explainer opens with an uncomfortable reminder for site owners: robots.txt blocks no one. The file, formalized by the IETF as RFC 9309 in 2022, is a set of stated preferences that crawlers are asked, not required, to follow. The spec itself puts it plainly, saying its rules "are not a form of access authorization."

A convention, not a control

Under RFC 9309, a cooperative crawler downloads robots.txt separately and then decides for itself whether to comply. Nothing in the protocol stands between the crawler and your server, so compliance is entirely the operator's choice. Cloudflare's documentation, cited in the post, makes the same point: the directives are voluntary and a crawler can disregard them. Well-behaved bots obey because their operators choose to, which worked well enough when the main audiences were the major search engines. AI crawlers have changed the calculus.

AI labs split crawling by purpose

According to the post, which draws on OpenAI's crawler documentation, OpenAI runs three distinct user agents. GPTBot gathers content that may feed model training. OAI-SearchBot determines whether a site surfaces in ChatGPT search results, and OpenAI says an opted-out site disappears from those answers while it can still appear as a navigational link. ChatGPT-User fetches pages when a person asks ChatGPT something, and because those requests are user-initiated, OpenAI notes that robots.txt rules may not apply. The three settings are independent, and search behavior takes roughly a day to reflect a robots.txt change.

Anthropic mirrors the structure with ClaudeBot for training data, Claude-SearchBot for search quality, and Claude-User for question-driven fetches. Anthropic says its bots honor robots.txt, supports the non-standard Crawl-delay directive, and expects rules to be set on every subdomain you want covered. As of October 30, 2026, Anthropic also began recognizing a machine-readable "no AI training" signal derived from IETF draft work, rolling out gradually, though a robots.txt rule naming ClaudeBot directly takes precedence over it.

The upshot, the author argues, is that "block all AI" and "allow all AI" are false binaries. A site can disallow the training bots while leaving the search bots alone, keeping its pages citable in chatbot answers, a distinction many site owners miss.

Enforcement belongs on the request path

When a refusal has to hold, the post argues the block must live where the request lands: a CDN or WAF rule, the origin server's configuration, or an application login. Cloudflare's AI Crawler and Agent access policies are the clearest commercial example, letting administrators set an action per behavior (Search, Agent, Training), with the Block action stopping matching traffic at the edge. Two wrinkles are worth knowing: the Training behavior also offers a Disallow action that only writes robots.txt directives, which stay voluntary unless the optional edge-blocking setting is enabled, and on new domains all three behaviors default to Allow, meaning inaction is itself a policy.

Sites without Cloudflare can deny requests by user agent in the server configuration, or block the IP ranges the labs publish; OpenAI maintains a JSON list per bot. Anthropic's documentation adds a caveat: hard IP blocks can prevent its crawler from even reading robots.txt, which makes them unreliable as a standalone opt-out. The author's recommendation is to keep robots.txt as the stated preference and treat request-path rules as the enforcement layer behind it, not a replacement.

llms.txt covers the opposite goal

Exclusion is only half the decision. The llms.txt proposal, first published in 2024 and updated to v2 in 2026, describes a markdown file that gives agents concise, curated information plus links to markdown renditions of key pages. It remains a proposal rather than a standard, but the post notes that thousands of sites now publish one, and OpenAI, Anthropic and Gemini each maintain their own.

Why it matters

Training datasets, AI search answers, and user-triggered fetches are now governed by separate bots with separate rules, so a blanket Disallow is a blunt instrument that also erases a site from chatbot answers. And since RFC 9309 makes robots.txt advisory by design, any owner whose no must actually bind has to pair the file with enforcement they control, whether an edge rule, a server rule, or a login wall. The post reduces the decision to three questions: should your content enter training data, appear in AI search answers, and remain fetchable on user request? Answer those, write one robots.txt group per bot, and none of it is permanent, since the file can be changed tomorrow.

  • #robots-txt
  • #ai-crawlers
  • #web-standards
  • #cloudflare
  • #llms-txt