OpenAI, Google, Microsoft and Anthropic back shared robots.txt signal to opt out of AI scraping

OpenAI, Google, Microsoft and Anthropic have committed to a shared standard that lets publishers opt out of AI scraping with a single signal, according to The Verge, which credits Search Engine Roundtable with the earlier report. Instead of stating their preferences to each AI vendor one by one, website owners will be able to declare them once and have all four companies honour the setting.

A policy token inside robots.txt

The mechanism, a "policy token," was proposed by Kevin Kelley, a technical lead for standards at Anthropic, and is being advanced through the Internet Engineering Task Force. It lives inside a website's robots.txt file, the long-standing convention sites use to tell crawlers what they may and may not access, whether for search indexing, archiving or AI training.

Because each AI company currently reads those signals its own way, publishers have had to configure their preferences vendor by vendor. A token that all the major players recognise collapses that patchwork into one instruction. All four firms say that when their crawlers encounter the token, they will stop accessing the site in question.

Staggered rollouts and Microsoft's extra granularity

Adoption is not happening simultaneously. The Verge reports that Anthropic has already implemented the signal, OpenAI expects to do so in the coming months, and Google and Microsoft plan to add support next year.

Microsoft is also offering finer-grained controls than its peers. Publishers will be able to exclude themselves from AI training and from AI grounding — the practice of directing a model to a verified source of information to reduce hallucinations — as separate choices. The company also notes that a site's preferences will not affect its presence in Bing's search results, keeping ordinary search indexing apart from AI-specific usage.

Pressure has been building on AI crawlers

The move comes as publishers wrestle with a steady stream of AI crawlers feeding their content into models. According to The Verge, other responses include the Really Simple Licensing (RSL) standard, which lets publishers charge crawlers that visit their sites, and Cloudflare's decision to block AI crawlers by default. A single opt-out honoured across the industry could reduce the need for such blunt measures.

Why it matters

robots.txt has never been enforceable; it works only because crawlers voluntarily comply. Getting the four most significant AI vendors to back one IETF-tracked signal turns a fragmented, goodwill-based system into something publishers can actually rely on — at least for these companies.

The caveats are real. Until Google and Microsoft ship support, protection will be uneven, and crawlers outside this group can still ignore the token entirely. The arrangement also remains opt-out rather than opt-in: publishers must act to protect their content, and those who do nothing stay fair game. Microsoft's decision to separate training from grounding, and to insulate Bing results from any of it, points to the next debate — whether search, grounding and training are genuinely distinct uses of a publisher's work, and who gets to draw that line.