Cloudflare lets sites block AI training while keeping search visibility

Cloudflare has introduced a bot control that breaks the link between search visibility and AI training. The setting, called Disallow AI Training, publishes a no-training instruction in a site's robots.txt while leaving mixed-use crawlers free to index the site for search. According to the Cloudflare blog, Apple, Google and Microsoft either honor the setting today or have committed to honoring it within a stated timeframe.

The problem mixed-use crawlers created

Some of the largest operators on the web run a single crawler that simultaneously builds a search index and gathers data for model training. Cloudflare names Applebot, Bingbot and Googlebot as examples. Because the two functions share one crawler, a site owner who refused one use automatically refused the other — stopping training meant disappearing from search results.

Cloudflare's own data shows how lopsided site owner preferences are: fewer than 1% of sites on its network block search bots, while 17% use some mechanism to block training. A single all-or-nothing switch did not match that reality.

How the new setting works

Disallow AI Training operates through a feature Cloudflare calls Bot Preference Sync, which writes the applicable no-training directive into robots.txt on the site's behalf. Mixed-use crawlers classified as Accountable remain allowed to crawl for search, while every other training crawler is blocked outright — including the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI, which Cloudflare notes can be blocked without affecting search at all.

The setting joins a revised menu of domain-level options: Allow, Disallow AI Training, Block on pages with ads, and Block. There is no ads-only variant of the disallow option, because robots.txt cannot enumerate a large and constantly changing list of ad-serving pages. There is also no disallow setting for agents yet; Cloudflare says it will revisit that as standards such as ai-prefs mature.

The Accountable designation

Enforcement rests on Cloudflare's network rather than robots.txt alone. As Cloudflare points out, robots.txt cannot identify who is crawling, work out why, or stop a crawler that ignores it. Cloudflare instead publishes the preference, verifies and classifies the crawler, blocks those that disregard it, and publishes what each operator actually does on its Radar service.

To recognize operators that respect these choices, Cloudflare created a designation called Accountable. To qualify, an operator must offer — now or under a time-bound commitment — an opt-out from AI training via robots.txt or an equivalent standard, an opt-out from AI summaries, URL-level visibility into which pages were used for training together with metrics on how content appeared in search, and an assurance that opting out of training leaves ordinary search results untouched. Apple, Google and Microsoft all meet the bar, combining current capabilities with dated commitments for features still in development. Cloudflare adds that it has been talking directly with crawler operators since July, and that almost all agreed site owners deserve control and transparency over how their content is used.

What changed on September 15

Alongside the new setting, Cloudflare changed how existing controls behave. Block and Block on pages with ads now apply to mixed-use crawlers, meaning either choice affects search as well as training. The older "Block AI Bots" toggle is deprecated in favor of the separate Search, Training and Agent controls, and Managed Robots.txt is being retired in favor of Bot Preference Sync. Disallow AI Training also becomes part of the recommended configuration for certain new domains.

Most existing customers need to do nothing, since preferences carry over automatically. Domains that never used the granular controls will be migrated based on their legacy "Block AI Bots" setting. Site owners who want mixed-use crawlers gone entirely must now say so explicitly by selecting Block, which stops Applebot, Bingbot and Googlebot from reaching the site — search included.

Why it matters

Much of the open web is funded by advertising, subscriptions or direct reader relationships, and those models only pay when someone actually arrives at a site. Training crawlers return no traffic. Cloudflare's setting gives publishers a middle position that previously did not exist: fully visible in search while off-limits to model training. It also shifts enforcement from voluntary robots.txt compliance to network-level verification with public reporting, a pattern that could shape how crawler accountability works well beyond Cloudflare. The next frontier is already in view: Cloudflare requires an AI summaries opt-out for Accountable status, and says that by early next year it aims to let sites control how much of their content appears in summaries — set once in one place rather than negotiated with each operator separately.