Cloudflare's Disallow AI Training setting blocks model training without touching search indexing

Cloudflare has introduced a Disallow AI Training setting that lets website owners keep their pages out of AI model training data while leaving search indexing alone. According to dev.to, summarizing Cloudflare's announcement, the control sits in Cloudflare's Bot Management and AI Crawl Control suite and targets one of the more awkward trade-offs in web publishing today: sites that wanted to stop AI companies harvesting their content often had to accept reduced search visibility as a side effect.

The problem it solves

Web crawling has split into distinct activities. A bot might build a conventional search index, scrape material to train or fine-tune a model, or fetch a page on a user's behalf, as chat-based retrieval tools do. Cloudflare's earlier Block AI Bots option did not separate these cases cleanly, so a broad block could ripple out to search discovery. The new setting expresses a narrower preference: no training, yes indexing.

How it works

The setting writes a training opt-out into the site's robots.txt using a mechanism Cloudflare calls Bot Preference Sync. Crawlers that Cloudflare classifies as Accountable, and that respect the directive, can keep indexing a site for search while being barred from using its content to train models.

The classification rests on three signals, as dev.to outlines: Search, for crawls that build search indexes; Training, for crawls aimed at model training or fine-tuning; and Agent, for human-directed or bot-assisted access such as retrieval on behalf of a chat user.

Per the report, most training crawlers run by Amazon, Anthropic, Meta and OpenAI will be blocked from training under the preference. Applebot and Googlebot are named as Accountable crawlers expected to honor the no-training signal while continuing to index for search, with Bingbot due to gain comparable support in early 2027. Cloudflare says Apple, Google and Microsoft have committed to honoring the setting.

Rollout and migration

Cloudflare is retiring the legacy Block AI Bots control and migrating existing customers to the granular Search, Training and Agent options. Newly added domains get one of two onboarding presets depending on whether the site monetizes through advertising. Cloudflare also plans to phase out Managed Robots.txt in favor of Bot Preference Sync.

The wider roadmap, according to dev.to, includes per-URL transparency and metrics through Cloudflare Radar, plus forthcoming AI Summaries controls that begin with an opt-out for AI-generated summaries and expand next year.

Limits to keep in mind

The system only works if crawler operators accurately declare what they do with fetched content and then honor the published preference. It also does not collapse every AI access question into a single switch: search visibility, training use and agent-style retrieval remain separate decisions, and Cloudflare's controls make those choices explicit rather than making them for you. Dev.to suggests teams review their crawler policy deliberately — which content must stay searchable, whether training on public pages is acceptable, and whether agent retrieval deserves different treatment from indexing — rather than flipping one checkbox and moving on.

Why it matters

For publishers, e-commerce sites and knowledge bases, search traffic and AI training were previously coupled: objecting to one meant risking the other. Separating them makes content-access policy genuinely negotiable, and it pushes robots.txt toward treating training preferences as a first-class concern rather than a byproduct of indexing rules. The catch is dependence on operator cooperation — the model functions because Google, Apple and Microsoft have agreed to take part, and its credibility rests on whether that cooperation is verifiable rather than merely promised. If the Bingbot timeline and the planned Radar metrics arrive as described, site owners will gain ways to check compliance instead of trusting it blindly, which would make this framework durable rather than just convenient.