Cloudflare splits AI bot controls so sites can block training but keep search indexing

What Cloudflare announced

Cloudflare has introduced new AI-crawler controls that untangle search indexing from AI model training. According to dev.to, the centerpiece is a setting called Disallow AI Training, which publishes a no-training directive in a site's robots.txt while leaving eligible mixed-use crawlers free to continue indexing pages for search. The change is part of an update dated September 15, 2026 covering how Cloudflare handles crawlers that serve more than one purpose.

Three policies replace one switch

The update retires the old Block AI Bots toggle, a single button that could not distinguish between a crawler indexing content for a search product and one harvesting pages to train a model. In its place, Cloudflare now offers three separate policy categories:

• Search, for crawlers that index content for search
• Training, for crawlers that collect content for AI model training
• Agent, for AI agents that access websites to carry out tasks

Under the defaults effective September 15, 2026, Training and Agent activity can be blocked while Search stays allowed. Cloudflare also states that mixed-use crawlers, those combining Search and Training functions, are caught by any configuration that blocks training. As dev.to reports, the intent is to stop a crawler from using search-related access as a side channel for collecting training material.

robots.txt stays in step

To keep a zone's robots.txt consistent with whatever policy an owner picks, Cloudflare is replacing Managed Robots.txt with Bot Preference Sync, which automatically aligns the published robots.txt with the AI bot policies configured in the dashboard. The feature is enabled by default for new customers, and existing customers will be guided through a migration. Cloudflare is also differentiating its onboarding recommendations for ad-supported and non-ad-supported sites, on the reasoning that different business models lead to different choices about discovery and automated access.

A signal, not a guarantee

A robots.txt directive is a statement of preference, not an enforcement mechanism. Cloudflare says Apple, Google and Microsoft have committed to honor the Disallow AI Training setting within a stated timeframe, which matters because a no-training directive only has practical effect if major crawler operators respect it. Site owners should read the setting as a clear policy signal rather than a technical barrier against every crawler on the internet.

What site owners should check

dev.to suggests starting with a strategic decision: whether search visibility, training restrictions and agent access should be treated as three separate permissions. Owners on Cloudflare should then review their AI Crawl Control and Bot Management settings, check whether an existing Block AI Bots configuration needs migrating, and confirm that Bot Preference Sync is publishing the intended robots.txt and that the onboarding preset matches the site's business model.

Why it matters

Until now, a site that objected to AI training faced a blunt trade-off: block AI bots wholesale and risk degrading indexing by mixed-use crawlers, or allow everything. The new split lets a business keep product pages, documentation, guides, pricing and support content discoverable in search while publishing an explicit refusal of training use. For publishers whose content is a core asset, that turns an all-or-nothing permission into a genuine access policy. The caveat is compliance: the value of the directive still rests on crawler operators honoring what robots.txt says, and on site owners reviewing their configuration rather than setting it and forgetting it.