Cloudflare and OpenAI pilot surfaces hundreds of millions of pages for ChatGPT search

Cloudflare and OpenAI share early results from AI discovery pilot

Cloudflare and OpenAI began a research pilot in July to test whether signals from Cloudflare's network could help ChatGPT users find useful content more efficiently. In a blog post published October 15, Cloudflare reports that the experiment has already allowed OpenAI's search crawler to discover and index hundreds of millions of pages that were absent from its index before, expanding the pool of sources ChatGPT Search can draw on, cite and link back to.

The pilot ran across a participating cohort representing roughly 1% of Cloudflare's total network. Cloudflare describes the setup as a feedback loop: its network surfaces signals pointing the crawler toward pages worth visiting, and OpenAI then independently evaluates and ranks those pages for relevance and helpfulness. The post adds that the effort is meant to complement OpenAI's GPT-6 and Intelligent UI work, which aims to make ChatGPT a better place to shop, learn and find answers with more visual, interactive experiences.

Discovery is being redefined around AI

Cloudflare illustrates the problem with everyday examples: someone repairing a washing machine needs the manual for the exact model, a passport applicant needs current requirements and fees, and a diner needs a restaurant's latest menu and hours. These pages can matter even when few other sites link to them, which is precisely where link-driven discovery falls short. In the older model of the web, visibility meant reaching the top of a results page, and the human visitor judged the value from there. As more people delegate that journey to AI, the system itself now has to locate the right source, judge its usefulness, and connect the person to it through an answer, a citation or a visit to the site.

Signals that reduce wasted crawler work

A URL by itself reveals little, so the pilot examined whether an address leads to a page people can actually access, whether that page offers anything distinct, and what it might help someone do. Cloudflare says it learned how easily automated traffic can make nonexistent pages look interesting: bots repeatedly request files that are not there, some sites redirect nearly any address to their homepage, and some URLs resolve to copies of identical content. Recognizing these patterns lets the crawler skip them before fetching anything, saving resources on both sides.

The companies are also testing signals that flag potentially harmful pages. A legitimate site can carry a single compromised page, such as a fake login form planted by an attacker, and Cloudflare argues that flagging that one page rather than the whole domain keeps people safe without penalizing the site owner. Early results suggest these signals can identify pages that should stay out of the index. Separately, change signals indicate when a page merits another visit and when it does not, addressing the cost of repeatedly downloading unchanged content.

Cloudflare acknowledges a trade-off: discovering new pages can initially increase crawling while those pages are evaluated and indexed. The stated goal is to stay current while shrinking the overall effort involved, which means less avoidable load for Cloudflare customers and more of OpenAI's search resources spent on information that genuinely helps people.

Permission and training boundaries

Domain owners' preferences remain in force throughout, Cloudflare emphasizes: a signal about a page does not grant permission to access it. The research draws on metadata about the web for search and discovery rather than sharing page content for model training, and site owners can use Cloudflare's AI Crawl Control to understand and manage how AI systems access their sites. The next phase will refine the selection of pages, test faster and more reliable change signals, and measure what happens after content is discovered, including whether better discovery translates into tangible business outcomes the companies intend to keep sharing.

Why it matters

For publishers and anyone working in SEO, this is one of the clearest illustrations yet that discoverability is being rebuilt around AI systems rather than classic search rankings. Pages that never accumulated backlinks, from manuals to local business details to first-hand product accounts, can now reach ChatGPT users if AI crawlers can find them, and ChatGPT Search links back to the original sources. The efficiency gains matter too: smarter filtering means fewer pointless bot visits and lower server load for site owners. Just as importantly, the stated guardrails, covering metadata-only use, no training on shared content and access still governed by the domain owner, sketch the terms on which publishers are invited to participate. The open question, which the next phase is designed to answer, is whether AI-driven discovery actually converts into traffic and revenue.