More than half of internet traffic is now non-human, Cloudflare says

More than half of the traffic crossing Cloudflare's network now comes from software rather than people, the company reported in a September 30 blog post. It is the first time automated requests have outnumbered human ones on the network, and Cloudflare frames the shift as the arrival of what it calls a "second audience": AI agents acting on behalf of people, which sites can neither ignore nor serve with tools built for human visitors.

Traffic nearly doubled, revenue per request fell

At the end of 2024, Cloudflare handled an average of 63 million HTTP requests per second. According to the company, that figure has since climbed to almost 115 million, with peaks above 150 million, and daily requests from AI agents grew by more than 1,700% over the past year. Human traffic has not disappeared; machine traffic has been layered on top of it.

The problem for site owners is that the new arrivals often do not pay. Answer engines read a page and hand the reader a summary, consuming bandwidth, compute and origin capacity without producing a referral, an ad impression or a subscription. Cloudflare says some of the most heavily crawled categories — retail, computer software, IT & services and financial services — have seen human traffic fall by as much as 40% in under a year. The share of crawler requests whose stated purpose is AI training rose from 22% in spring 2025 to 52% by June 2026.

Agents are not simply crawlers

The fastest-growing automated traffic is no longer training crawlers but agents: software fetching pages for a person, typically because that person asked a chatbot a question. According to Cloudflare, agent traffic follows human routines, with a weekly rhythm and a dip over the summer holidays, and it grows with the number of questions people ask rather than with how much a site publishes. The company's argument is that an agent booking a table or comparing insurance quotes is a customer, just not a human one, and that turning it away may mean turning away the person who sent it.

From a single switch to per-purpose controls

Cloudflare's first instinct, it admits, was to block. Last year it recommended blocking AI training crawlers on new domains, but it now says a blanket refusal is too blunt an approach. In July it replaced its single "block AI bots" toggle with separate Search, Agent and Training controls, available on every plan including Free; its data shows fewer than 1% of sites block search crawlers while 17% block training. On September 15 it shipped Disallow AI Training, which keeps a site indexed while instructing operators not to train on its content, a commitment Apple, Google and Microsoft have made.

Identification tools accompany the policy change. AI Crawl Control, Business Insights and BotBase show who crawls a site, what they take and which URLs they want most, while Web Bot Auth lets operators including OpenAI, Google and AWS cryptographically sign their requests. Cloudflare says it now sees more than 500 billion verified bot requests per week. Mixed-use crawlers that both search and train remain awkward: refusing one use means refusing the other.

Pay Per Use as the bridge

On payment, Cloudflare points to the licensing market — more than 50 publisher-AI deals signed since 2023 — as proof that content has value, while noting that nearly all are bespoke agreements between large publishers and large AI companies. Its Pay Per Use system is meant to reach everyone else. Verified crawler-buyers define what counts as a use and what they will pay; publishers opt in and can opt out; the buyer reports each use, and Cloudflare checks the reports, bills the buyer and pays the publisher. Publishers also see what was used and what it earned, visibility that licensing deals rarely provide.

Why it matters

The search-era bargain — let engines crawl your site and they send visitors you can monetize — has broken, according to Cloudflare. Sites that treat every bot as identical risk either blocking paying customers or being crawled for free, while those that can identify, set terms for and charge agents gain a new revenue channel. Cloudflare's controls and marketplace are one company's answer, but the underlying shift, with software now the majority audience, will affect SEO, ad economics and publishing strategy regardless of whose tools win.