Meta bots took 52% of one site's AI crawler traffic, none of it user-initiated

A developer who has been instrumenting a small WordPress site reports that Meta's crawlers accounted for more than half of all AI bot traffic over a thirty-day window — and that not one of those requests came from a person asking an AI assistant a question.

The figures appear in a post on dev.to, where the author describes months of logging AI crawler requests and sorting them by purpose. In the thirty days ending September 11, the site recorded 11,818 crawler visits, 8,979 of which landed on pages carrying written content.

One vendor, more than half the traffic

Meta took 52.3 percent of all tracked AI crawler requests — a larger share than every other vendor combined, which together accounted for 47.7 percent. Apple followed at 12.0 percent, then Anthropic at 9.6 percent, Perplexity at 8.3 percent, OpenAI at 7.9 percent and ByteDance at 5.8 percent. Common Crawl, Amazon and Google filled out the tail at 2.1, 1.5 and 0.5 percent respectively.

The share column alone, the author argues, is not what determines whether any of this activity can benefit a site. The purpose column is.

Meta's three AI crawlers, and what each did

Meta documents three AI-facing crawlers. meta-externalagent collects material for training foundation models and indexing content for products. meta-webindexer crawls to improve Meta AI search quality, the component responsible for accurate citations. meta-externalfetcher fetches individual links when a user requests one — the signal that a live person wanted that page seconds earlier.

Over the thirty days the split was lopsided: 6,027 requests from the training crawler, 155 from the citation-quality crawler, and zero from the user-initiated fetcher. That works out to roughly thirty-nine training requests for every one aimed at answer quality, with an empty row where a human-driven fetch would appear.

Five other vendors — OpenAI, Perplexity, Anthropic, DuckDuckGo and Mistral — also operate user-initiated fetchers, and all five showed up during the period. Measured that way, Perplexity's traffic on the site was 93 percent user-initiated, OpenAI's 21 percent and Anthropic's 6 percent. Meta's was zero.

The author is careful about interpretation: a training crawler doing training work is not misconduct, and Meta states its purposes openly. But the mix, the post suggests, indicates what a company currently wants from a site's writing, and the mixes across vendors are strikingly different.

A measurement the author cannot fully verify

The post carries an unusual caveat. Meta publishes no IP ranges and no reverse-DNS scheme for these crawlers, so there is no way to confirm that any of the 6,182 requests identifying as Meta actually originated at Meta. The author considers forgery unlikely — the volume is steady and the behaviour consistent — but states plainly that the site's largest reader is also its least verifiable one.

Other blind spots are acknowledged. Pages served from cache are never recorded, so every count is a floor rather than a total. AI clients running as desktop or phone apps send no referrer, so humans who do follow a link can appear in analytics as direct traffic. And the site is a single data point: a Japanese-language publication about a city in the Philippines, with one subject, one language and one publishing rhythm.

Disclosure worth noting: the author builds the measurement instrument, a free WordPress plugin called AILYS Lens that runs all calculations locally on the server. In an earlier post using the same instrumentation, the site logged roughly ten thousand monthly bot reads against two recorded human arrivals — one from ChatGPT and one from DuckDuckGo.

Why it matters

For anyone running a website, the composition of AI crawler traffic — not just its volume — indicates whether a vendor's presence is purely extractive or carries a plausible path to referral traffic. On this site, the crawler that could route readers back ran at about two percent of the volume of the crawler that takes content away, and the user-driven fetcher never fired once.

That asymmetry is directly relevant to decisions about robots.txt rules, rate limiting and blocking, because it frames what tolerating crawler load buys in return. The verification gap compounds the problem: the vendor generating the most requests is also the one site owners can least authenticate, which makes fine-grained access control harder to enforce.

One site and one month is not a census, and the author says as much. But the method is cheap to replicate, and the underlying question — who is reading your site, and why — has a factual answer that any operator can start collecting in about a week.