One blog's 30-day AI crawler log: 4,040 requests, half for training, zero ChatGPT citations

A month of crawler traffic, counted

A developer has published a 30-day audit of the AI crawler traffic reaching a personal blog with eight posts, based on Cloudflare's free-tier AI Crawl Control dashboard. In the window ending 20 October 2026, crawlers that Cloudflare recognises made about 4,040 requests to the site, according to the post on dev.to. Roughly 3,540 of those requests succeeded.

Ten companies, with Google out front

Google sent the most traffic at 961 allowed requests, followed by Anthropic with 680, OpenAI with 508, Apple with 377 and Microsoft with 309. Huawei's PetalBot (238), Meta (212), Common Crawl (138), Amazon (128) and the Internet Archive (111) completed the top ten. Cloudflare groups each company's bots together, so OpenAI's figure covers GPTBot plus its siblings.

Googlebot appears in an AI category because Google's AI answers draw on the search index it builds, the author explains. Perplexity, meanwhile, did not make the top ten despite being the one engine that cited the site in the author's tests.

An earlier snapshot from 13 October showed about 3,000 requests over the previous 30 days. The author cautions that the two windows overlap by 23 days, so this is not a clean month-on-month comparison, but the direction of travel is clearly upward.

Training dominates, assistant fetches barely register

Cloudflare sorts crawlers by purpose. On this site, about 1,720 successful requests — roughly half — were bots collecting pages to train models. Another 1,190, about a third, were building a search index, and 482 were building an AI search index. Archivers accounted for 110 requests. The smallest and, for the author, most telling category was the last one: 26 requests, under one percent, were an AI assistant fetching a page because a person had actually asked it something.

The sitemap is the most-read file

Crawlers spent more effort discovering what exists than reading it. The four sitemap files together received 687 requests, while the single most-fetched post got 66. In all, 271 distinct paths were requested, totalling about 74 MB of transfer, with Googlebot accounting for 18.63 MB and GPTBot for 11.24 MB.

In a slightly earlier 30-day window, crawlers fetched the site's llms.txt zero times and its robots.txt 605 times. The author's conclusion: a correct, current sitemap matters far more than any file created specifically for AI systems.

Scanners wearing a crawler's name

Not everything counted as an AI crawler came from an AI company. Eight of the ten most-requested nonexistent paths were files that hold credentials or configuration — among them .env.local, terraform.tfvars and .git/config — and every request returned a 404. The author's reading is that these were security scanners spoofing a crawler's user-agent string, and because Cloudflare's free plan identifies crawlers by the name they declare, they were logged as AI traffic.

Being crawled is not being cited

The sharpest finding is the gap between crawling and citation. OpenAI's crawlers made 508 requests in the month, yet when the author asked ChatGPT twelve questions the site could answer, on 17 October, it cited the site in none of them. Bing Webmaster Tools' AI Performance report showed zero citations over three months despite 309 BingBot requests. Google Search Console recorded 14 impressions in Google's AI features over 28 days. Perplexity, barely visible in the logs, cited the site in four of eight answers. Crawl volume, the author concludes, does not predict citation.

There is also a measurement blind spot: Cloudflare's referral-traffic column showed a dash on every path, which on the free plan means the figure is simply not counted rather than zero.

How to run the same check

Sites already on Cloudflare, free plan included, can open AI Crawl Control from the dashboard. Overview shows totals by company, Crawlers lists each bot with its purpose and allow/block state, and Metrics provides charts with a maximum 30-day range, including the most-crawled paths. robots.txt requests appear on the separate Signals page. Sites not on Cloudflare can count crawler names in raw access logs instead.

Why it matters

Arguments about AI bots and publishers usually run on anecdote. This dataset, small as it is, quantifies the picture for an ordinary site: most AI crawler traffic is speculative bulk collection for training and indexing, requests triggered by real people are vanishingly rare, and hundreds of crawls can still yield zero citations while a low-volume engine cites freely. It also shows that AI crawler statistics can hide plain security scanners, so raw log counts overstate genuine AI interest. With Cloudflare's free dashboard, any webmaster can produce comparable numbers in minutes and make blocking decisions on evidence rather than guesswork.