Spoofing ClaudeBot took one curl command, and the crawler dashboard believed it

One line of curl became three ClaudeBot requests

A developer posting on dev.to has demonstrated how easily AI crawler analytics can be manipulated. Using a single curl command and a user agent string lifted from Anthropic's own documentation, the author sent three requests to an article on their own site from a laptop in Japan over ordinary home broadband. The site's crawler dashboard then incremented its Anthropic ClaudeBot training counter from 1,698 to 1,701 and refreshed the last-seen timestamp to 09:22.

None of those three requests came from Anthropic. The originating address appears on no crawler list anywhere. As the author puts it, the tool was working exactly as designed: it records what arrived and what each request called itself. A user agent string is a self-declaration, and anyone can type one.

The verification gap

To quantify the problem, the author examined thirty days of traffic on davaonline.net ending September 4, 2026: 13,491 crawler requests, 10,390 of which landed on pages with actual writing. Only about 2,200 — roughly one in six — could be confirmed against IP ranges the vendors themselves publish. Everything else is logged on the strength of a string the request supplied.

According to the dev.to post, the unverified majority splits into three distinct failures:

• 6,947 requests (51%) came from vendors that publish no verification method at all.
• 2,226 (16%) were unverifiable only because the author's bundled copy of vendor ranges had gone stale.
• 4,318 (32%) were checked against a list the author held, and the addresses did not match.

Vendors that publish nothing

Meta, ByteDance and Amazon publish no machine-readable IP ranges or reverse-DNS scheme for the crawlers observed on the site, which makes verification not difficult but impossible. The single largest crawler in the sample, meta-externalagent at 5,728 requests, sits in this group. Amazon comes close to publishing a list for Amazonbot, but the addresses only appear after JavaScript runs, so automated tooling cannot consume it. ByteDance sets a trap of its own: an obvious candidate URL returns HTTP 200 while the body is a 404 page, which fools anyone checking status codes.

Stale snapshots

The second failure belonged to the author's own tooling. Anthropic began publishing an IP range list on August 18 and Common Crawl followed on August 11, both after the July 20 snapshot shipped with the analytics plugin. The instrument kept reporting the old world, classifying 2,226 requests as unverifiable when they were merely unverified. A September 4 fix raised the number of verifiable crawlers from eleven to fifteen and also exposed a Google crawler that had been listed as verifiable for months even though its ranges were never bundled. The lesson drawn on dev.to: verifiability is not a property of a crawler but of how fresh your copy of somebody else's list happens to be.

Zero matches for Perplexity

The third failure is the hardest to wave away. Perplexity publishes IP ranges, and the author had them. Yet of 1,087 requests over thirty days identifying as Perplexity-User (984) or PerplexityBot (103), not one arrived from an address on Perplexity's own list. OpenAI keeps its list current — ChatGPT-User ranges were refreshed on August 14 — but only 6% of 887 requests claiming that identity matched. For comparison, Applebot verified at 98%, GPTBot at 81%, and GoogleOther at 100%.

The post lays out four explanations, none of which can be excluded: the requests are spoofed by parties who benefit from names site owners are told to allow; the published lists are incomplete, with Perplexity's user-fetch list at four ranges untouched for eleven months; legitimate fetches may run outside published infrastructure; or the author's hourly verification job never processed them, since it works in bounded batches and discards entries older than a day. The dashboard cannot distinguish "checked and failed" from "never checked", a defect the author concedes.

What the numbers still show

Verified counts form a lower bound on genuine crawler activity, never an upper one, because every failure mode converts real visits into unconfirmed entries. An unverified tally does not imply bad faith, since most of it reflects vendors who publish nothing at all. The gap is most meaningful per vendor: a company that publishes ranges yet matches none of its claimed traffic is saying something, while a company that publishes nothing is saying only that.

Why it matters

Decisions about blocking, allowing, licensing and metering AI access all rest on crawler identification, and so do public claims about which company trains on what. If an identity can be forged in one line of curl, and most major vendors offer no way to check one, then published crawler statistics — including attribution of training traffic to specific firms — cannot be taken at face value. Until IP-verified counts become standard practice, any argument built on raw user-agent numbers is an argument built on self-reporting.