Cloudflare AI logs showed half of one site's ChatGPT-User traffic scanning for secrets

A failure rate that pointed the wrong way

A developer writing on dev.to recently switched on Cloudflare's AI Crawl Control, which had been quietly gathering data all along. The first 24-hour window it surfaced showed 1,890 requests attributed to AI agents, with 483 of them failing — a 63 percent rise in failures compared with the previous day. The busiest entry was ChatGPT-User, the agent OpenAI documents as fetching a page when someone asks ChatGPT or a Custom GPT a question, rather than crawling on a schedule. It was credited with 859 requests, and only 412 of those returned a 200 status.

According to the post, the author's first explanation was reassuringly mundane: documentation slugs had been migrated more than once, so stale URLs throwing errors would account for the failures, and a redirect map would solve it. Then the actual failing paths were inspected.

The failing paths looked like a vulnerability scan

The requests were hitting endpoints such as /id_rsa, /terraform.tfstate, /.env.prod.bak, /elmah.axd, /@fs/proc/self/environ, /.github/workflows/deploy.yml, /.claude/settings. and /api/config. Of the 483 failures, 447 matched this pattern, and every one of them carried the ChatGPT-User user agent. These are textbook targets for automated scanning: private SSH keys, Terraform state files that frequently embed credentials, environment backups, CI pipeline configuration and framework-specific paths known to leak information.

Every probe returned a 404 and nothing was exposed. As ordinary internet background noise, the post notes, any public site receives this kind of traffic hourly. What made it notable was the channel it arrived through and the name attached to it.

User-agent strings are not proof of origin

The author's second correction concerns the label itself. Cloudflare's documentation, quoted in the post, states that on the free plan AI Crawl Control identifies AI crawlers from their user-agent strings — and a user agent is simply a string the client chooses to send. The defensible version of the finding is therefore narrow: 859 requests claimed the ChatGPT-User name, 447 of them probed for secrets, and 412 received 200 responses. Whether any request actually originated from OpenAI is a question a user-agent label cannot answer. The author also retracted an earlier draft claim that the 412 successful requests were genuine fetches, since nothing in the data supports that either.

What verification is possible

The post lists ways to separate verified traffic from merely claimed traffic without an enterprise contract:

• OpenAI publishes IP ranges per agent, including a dedicated list for ChatGPT-User, so logged addresses can be checked against it.
• Cloudflare exposes cf.client.bot in WAF custom rules on every plan; a rule matching the ChatGPT-User user agent while excluding cf.client.bot distinguishes verified bots from impersonators.
• The stronger verified-bot field and detection IDs require Enterprise with Bot Management, which the author had assumed was in play and was not.

A caching wrinkle

The post adds one more correction: research published this month, cited by the author, found that ChatGPT serves opened pages from a cache shared across accounts and keyed by URL. One account's copy can be served to another account in a different country with no request reaching the origin server at all. A ChatGPT-User hit in server logs is therefore a refresh for whoever asks next, not necessarily a live fetch for the person currently asking. The author discloses building Mnemoverse, a memory engine for AI agents, and says the data came from watching their own site's traffic.

Why it matters

This is a compact lesson in reading dashboards critically. An aggregate metric — failed AI requests up 63 percent — read like a site-health problem, and a redirect map was nearly built to remedy what was actually automated scanning. It also shows that AI crawler analytics inherit the weaknesses of user-agent identification: on entry-level plans any client can present a well-known bot's user agent, inflating AI-traffic counts and potentially skewing decisions about which crawlers to allow or block. The remedy is unglamorous but effective — cross-check claimed identities against published IP ranges and verified-bot signals before acting on what a dashboard says.