· via dev.to (home feed)
Google publishes CDN and WAF guidance for allowing Google-Agent AI traffic
Google's crawler documentation now includes a dedicated page on letting Google-Agent, its user-triggered AI fetcher, pass through CDNs, WAFs and bot-detection services via User-Agent, IP-list and optional signature checks.

Google's crawler documentation now includes a dedicated page explaining how to let "Google-Agent" traffic through content delivery networks, web application firewalls and bot-detection services, according to a write-up on dev.to. The page addresses a practical problem the article describes: after years of tightening "block AI agents" policies, hosts and agencies frequently stop user-triggered agents running on Google's infrastructure before they ever reach search, product detail or checkout pages.
What Google-Agent actually is
Google-Agent is the name Google gives to agents hosted on its infrastructure that navigate the web and carry out actions when a user asks. It is a different job from Googlebot, which crawls for Search, and a different product from the Vertex or Workspace agent tools that dominate results for the bare query "google-agent," the dev.to article notes. It is also distinct from AI training crawlers such as GPTBot, which are usually handled through robots.txt and crawl budgets. Google-Agent allowlisting sits one layer earlier: if a WAF challenges or drops the request, questions of crawlability and page performance never come into play.
The article reports that Google's user-triggered fetchers table lists User-Agent strings containing "Google-Agent" with a documentation URL, in mobile and desktop variants, and that the traffic originates from IP ranges published in a user-triggered-agents.json file.
Three signals, one of them optional
Google documents three identification signals, and the split matters in production. The first two form the baseline today; the third is still rolling out.
- User-Agent containing "Google-Agent" — a cheap first filter, but spoofable on its own since anyone can send that string.
- Client IP listed in user-triggered-agents.json — this is what makes an allow rule safe.
- Web Bot Auth, optional for now — signed requests identify as agent.bot.goog, with a public key set published at a well-known endpoint and signatures verified per RFC 9421.
Google's Web Bot Auth guidance is explicit that not every agent request is signed yet, so verification falls back to User-Agent plus IP. The dev.to article warns that a rule accepting only signed requests will fail a large share of legitimate traffic for as long as the experiment remains partial.
The recommended shape is therefore: match the User-Agent, confirm the IP against the published list, verify signature headers when present, and for requests that pass, skip interactive CAPTCHAs and JavaScript challenges as well as hard HTTP blocks.
Scope the whole journey, not the homepage
Google's troubleshooting list, as summarised by dev.to, calls out on-site search, filtering, detail pages, add-to-cart, booking flows and the underlying XHR, GraphQL and api.* endpoints as the surfaces an agent touches after the first document. An allow rule that clears only the root path fails the moment an assistant filters a catalogue or posts to a cart API.
Provider verified-bot or managed allowlist features from Cloudflare, AWS WAF, Fastly and bot-detection vendors are described as a less fragile starting point than hand-rolled rules, but the article cautions that a checkbox is not the same as an end-to-end journey that clears search and cart APIs.
Where implementations break
Failures follow a predictable order. Allow rules must evaluate before broad cloud or data-centre ASN blocks and category-level "block AI agents" packs; on agency accounts, the article observes, rule order is usually the bug rather than rule text. Challenges still break journeys when CAPTCHA or JavaScript widgets fire on search or form submission, even after the HTML loads. Load balancers and reverse proxies can substitute client IPs and strip signature headers, breaking verification in stacked edge setups. Multiple users' agents share Google's egress pool, so per-IP rate limits can throttle legitimate concurrency. And IP ranges drift, so a list pasted once rather than synced from the JSON feed goes stale.
The article also notes a diagnostic split: requests that never appear at the origin were blocked upstream, while requests that reach the origin still requiring a challenge cookie need the skip-challenge step completed.
Why it matters
The guidance turns "block AI agents" from a set-and-forget security posture into a decision with direct conversion consequences. If a user-triggered agent cannot complete a search, booking or checkout journey on a client's site, that traffic simply does not convert, no matter how well the underlying pages perform. For agencies managing many clients across different edge providers, the change argues for repeatable per-client checklists rather than ad-hoc exceptions. It also signals that Web Bot Auth, built on RFC 9421 message signatures, is moving from experiment toward standard practice — operators who automate CIDR feed refreshes and preserve verification headers through every proxy hop now will be better placed for the next verified-bot rollout.
- #cdn
- #waf
- #google-agent
- #web-bot-auth
- #bot-management