Cloudflare's x402 gateway and a budget-aware Go crawler point to a pay-per-fetch web

A developer has published a Go crawler that can pay websites for access using x402, the open protocol that turns HTTP's long-dormant 402 status code into a working payment mechanism. The build, documented in a dev.to post, lands alongside Cloudflare's Monetization Gateway, which supports x402 and points at a web where bots buy content per fetch instead of taking it for free.

According to the post, AI and scraping bots now account for 57% of internet traffic, up from 20% in 2021. Site owners have been stuck with a binary choice: block bots outright and risk cutting off useful automated traffic, or allow them and receive nothing in return. x402 proposes a third option — charge them.

How x402 payments work

The 402 'Payment Required' status has existed in the HTTP specification since 1991 but was never practically implemented, the author notes. x402 revives it. A client requesting a protected endpoint receives a 402 response carrying the price, a destination wallet address and the expected network. The client pays on-chain through a facilitator service, which settles the transaction and returns a transaction hash. The client then retries the original request with that hash in an X-Payment-Signature header, and the server verifies the payment through the facilitator before serving the content.

The protocol sits under the Linux Foundation with backing from Cloudflare, Coinbase, AWS, Google, Stripe and Visa, according to the post. Because Coinbase co-governs the standard, most sites settle on Base, though Solana, Ethereum and Polygon are also supported. Facilitators are optional but recommended; the author tested against Base Sepolia, Base's testnet, using a free public facilitator from the FareSide open-source project.

A crawler that watches its wallet

The x402-crawler project, published on GitHub, deliberately splits three concerns. A client package owns the payment protocol — 402 handling, signing and retry classification — a crawler package owns traversal, and a BudgetTracker owns every spending decision. Payment authorization is mandatory with no default: a client with no policy would pay whatever a server asked, and that failure would be invisible until the money was gone. Callers who genuinely want no spending ceiling must opt in explicitly.

The implementation handles the failure modes that make paid crawling hard. Per-payment and cumulative budget caps are claimed under a single lock, so concurrent workers cannot collectively overspend. Every failure reason in the x402 spec is classified as terminal, retryable or needing reconciliation, and a settlement_pending result is checked against the chain rather than blindly retried, since blind retries are how one fetch gets paid for twice. A dry-run mode prices an entire crawl without funding it, still respecting the budget caps. PDFs are streamed to disk under a size cap with filenames hashed from the URL, so a server cannot choose what lands on the local filesystem. Concurrent workers racing to fetch the same URL collapse onto a single request and share the result, and a redacting log core strips private keys from output regardless of the field name they were logged under.

Cloudflare's two-sided bet

The post's broader argument is that Cloudflare is running two crawler-payment systems at once, not one. Pay Per Crawl, the older system, uses no blockchain: Cloudflare acts as merchant of record, both the crawler operator and the publisher need Cloudflare billing accounts, crawlers are verified via Web Bot Auth, and the target market is enterprise deals with companies such as Perplexity and OpenAI. The Monetization Gateway's x402 support is the second system: no Cloudflare account required, the wallet itself serves as identity, and any agent can pay any site autonomously with no prior relationship.

The author reads these as complementary halves of one strategy — Pay Per Crawl locks in enterprise billing relationships while x402 captures open-protocol traffic — and draws a comparison to Visa: Cloudflare sits in the middle of every transaction without holding anyone's money, monetizing position rather than custody.

Why it matters

If x402 gains traction, the standoff between publishers and AI crawlers could shift from blocking versus free-riding to metered access. Publishers get a possible answer to uncompensated scraping; agent developers get a standardised way to buy content programmatically instead of negotiating bespoke deals. The crawler also previews what agentic commerce will demand in practice: spending ceilings, retry semantics that avoid double payment, and identity tied to cryptographic keys rather than platform accounts. What remains unproven is adoption — whether enough sites deploy 402 endpoints, whether on-chain settlement is fast and cheap enough for per-fetch traffic, and whether it is healthy for one edge network to occupy the payment path across most of the web.