Server-side rules move 74.5% of a blog's browser User-Agent traffic out of the human bucket

What the measurement found

A blog post on gkoreli.com, which surfaced on the front page of Hacker News, describes an attempt to separate readers from bots using only signals visible at the edge, with no JavaScript-based detection. The author serves the site through a Cloudflare Worker and classifies every eligible successful page GET using network data and request headers.

The headline figure: across the measured window, 277 of 372 requests carrying a browser User-Agent were moved out of the Browsers category by those rules — 74.5%. Network origin, not header anomalies, was the biggest single cause. In a 72-hour log sample ending September 3, 430 of 844 successful page GETs looked like genuine page navigations but arrived from hosting networks. The largest cluster, 374 requests, was attributed to a single Google Cloud client claiming to be Chrome Mobile 114, and it passed every header check the browser rule applies.

Header checks alone did catch some traffic: 60 requests that the network rules classified as cloud-carried nonetheless carried the navigation headers — Sec-Fetch-Mode: navigate and Sec-Fetch-Dest: document — that the browser rule requires.

Why the author started counting

The investigation began with a mismatch between two counters. Over seven UTC days ending September 2, a browser-UA count recorded 1,209 page events and 578 daily client identifiers, while the rolling Cloudflare Web Analytics dashboard showed 113 page loads and 52 visits. The author notes the comparison was not clean: a daily client identifier is not a visit, the time windows differed, and the script-based dashboard included the /stats route.

Individual requests looked stranger than the totals. One client classified as mobile fetched 31 distinct pages within the same timestamp second, and 156 of 164 browser-UA page observations arrived with no referrer. The post stresses that such facts alone do not prove automation, which is what prompted building a classifier rather than trusting either number.

How the classifier works

Four inputs feed the rules: the autonomous system number Cloudflare attaches to each connection, Fetch Metadata headers, the Accept and Accept-Language headers, and the User-Agent string. A client cannot rewrite its ASN, though it can reach the site through another network or proxy; a User-Agent can be imitated at will. Results, selected headers and classification reasons are stored in Cloudflare's D1 database, with prefetches, the /stats route, API routes and non-page responses excluded from the counter.

Two ordering decisions do most of the work. First, requests from a curated list of hosting networks are labelled as cloud browsers before any browser-shape test runs, even when the navigation headers are present. Second, a request that claims a modern engine — Chromium 76+, Firefox 90+ or Safari/iOS 16.4+ — but lacks Fetch Metadata is treated as an HTTP client, while older or unreadable claims fall into a legacy-browser category.

The audit also caught bugs in the author's own tooling: the Accept check mishandled some valid headers, and newer rows were missing a network-provenance marker. Both have since been repaired.

Where the evidence stops

The post is explicit that shrinking the Browser count does not establish real readership. Over two complete UTC days, the remaining 95 browser-classified HTML observations still disagreed with 14 Cloudflare Web Analytics page loads. According to the author, Cloudflare documents that script blockers and browser or network loss can make its beacon miss page loads, and a script can equally run inside an automated browser — so neither counter is ground truth. Nine stored signature verifications identify specific signers, including crawlers and deliberate tests, but say nothing about people asking an assistant to read a page.

The curated hosting list also leaves out several shared-service and consumer-VPN networks, accepting the opposite cost: some automation will arrive through networks the list does not cover. And a cloud browser may be doing useful work for a person, so that traffic stays recorded and visible rather than discarded. An earlier under-two validation threshold was retracted as unsupported and removed from the article.

Why it matters

If roughly three quarters of browser-User-Agent requests to a small personal blog carry bot signals, then site analytics that equate browser UAs with readers are likely overstating audiences — and that distortion flows straight into advertising economics and every decision made from a dashboard. The method matters as much as the number: ASN data and Fetch Metadata are available server-side with no JavaScript, no fingerprinting and no privacy cost, making this kind of measurement cheap for any site behind a proxy to replicate. The honest caveats — one small site, short windows, no claim of actual readership — make this a starting point rather than a census, but a well-documented one at a moment when automated browsing and AI assistants are growing share of web traffic.