AI scrapers now consume about 20% of git.kernel.org's CPU capacity

Operators put numbers on the crawler problem

Operators of git.kernel.org, the infrastructure hosting Linux kernel development, say AI scrapers have become a permanent tax on the service, consuming roughly a fifth of its CPU capacity to render pages that almost no human will ever read.

According to a post on people.kernel.org by one of the site's operators, which reached the Hacker News front page, 14 to 16 of the 90 CPU cores spread across five geo-distributed nodes are constantly busy rendering git commits as HTML for crawlers. More processing power now goes to serving scrapers than to all forms of legitimate access combined, including git clones.

The scale of the scraping

git.kernel.org receives about 6 million requests per day asking to view random commits, the post reports. A proof-of-work challenge based on the Anubis project instantly rejects 66% of them, but the remaining third solves the math and reaches the site. Under generous assumptions, the operator estimates legitimate requests at only about 2% of traffic — everything else is scraping.

The request patterns are the giveaway: repeated hits on old commits in long-abandoned forks of the Linux repository are unlikely to come from developers doing actual work.

Why kernel history is a target

Linux development happens entirely in the open, from cloneable repositories to archived discussions, and the post argues this makes it especially valuable training material. Because training on model-generated text degrades language models, a corpus guaranteed to predate the LLM era — such as the full kernel commit history — is treated as premium data.

The irony, the operator notes, is that everything is already offered in the most efficient form: a plain git clone. The main linux.git repository holds about 1.48 million commits, and git.kernel.org hosts roughly 922 forks of it, which stay cheap on the backend because they share the same underlying objects. Instead of cloning once, crawlers request the enormous number of valid URLs the cgit frontend can generate — commit pages, patches, plain-text renders and diffs between arbitrary commits — re-downloading duplicate copies of the same content. The post calls this the least efficient possible way to obtain the data.

How the blocking arms race unfolded

The defenses evolved step by step, and the bots kept pace:

Bots that identified themselves through their user-agent strings were banned first. When they began faking ordinary browser identities, bans moved to the IP level. When scrapers fanned out across subnets, entire network blocks associated with cloud providers were cut off, occasionally catching legitimate automation in the process.

Then the traffic shifted to millions of residential and mobile IP addresses, each making only a handful of requests before disappearing forever. Banning those addresses is pointless, since they never return, and the post attributes the pattern to proxy SDK monetization — schemes in which consumer devices, potentially including smart TVs, are used as proxy endpoints.

The proof-of-work challenge bought real relief at first. When bots began solving difficulty level 4, it was raised to 5, which takes seconds of number crunching on a phone and warms the device. Bots now solve difficulty 5 as well, and the operator says it is no longer possible to reliably tell which challenge-solvers are automated.

Notably, the scrapers have not taken the site down. The usual outage culprit, according to the post, is poorly designed CI systems shallow-cloning repositories from many nodes simultaneously. The crawler load is a constant drain that arrives in waves rather than a single kill shot.

What changes next

To shrink the crawlable surface, the operators are removing features and gating expensive operations. Anonymous visitors should expect to lose some functionality, a trade-off the post describes as unwelcome but necessary. The author sees no simple fix while new AI ventures appear daily, all hungry for training data; the plausible endings are either a contraction of the AI sector or crawlers finally switching to the efficient route of cloning repositories.

Why it matters

This is one of the clearest quantified accounts of AI crawler traffic as an operational cost. A public resource central to Linux development is losing around 20% of its capacity to bots, and the mitigations — proof-of-work challenges and feature removal — degrade the service for humans as collateral damage.

It also documents how quickly standard defenses fall in sequence: user-agent filtering, IP bans, ASN blocks and now proof-of-work have each been defeated, with scraping traffic shifting onto residential proxy networks that are nearly impossible to filter. For anyone running a public site with a large URL space, the kernel.org experience is a preview of where things are heading: the question is no longer whether AI crawlers arrive, but how much infrastructure capacity can be affordably surrendered to them.