Yandex bot traffic fell 99% overnight, a reminder to audit the AI crawlers hitting your server

Yandex bot traffic vanished overnight

According to a post on dev.to, two of Yandex's five data centres were hit by Ukrainian drone strikes last weekend, in reports the author attributes to Reuters and the BBC. The effect on crawler traffic was immediate: Cloudflare's traffic data showed bot activity from the Yandex network falling to 0.4 percent of its recent maximum within a day. A fleet that had been hammering sites for weeks effectively disappeared, and before-and-after charts of the collapse reached the top of Hacker News with more than 600 points.

The author argues the strike itself is not the point. What the charts demonstrated is that a single bot network can account for a large share of the requests hitting your server — and that most operators never look.

AI crawler traffic is climbing fast

For context, the post cites HUMAN Security's 2026 benchmark report, which analysed more than a quadrillion interactions. It found AI-driven traffic nearly tripled in 2025, while AI agent traffic grew 7,851 percent. That growth does not show up neatly in CDN dashboards; it lands on origin servers, where someone pays for the CPU.

Measure before you block

The first half of the guide is a ten-minute audit using nothing but access logs. Because nginx and Caddy's combined log format stores the User-Agent in a fixed field, a short awk pipeline — extract, sort, count — ranks your heaviest clients by request count. On a quiet personal site, the author warns, the results are often uncomfortable: alongside your own monitoring and a couple of search engines sits a wall of Bytespider, meta-externalagent, Amazonbot, PerplexityBot and anthropic-ai entries.

The line between AI crawlers and search engines matters, since blocking Googlebot has real costs. The post points to the community-maintained ai.robots.txt project, which tracks AI crawler user-agent strings; grep the log for those names and divide by the total request count. Under 5 percent, a robots.txt update is probably enough. Near 30 percent, keep reading.

Just as important is what crawlers request. AI crawlers favour expensive endpoints: diff and blame views on code forges, every revision of every page on documentation sites. Xe Iaso, Anubis's developer, wrote the tool after an Amazon crawler worked around robots.txt rules and pulled apart a self-hosted Gitea instance that way — a crawler following every link can amount to an accidental denial of service.

An escalation ladder

The guide orders the options from cheapest to heaviest:

1. robots.txt is free and honoured by most major AI crawlers — Google-Extended, GPTBot, ClaudeBot and PerplexityBot all respect its opt-out tokens — but it is a request, not a control, and crawlers that spoof a browser user agent ignore it.
2. Rate limiting, via nginx's limit_req or Caddy's rate limit module, caps damage per IP. It works against single-origin floods and poorly against distributed crawlers rotating through IP ranges, which is the pattern large AI crawlers actually use.
3. A web application firewall or Cloudflare. Bot fight mode covers much of the problem if you are already behind Cloudflare. Anubis exists for cases where you cannot or will not route everything through someone else's edge: self-hosted infrastructure, commodity hosting, operators avoiding dashboards and contracts.
4. Anubis itself, described in the project's own terms as the nuclear option, and one most sites do not need.

How Anubis works

Anubis is a single Go binary running as a reverse proxy, driven by a YAML policy file that classifies each request as ALLOW, DENY, CHALLENGE or WEIGH. It now fronts GNOME's GitLab, kernel.org, the Arch wiki, Codeberg and FFmpeg's infrastructure.

Requests whose user agent contains the string "Mozilla" receive an interstitial page requiring the browser to compute SHA-256 hashes until one begins with enough leading zeros. At the default difficulty of 4, that is roughly 65,000 expected hashes — a second or two on a desktop, up to fifteen on a phone, and impossible for an HTTP client with no JavaScript engine. The default policy also denies the ai.robots.txt denylist outright and allowlists Google, Bing, DuckDuckGo, Kagi and the Internet Archive, so honestly identified bots are blocked before any hashing begins.

One detail the post singles out: denials and challenges are answered with HTTP 200 rather than 403. Aggressive scrapers treat anything else as a transient error and retry harder; hand them a 200 with a mascot page instead of your content, and their state machine records the page as fetched.

The trade-offs are stated plainly. Feed readers, Slack unfurls and uptime monitors send no JavaScript and need explicit allow rules, or they will report the site as down. There is no horizontal scaling by default: Anubis generates a fresh Ed25519 keypair on every start, so separate instances reject each other's cookies and every restart re-challenges readers. The author, who has not run Anubis in production, also lists a limitation involving determined attackers.

Why it matters

The Yandex incident is a natural experiment in how much web traffic is automated: remove one operator's infrastructure and measurable volume evaporates within a day. The practical lesson is to measure before acting. AI crawler traffic is growing fast, it targets expensive endpoints, and robots.txt compliance is voluntary. Between doing nothing and handing your edge to a third party, tools like Anubis offer a self-hosted middle path, with documented costs worth understanding first.