Two weeks of nginx logs show how AI agents actually fetch Markdown pages

GoodBarber, an app-building platform, has published two weeks of nginx log data on what actually happens when a site offers Markdown versions of its pages to AI systems. According to an engineering post on dev.to by the company's co-founder, every public page of its marketing site on eleven language hosts has had a Markdown twin since September 4, reachable either through content negotiation or by appending .md to a URL. Dedicated logs recorded every request through either route until September 18, and the author read all of them.

How the Markdown twin is served

The site runs Django behind nginx with a full-page cache. Instead of duplicating content, Markdown is produced from the HTML the cache already holds, by a middleware placed last in Django's stack so it is the first to see outgoing responses. Conversion happens once per distinct HTML, keyed by a fingerprint of that HTML, which keeps output identical on cache hits and misses.

Two routes reach it. Sending Accept: text/markdown — with q-values that prefer Markdown — returns it at the same URL, with Vary: Accept. Appending .md to a path resolves the canonical URL and hands the request to the same middleware, so the two routes cannot disagree. Every eligible page declares its twin through an alternate link element and a Link header, and the Markdown carries YAML front matter, a canonical link and a noindex robots tag. Before launch, the team fetched three pages as seven client types with the feature on and off and compared the twenty-one responses byte for byte: identical, at roughly eight microseconds of extra cost per request. The same pipeline regenerates llms.txt and llms-full.txt nightly in eleven languages, with currency forced per country so prices match the page. Cloudflare, Zapier and Vercel already do this, the author notes; the contribution is the measurement.

Two doors, two populations

The .md route saw about 80,000 requests, nearly all GET, from roughly 5,800 addresses across 26,000 distinct host-and-path pairs; 93.8% were served with a 200, and there were no server errors in two weeks. Just over half that traffic was declared AI crawlers, followed by Baidu, Bing and Apple.

The header route saw about 22,500 requests, two-thirds GET and one-third HEAD, but only 52.7% received a 200. About 10,500 were redirects, most of them (8,500) caused by missing trailing slashes — a router-level 301 the middleware never sees, driven largely by ShapBot, which requests pages HEAD first and then GET.

Among named crawlers on the .md route, Amazonbot made about 13,000 verified requests with no referer, and Meta-ExternalAgent about 10,000. GPTBot made about 5,800 verified requests, every one carrying a referer from the site itself — almost always the HTML twin the same client had fetched within the previous ten minutes, in 98.9% of cases. OpenAI's crawler, in effect, follows the alternate link. Bing made about 6,800 verified requests, 91.7% of them aimed at the site's internal search pages. The real Googlebot requested zero .md URLs, and every request claiming to be Googlebot came from outside Google's published ranges.

Which agents negotiate

The Accept: text/markdown crowd is a different population. ShapBot and ExaSearchBot, two answer engines built for agents, account for 80% of negotiated requests — about 12,000 and 4,700 page requests, sending the header 98.3% and 93.5% of the time. Claude Code is the one coding agent in the group: about 1,400 requests, 99.1% with the header. The big consumer assistants and their crawlers barely participate. ClaudeBot, ChatGPT-User, GPTBot and PerplexityBot each made between roughly 5,800 and 24,000 page requests and sent the header in 0.0% to 0.1% of them. This matches Checkly's February measurement, which found Claude Code, Cursor and OpenCode send the header while Codex, Gemini CLI, Copilot and Windsurf do not.

Counting method and its limits

Requests were matched against operators' published IP ranges — OpenAI, Google, Bing, Meta and others — with Baidu, Yandex and PetalBot verified by reverse DNS. Anthropic publishes no ranges, so ClaudeBot figures are user-agent-declared, an indication rather than proof. Receiving Markdown on a negotiated request was established by comparing response size against the .md twin of the same path. The site never inspects user agents to decide what to serve, which sidesteps the user-agent sniffing John Mueller criticised when a Next.js middleware tried detecting GPTBot that way; as Search Engine Journal reported, his objection was to sniffing, not content negotiation.

Why it matters

For publishers weighing Markdown serving, this is field data rather than vendor claims. It suggests both doors are needed: corpus crawlers arrive via .md URLs, often with no referer, while agent-facing answer engines and one coding agent use content negotiation — and the big consumer assistants use neither, at least against this site. Declaring the alternate link matters, since GPTBot demonstrably follows it. The author's earlier four-month count is equally blunt: no assistant arrived through llms.txt on its own, while assistants and their crawlers made 1.6 million requests to ordinary pages. An index file alone reaches no one; the pages themselves, and how they are fetched, are what count.