HTTP headers are already steering AI agents, 217M-response Common Crawl scan finds

A 217-million-response scan for AI-directed headers

Every HTTP response arrives in three parts: a status line, a block of header fields, and the body that a browser actually renders. Headers are metadata intended for software, and a server can invent custom names at will. Since a browser shows a human reader almost none of them, headers are a natural channel for anyone who wants to address an AI agent rather than a person.

Signals Corps, which published the analysis, sampled 10% of Common Crawl's September 2026 crawl, or 217,097,223 page responses, and searched the response headers for text written for AI systems. According to the firm, 7,959 sites sent at least one header aimed at an AI reader.

What the headers say

Most of them are pointers. The firm found 7,271 of those sites directing agents to another resource such as an llms.txt file or an MCP server, with 6,989 using a header named x-llms-txt. In Signals Corps' reading, this is already routine behavior: servers telling agents where to go next.

Thirty sites went further and sent instruction-like text, 24 of them through a header called X-AI. The researchers judged 22 of the 30 to be pranks, plain instruction-override text with no attempt at concealment. The few that request real actions only matter when the agent happens to hold matching capabilities, such as shell or database access.

Servers are also distinguishing agents from people. 139 sites sent a header recording whether they had classified the client as an AI bot, and 136 of those classifications said yes. That is the prerequisite for serving agent-only content, and Signals Corps suggests auditing important sites using an agent's user agent to see what they serve.

Some headers are written against agents rather than to them. Two sites placed strings named after Anthropic's test triggers inside protest headers such as X-AI-Makes-Us-Dumber, and one site warns hacking agents away on its error pages. The firm expects agents to encounter text designed to stop them as well as to steer them.

Detection is hard and models can be talked around it

Matching on header names alone is a poor defense. The scan turned up 283 distinct header names aimed at AI, but another 1,709 sites matched AI keywords without being AI-directed at all. A name-based detector, the firm calculates, would have wrongly flagged 910 sites. Its recommendation is to watch what an agent does after fetching a page rather than what the fetch returned.

In October 2026 tests, Claude Opus, Claude Sonnet and Claude Haiku running in Claude Code did surface the test headers, and everyday tasks such as an uptime check were enough to put the headers in front of the model. They did not act on them by default. But when the user's prompt told the models to follow the site's directions, they obeyed in 7 of 9 runs. The refusal, in other words, depends on how much authority the user's prompt hands to a website.

Prior research missed this channel

Signals Corps situates the finding against existing work. Palo Alto Networks Unit 42 catalogued ten ways to deliver prompt injection on live pages, from zero-size text to URL fragments, and none of them relied on response headers; social engineering accounted for 85.2% of jailbreak methods in that dataset. Google's Threat Intelligence Group searched Common Crawl's page text and sorted the results into pranks, helpful guidance, search optimization, agent deterrence and malicious content, a taxonomy the header injections closely mirror.

The broader risk is well documented. In Anthropic's 2025 tests of Claude for Chrome, indirect attacks succeeded 23.6% of the time without mitigations and 11.2% with them, meaning even a hardened agent acted on roughly one attempt in nine. Indirect prompt injection, a term from a 2023 paper by Greshake and colleagues, works because a model reads instructions and fetched data as one stream of text; OWASP ranks it first in its Top 10 for LLM applications.

Why it matters

Whether any header text reaches a model depends entirely on what a fetch tool hands over from each response, so the defense starts there, not at the model. This scan shows the channel browsers hide is already in active use at scale. The practical follow-ons are concrete: treat response headers as agent-readable content, treat llms.txt and MCP destinations as fresh, untrusted fetches rather than instructions, and monitor post-fetch agent behavior instead of pattern-matching names. Because sites already detect and label AI clients, what an agent sees can differ from what a human sees, which makes auditing with an agent's own user agent part of any serious security review.