llms.txt explained: the Markdown file sites publish for AI models, and whether it works

Over two years after Jeremy Howard first proposed it, llms.txt — a plain-text file that hands AI models a curated map of a website — has grown from a side proposal into a standard published by hundreds of sites. According to a guide published on dev.to, Howard, who founded fast.ai and is a former president of Kaggle, floated the idea in September 2024. Cloudflare, Stripe, Anthropic, Vercel, Supabase and Zapier have since published files, and community directories now count more than 780 adopting sites.

What the file actually does

When a chatbot such as ChatGPT or Perplexity answers a question about a product, its crawler fetches raw HTML: headers, footers, navigation, cookie banners and popups, with the substantive content buried somewhere inside. An XML sitemap does not fix this, because it lists every URL — homepage, privacy policy, long-forgotten test pages — with equal weight and no indication of where to start. llms.txt, served at the site root, is a Markdown document that states what a site is about and links to the pages that matter most.

A deliberately minimal spec

The format mandates exactly one element: an H1 heading with the site or project name. Everything else is optional — a blockquote summarising the site, H2 sections that group related links, and Markdown links with short descriptions. Per the dev.to guide, Howard kept it minimal on the grounds that sites already have sitemaps, robots.txt and meta tags, but nothing that serves as an overview written for language models.

Two variants: the summary and the full text

The standard defines two files. llms.txt is the condensed version — a summary plus links to key pages, typically at most a few hundred lines. llms-full.txt concatenates the complete Markdown content of important pages into a single file, so a model with a large enough context window can absorb an entire documentation set in one pass without crawling or parsing HTML. The sizes reported in the guide underline the difference: Cloudflare's full file runs to 3.7 million tokens, Anthropic's to roughly 481,000, and Vercel's has been compared to a 400,000-word novel.

Who is using it

Adoption skews toward developer-facing companies. The guide notes that Stripe organises its file by product category and explains how agents can retrieve plain-text documentation; Cloudflare structures entries by product line, with descriptions that help a model decide which page to fetch. Anthropic maintains both variants — an 8,364-token summary and a 481,349-token full version covering its API documentation — while Vercel has extended the concept with a proposal for inline LLM instructions inside HTML pages. Supabase, Zapier, Modal and Coinbase are also listed among adopters.

The evidence is mixed

Search Engine Land tracked ten websites for 180 days — 90 before adding llms.txt and 90 after. Two sites saw AI traffic rise (12.5% and 25%), eight saw no measurable change, and one declined 19.7%. The guide points out that the two gainers had also launched new content and PR campaigns during the period, so the file alone was not shown to cause the improvement.

Reach is also narrow: Rankability scanned the 1,000 most-visited websites globally and found only 0.3% carried an llms.txt file as of mid-2025. No AI provider — OpenAI, Anthropic or Google — has confirmed its crawlers consistently follow the file, and Google's John Mueller has compared the standard to the long-deprecated keywords meta tag.

The argument for adopting anyway is mostly about cost. The guide contends setup takes minutes with no downside, and cites forecasts of AI search growth, including a Gartner prediction that a quarter of search volume will shift away from traditional engines by late 2026.

Not a replacement for robots.txt or sitemaps

The three files do different jobs. robots.txt controls which crawlers may access a site; an XML sitemap lists every URL for indexing; llms.txt explains in prose what the site is and which pages matter. The guide likens them to a security guard, a phone book and a tour guide respectively — and suggests most sites need all three.

Setting it up on WordPress

The manual route means creating the Markdown file in the WordPress root directory, alongside wp-config.php, uploading it via FTP or a hosting file manager, and adding rewrite rules in .htaccess or functions.php so WordPress serves it correctly. The drawback is maintenance: every new post or restructured page requires a hand edit, which becomes tedious on content-heavy sites. The guide recommends generating the file automatically with a plugin instead, and its plugin walkthrough continues beyond the published excerpt.

Why it matters

llms.txt is a low-cost bet on how AI systems will consume the web. The measured evidence does not show it driving traffic by itself, and none of the major model providers has committed to reading it — yet Stripe, Cloudflare, Anthropic and hundreds of others publish anyway. If AI-mediated search keeps growing, a curated machine-readable overview starts to look like basic publishing infrastructure, much as sitemaps did for early search engines. Publishing now shapes a convention before its readers formally exist, at a cost of a few minutes of work.