Why ChatGPT Search and Perplexity skip websites: a four-point technical audit

AI search plays by different rules than Google

According to a technical audit published on dev.to, the spread of conversational search — ChatGPT Search, Google's AI Overviews and Perplexity — has created a class of queries where the engine returns a synthesized answer plus a small number of citations instead of ten links. The post cites unnamed search ecosystem data claiming that more than 70 percent of websites cited by ChatGPT Search do not appear in Google's top ten results for the same query, and argues that sites tuned only for Googlebot may be effectively invisible to answer engines. The author identifies four technical causes, each with a fix.

An empty HTML shell gives crawlers nothing to read

The first and most common problem, per the post, is client-side rendering. Googlebot can afford a rendering pipeline with headless Chromium and will eventually execute JavaScript. AI crawlers such as GPTBot, PerplexityBot and ClaudeBot operate under tight throughput and latency budgets: they download the initial HTML and move on. If the document is nothing but an empty mount point and a bundle script, the crawler extracts no text at all — no content, no citation, no visibility.

The recommended fix is to make high-value material — definitions, feature breakdowns, pricing tables — present in the server-delivered HTML, using server-side rendering, static generation or React Server Components.

Blocking training bots can block citations too

The second pitfall is a robots.txt file written in haste. When AI scraping first became controversial, many teams added blanket Disallow rules for GPTBot and CCBot. According to the post, OpenAI separates its crawlers by purpose: GPTBot gathers training data, while OAI-SearchBot powers live retrieval and citation inside ChatGPT Search. A blanket block can therefore keep your pages out of ChatGPT's answers even when a user asks about your product by name.

The post's suggested configuration explicitly allows OAI-SearchBot, PerplexityBot and ClaudeBot, alongside a general allow rule and a sitemap reference. Whether to adopt it is a policy decision: it reopens access to AI companies that some publishers have deliberately excluded, in exchange for eligibility as a citation source.

Answer-shaped content and structured data

The third finding draws on generative engine optimization research, which the post says includes a Princeton paper: language models favor pages that front-load conclusions. The practical advice is to place the direct answer in the first two sentences under each heading, then add depth. Concrete numbers matter too — the post claims verifiable statistics and metrics can raise extraction rates by 30 to 40 percent. Structured data helps as well: JSON-LD schemas such as Article, FAQPage and SoftwareApplication give retrieval systems unambiguous semantic signals about what a page actually is.

llms.txt as a new root-of-domain convention

The final item is llms.txt, a markdown manifest served at the domain root. Analogous to what robots.txt became for crawlers in the 1990s, it summarizes what a site or project does in plain text and links to its key documentation, letting an agent consume the essentials in one pass rather than wading through full page payloads. According to the post, the convention has spread across AI developer tooling, including Anthropic and Cursor.

A short verification pass

The audit closes with a checklist that takes minutes to run: fetch the raw HTML with curl and grep for a core value proposition, confirming the content exists without JavaScript; verify that OAI-SearchBot and PerplexityBot receive 200 responses and are not disallowed; validate JSON-LD in Google's Rich Results tool; and publish an llms.txt file. The author also promotes a free scanning tool, SiteReadyForSi, that simulates how each bot perceives a page — though the same checks can be done by hand.

Why it matters

For anyone publishing on the web, discoverability is splitting into two channels with different technical requirements. A site can rank well in Google and still be unreadable to the systems answering conversational queries, because the failure modes — client-side-only content, blanket crawler blocks, unstructured prose — are invisible in a normal browser. The fixes are cheap relative to the risk: render key content server-side, distinguish search crawlers from training crawlers in robots.txt, shape pages so answers are extractable, and publish a machine-readable manifest. The specific figures in the post are the author's claims rather than independently verified measurements, but the audit steps themselves cost little to run and either surface a real problem or rule one out.