Audit claims WAF blocks and RAG chunking keep most corporate sites invisible to ChatGPT and Perplexity

An engineering audit published on dev.to by KusiAI, a research lab based in Barcelona, reports that more than 90% of the corporate websites it examined are effectively invisible to generative answer engines such as ChatGPT Search and Perplexity. After reviewing more than fifty corporate platforms, the team concluded that the failure is infrastructural rather than editorial: three stacked layers of plumbing prevent retrieval-augmented generation (RAG) pipelines from ingesting a site's content at all. The 90% figure comes from a single vendor's sample, so it reads as a directional finding, but the failure modes it describes can each be checked by any web operator.

Layer one: the firewall that answers AI crawlers with a 403

Answer engines no longer depend only on static indexes, the audit argues. At query time, systems such as ChatGPT Search and Perplexity perform live fetches of candidate URLs using identifiable user agents: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, and ByteDance's Bytespider. According to the post, most corporate sites sit behind web application firewalls such as Cloudflare, AWS WAF or Sucuri, with generic anti-scraper rules or features like Bot Fight Mode enabled by default. Those rules greet the crawler with an HTTP 403 or an interactive CAPTCHA challenge. Because the bot cannot solve the JavaScript challenge, it abandons the URL in under 200 milliseconds, and the model answering the user moves on to a competitor whose server responded normally. The authors publish a short Python script as a first diagnostic: it requests a target URL with each crawler's user-agent string and reports whether the perimeter returns a 200 or a block.

Layer two: chunking decides what survives ingestion

Content that clears the firewall then enters a vector ingestion pipeline, and this is where the audit says traditional SEO copy fails. Dense embedding models split text into fixed chunks of roughly 256 to 512 tokens, so a 2,000-word monolithic article with padded introductions dilutes the vector density of every chunk it produces. The authors also point to the documented "lost in the middle" attention effect, in which transformers weight the start and end of a chunk most heavily; facts stranded mid-chunk tend to be discarded during cross-encoder reranking. Their remedy, labelled rule E1b, is structural: every semantic heading should be followed immediately by a closed declarative block of 40 to 68 words containing a claim, a verifiable figure and a direct technical consequence, with no orphan paragraphs under 20 words. Kept below roughly 70 words, each retrieved fragment carries a complete answer regardless of where tokenization cuts it.

Layer three: an agent-native surface for models

Beyond writing style, the audit prescribes what it calls a level-5 agent-native architecture: a statically generated human-facing site, with the post naming Astro as an example, targeting time-to-first-byte under 50 milliseconds, plus machine-readable manifests at the domain root. These include an /llms.txt index describing the business and its endpoints, /agents.md with attribution instructions for models, and /.well-known/mcp. exposing tools via the Model Context Protocol. The authors also recommend replacing passive sitemaps with IndexNow notifications pushed within five seconds of each build, and deep JSON-LD markup that ties company entities to stable identifiers such as Wikidata and ORCID. KusiAI says the methodology is pre-registered in an open research repository (Zenodo) so the results can be reproduced.

Why it matters

If even part of this finding holds, visibility in AI-mediated search is being decided by firewall configuration and content structure, not keywords. Operators who want to be cited by ChatGPT or Perplexity can act immediately: probe the perimeter with AI crawler user agents, allowlist the bots they want reading the site, and restructure key pages so that any 512-token window contains a complete, self-contained statement. Two caveats deserve weight. First, the numbers come from one lab's sample of about fifty sites and from a vendor that sells this class of audit, so independent verification is warranted before treating 90% as an industry-wide rate. Second, the audit assumes visibility is the goal; operators who deliberately block AI crawlers to protect content, licensing or infrastructure are making the opposite trade-off, and a 403 to GPTBot can be policy rather than defect.