61% of live AI-built sites return near-empty HTML to crawlers like GPTBot

A test published on dev.to found that most websites built with AI app builders are effectively invisible to AI crawlers. On 20 September 2026 the researchers fetched 120 such sites using a single plain HTTP request, with no browser and no JavaScript execution, mimicking how GPTBot, ClaudeBot and PerplexityBot retrieve a page. Of the 104 sites that responded, 61% delivered HTML containing almost no readable content.

How the test worked

According to the dev.to post, written by Georgi Kalchev, the team assembled 120 websites created with AI app-building tools and requested each one the way an AI crawler would: one plain HTTP request, no headless browser, no JavaScript. That distinction matters because it is not how a human visitor experiences the web. People arrive with browsers that download HTML, then download and run scripts, then paint the finished page. Most AI crawlers stop at the first step.

Of the 120 sites, 104 answered the request. For 63 of those, the raw HTML contained fewer than 300 characters of visible text, which the researchers treated as effectively empty. Three hundred characters is roughly the length of a page title and a tagline, nowhere near enough to describe a product or answer a question about the site's contents.

The remaining 16 sites did not respond at all, so the 61% figure describes only the live portion of the sample.

Why crawlers see a blank page

The likely explanation is client-side rendering. Many AI app builders generate single-page applications in which the initial HTML is a thin shell, often little more than a script tag and an empty root element, with the real content assembled in the browser after JavaScript runs. A human visitor sees a finished site; a crawler that never executes JavaScript sees the empty shell.

The dev.to summary points to a longer write-up on 402registry, which the author says contains the full dataset, a control group and suggested fixes. The dev.to post itself does not name which AI builders were tested or how the control group performed, so those details are worth checking in the original before drawing conclusions about any specific tool.

What the study does and does not establish

This is a single snapshot taken on one day against a modest sample, and the 300-character threshold is a proxy rather than a formal definition of emptiness. It does not prove that every AI app builder produces crawler-hostile output, and the non-responding sites could be offline experiments, expired trials or hosting failures rather than rendering problems.

Even with those caveats, the core finding is concrete and easy to verify independently: fetch one of these sites with a plain HTTP client or simply view the page source, and see whether any meaningful text ships in the initial response.

Why it matters

If an AI crawler receives a blank page, the site cannot be understood, remembered or cited by the systems that increasingly mediate how people discover new products. A site that is invisible to GPTBot, ClaudeBot and PerplexityBot is absent from AI-assisted search and answers, a growing channel for the exact audience, early adopters and builders, that these AI-built sites tend to target.

There is also a pointed irony here: tools that use AI to generate websites are producing pages that AI systems cannot read. The builders optimise for what the site looks like in a browser and skip what it says to a plain HTTP request.

The standard remedies are well understood in web development: server-side rendering, pre-rendering to static HTML, or ensuring that critical content is present in the initial response rather than injected by scripts. For anyone shipping a site built with an AI app builder, the lesson from the 402registry test is to check the raw HTML before assuming the site is discoverable at all.