JavaScript-only navigation hid pages from GPTBot and ClaudeBot in 41-day crawl test

GPTBot and ClaudeBot failed to discover a single page reachable only through JavaScript-injected links during a controlled 41-day crawl experiment, while both crawlers reached every page in sections linked with plain HTML anchors. The test, conducted by Vinicius Stanula and reported by Search Engine Land, indicates that client-side navigation can lock major AI crawlers out of large parts of a site.

How the experiment was set up

According to a dev.to write-up of the report, the test site contained roughly 2,400 pages spread across 21 top-level sections. Navigation was isolated as the only variable: sitemaps, breadcrumbs and every other discovery route were disabled, so a bot could only find pages by following links on the page itself. To reach content in the JavaScript group, a crawler had to render the parent page and process its scripts.

The hierarchy under test was split in two. Eleven sections used ordinary HTML links and covered 748 pages; ten sections used JavaScript-injected links and covered 293 pages.

The crawl results

The gap between the two groups in the first phase was stark:

Crawler	HTML group (748 pages)	JavaScript group (293 pages)
GPTBot	748 pages	0 pages
ClaudeBot	748 pages	0 pages
Amazonbot	735 pages	0 pages
Meta External Agent	537 pages	0 pages
GoogleOther	495 pages (66%)	142 pages (48%)
Bingbot	282 pages	3 pages
Googlebot	35 pages (5%)	7 pages (2%)

GPTBot and ClaudeBot each crawled all 748 HTML-linked pages and zero JavaScript-linked pages. Amazonbot and Meta External Agent likewise found nothing through JavaScript, and OAI-SearchBot, ChatGPT-User and PerplexityBot also made no progress through the JavaScript-only hierarchy in the reported results.

Googlebot was the only crawler in the experiment that rendered JavaScript and moved deeper into that hierarchy, but its reach was limited on both sides. The GoogleOther crawler label showed substantially broader coverage — 66 percent of the HTML group and 48 percent of the JavaScript group — though the report cautions that crawling a URL is not the same as indexing it or using it in a search result.

The experiment measured discovery and crawl reach only, not rankings, traffic or final index inclusion.

Converting the links restored discovery

In a second phase, the JavaScript links were converted to standard HTML links, and several crawlers responded quickly. GPTBot reswept the converted sections in roughly 48 hours and picked up about 250 new pages. Bingbot discovered around 212, and Meta External Agent about 41.

Googlebot's recovery was far slower: by day 41 after the conversion it had rediscovered just one page in those sections. The contrast is a useful reminder that fixing discoverability does not guarantee that every system will promptly return.

How to reduce the risk

The guidance from the report is not to abandon JavaScript, but to stop making it the only way to expose important internal paths. Its recommendations include:

• Use standard anchor links with href attributes for essential internal navigation.
• Ensure critical navigation links are present in the raw HTML delivered by the server.
• Use server-side rendering or prerendering when a framework otherwise injects navigation after page load.
• Keep JavaScript for interaction and enhancement, not as the sole discovery mechanism.
• Give important pages more than one sensible discovery path, such as HTML navigation plus XML sitemaps.

The write-up flags sites that have migrated to a JavaScript framework, redesigned a menu, or adopted a headless content setup as particularly exposed, because clickable elements can silently replace crawlable anchors and create an invisible layer of category and supporting content. It also advises auditing the server response rather than relying on what appears in a rendered browser: if core navigation URLs are missing from the delivered HTML, the site has a discoverability dependency that many AI crawlers will not overcome.

Why it matters

Internal links are a discovery mechanism, not just a user interface choice. If a link is absent from a page's initial HTML, crawlers such as GPTBot and ClaudeBot may never reach the destination page regardless of its content quality — and pages those systems never crawl cannot appear in AI-generated answers. As AI assistants become a common route to information, a navigation layer that works in browsers but not in raw HTML can leave whole sections of a site effectively invisible. The remedy is cheap and well understood: keep the most important internal routes as static HTML anchors, and treat JavaScript as an enhancement on top of that baseline.