Server logs suggest OpenAI's crawlers now execute JavaScript and render pages

What changed

A developer running several Japanese sites reports that OpenAI's crawlers abruptly switched from fetching plain HTML to executing JavaScript and rendering full pages on September 25, 2026, Japan time. Writing on dev.to, the operator of atsoho.com, a Next.js freelance marketplace, says the shift was visible in server logs: OAI-SearchBot, which crawls for search, ramped up at 4 a.m. JST, and GPTBot, used for training data, followed at noon the same day.

The observation cuts against accepted wisdom. A Vercel and MERJ study from December 2024 found that none of OpenAI's three bots executed JavaScript. The author says they found no announcement from OpenAI, and verified every request against OpenAI's published IP ranges to filter out spoofed user agents.

The numbers

On atsoho.com, OAI-SearchBot requests went from 1,253 on September 22 to 41,444 on September 25. GPTBot climbed from 83 requests on September 22 to 152,009 by September 28, arriving from 36 to 131 distinct IPs per day where it previously used one.

Raw request counts overstate the activity, though. According to the dev.to post, most of the new traffic was Next.js link prefetches — requests carrying a ?_rsc= query string that appears only after a page renders and its JavaScript runs. On September 28, 84 percent of OAI-SearchBot requests and 97 percent of GPTBot requests were prefetches.

Counting distinct referring pages gives a better measure of rendering. By that measure, pages rendered by OAI-SearchBot grew roughly tenfold, from 56 on September 18 to 861 on September 25, while GPTBot's rendered pages went from 17 to more than 4,000, an increase the author calls well over a hundredfold.

Rendering from stored HTML

A curious detail: of 4,309 pages GPTBot rendered on September 28, only 28 percent had been fetched from the origin server that day, and the site's HTML is not cached at its CDN. The author's interpretation is that OpenAI is re-rendering HTML it stored earlier in its own headless browser, with only the prefetches triggered during rendering reaching the origin. That would fit OpenAI's documented policy of sharing crawl results between its bots to avoid duplicate crawling.

Not everywhere, apparently

All four Next.js sites on the same server saw OAI-SearchBot activity rise 5 to 15 times on September 24 and 25, despite running different Next.js versions and deployment setups, and the author changed nothing that week. Yet the author notes that Cloudflare Radar showed OpenAI's share of AI bot traffic flat through September, suggesting rendering is enabled selectively rather than network-wide.

Static sites are touched differently. On one non-Next.js site, OAI-SearchBot began pulling images and CSS on September 25, which is what rendering looks like, but since static pages fire no prefetches, request counts barely moved.

A possible traffic effect

Around the same time, visits from ChatGPT to atsoho.com's job pages increased and daily signups rose about 1.7 times. The number of people arriving from ChatGPT did not change; their landing pages did, shifting from blog posts to job pages. The author cautions that this is correlation, not proven causation.

Practical takeaways

The post's advice for sites that want to surface in ChatGPT: allow OAI-SearchBot in robots.txt and check that WAF or bot-protection rules are not blocking OpenAI's IPs, since the author's own server was returning 403s in the days before the change. Keep OpenAI's published IP lists current, because gptbot. and chatgpt-user. both changed that week. Put key content such as body text, titles, prices and dates in the initial HTML via server rendering, and build internal links to pages you want read, since rendered pages expose linked pages through prefetches.

Notably, the author argues against blocking Next.js prefetch requests: Next.js's built-in bot detection list does not include OpenAI's bots, and a prefetch response contains the linked page's content, making it one of the ways a rendering crawler reads a site. Marking freshness in structured data helps ChatGPT avoid sending users to stale pages such as closed job listings. The author adds that llms.txt, which OpenAI's bots read only once every few days, does not appear to determine whether ChatGPT mentions a site.

Why it matters

If confirmed, rendering changes the mechanics of AI visibility. Sites whose content loads only through client-side JavaScript have effectively been invisible to OpenAI's crawlers until now, and a year of guidance built on that assumption needs revisiting. Server-rendered, well-linked pages with machine-readable freshness signals become more important, while prefetch amplification means a few thousand rendered pages can translate into over a hundred thousand requests a day for the hosting site. OpenAI has not announced anything, so operators watching their own logs are, for now, the primary evidence.