Is your website invisible to AI? How JavaScript-loaded sites keep bots in the dark

If your website loads its words and links with JavaScript after the page opens, there is a good chance many AI crawlers never see them. Google renders JavaScript before indexing, though sometimes with a delay. But the best public measurement we have, a December 2024 study by Vercel and MERJ, found that none of the major AI crawlers it tracked (OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, and PerplexityBot) ran JavaScript at all. They read the first HTML your server sends and move on. Sites that are server-rendered or pre-built as static pages put everything in that first HTML, so every bot sees the full page. Here is how it works, how to check your own site and how to fix it.

Rendering in plain English

When someone visits a page, the server sends back HTML. What is in that HTML depends on how the site is built.

Think of it like ordering furniture. Server-side rendering is a store that assembles the table for you before delivery. Static site generation is a showroom where the tables were built ahead of time and are ready to carry out. Client-side rendering delivers a flat box and instructions, and you put it together at home. A person with tools (a browser running JavaScript) ends up with the same table. A bot that only looks at what arrived on the truck sees a box.

Hydration is the step where JavaScript "wakes up" a page that already arrived fully built, adding clicks, menus and forms on top of HTML that was already there. It gives you the best of both: complete content for bots and an interactive page for people.

How Google handles JavaScript

Google's JavaScript SEO basics guide describes three phases: crawling, rendering and indexing. Googlebot fetches the HTML, then queues the page for rendering in a headless version of Chrome, then indexes what the rendered page shows. On the wait, Google's wording is: "The page may stay on this queue for a few seconds, but it can take longer than that."

So Google can usually see JavaScript content, but it is a two-step process, and anything that breaks in the second step (a blocked script, an API that times out, an error) can leave the page looking thin. The same guide still says server-side or pre-rendering is "a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript." It also says Google can only discover your links if they are <a> elements with an href attribute.

Google's AI features draw on the same system. Google's list of common crawlers says Google-Extended is only a robots.txt control token with no separate user agent; the crawling is done by Google's existing crawlers. Vercel and MERJ also observed Gemini using Googlebot's rendering setup.

How Bing handles JavaScript

Bing's most detailed public statement is an October 2018 post on the Bing Webmaster blog. It says "bingbot is generally able to render JavaScript," but "it is difficult for bingbot to process JavaScript at scale on every page of every website." We could not find a newer official Bing document that updates this, so treat it as dated. This matters for AI because Microsoft Copilot and several other assistants lean on Bing's index.

Do AI crawlers run JavaScript?

This is where people get confused, so let's separate what is measured from what is assumed.

What was measured. Vercel and MERJ published The rise of the AI crawler on December 17, 2024, based on traffic across Vercel's network. Their findings:

• None of the major AI crawlers they tracked rendered JavaScript. That list included GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider.
• Some did download JavaScript files (about 11.5% of ChatGPT's fetches and 23.8% of Claude's) but did not run them.
• Gemini (through Googlebot) and AppleBot did render JavaScript.
• ChatGPT's and Claude's crawlers each hit 404 pages on about 34% of fetches, compared with about 8% for Googlebot.

What the operators say. OpenAI's crawler documentation describes OAI-SearchBot (search), GPTBot (model training), ChatGPT-User (user-requested fetches) and OAI-AdsBot. Anthropic's crawler page describes ClaudeBot (training), Claude-User (fetches for a user's question) and Claude-SearchBot (search quality). Perplexity's crawler docs describe PerplexityBot (search) and Perplexity-User (user actions). None of these pages says whether the bots run JavaScript. Apple's Applebot page does imply rendering: it warns that if JavaScript and CSS are blocked, Applebot "may not be able to render the content properly."

What is assumed. The Vercel and MERJ data is almost two years old, and any of these companies could add rendering at any time. User-triggered fetchers may also behave differently from bulk crawlers. The safe plan is not to bet on it. If your key content is in the first HTML, it does not matter which bots render.

Common culprits

Most sites are not fully client-side. The trouble is usually a few sections that load late:

• Single-page apps. A plain React, Vue or Angular app with no server rendering often sends an empty <div id="root"> and builds everything in the browser.
• Site builder widgets. Some builders load reviews, menus, listings or product grids from a script after the page opens.
• "Load more" buttons and infinite scroll. Bots do not click or scroll, so anything past the first batch may be missing.
• Tabs and accordions filled by API calls. Hidden-but-present text is fine. Text that is fetched only when someone clicks is not there for bots.
• Lazy-loaded text. Lazy-loading images is good practice. Lazy-loading the words on the page is not.
• Cookie walls and consent overlays that block or replace content until someone clicks. A banner on top of real content is fine; a wall with no content behind it is not. See do you need a privacy policy and cookie banner.
• Client-side redirects done with JavaScript instead of a server-side 301.
• Content in iframes, such as embedded booking tools or third-party FAQs, which belongs to another page and often another domain.
• Links built with JavaScript click handlers instead of real <a href> links, which crawlers cannot follow.

The intentional side: blocking bots on purpose

Some sites are invisible to AI because the owner (or their host) chose that. You can block crawlers by user agent in robots.txt, for example User-agent: GPTBot with Disallow: /. Note that the user-triggered fetchers work differently: OpenAI says robots.txt rules "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt.

Cloudflare added another layer. On July 1, 2025 it announced a move to block AI crawlers by default unless they pay, and new domains are asked whether to allow AI crawlers when they sign up. Its AI Crawl Control tools (formerly AI Audit) let you allow or block individual crawlers, and Pay Per Crawl, which lets you charge crawlers for access, is listed as a private beta. If your site is on Cloudflare and you are missing from AI answers, check these settings before anything else.

Why block? Publishers who sell their content, or anyone who does not want it used to train models, have good reasons. Why not? A local service business or B2B firm usually wants to be found and cited. A common middle path is to allow search and user-fetch bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and decide separately about training bots (GPTBot, ClaudeBot, Google-Extended).

How to test your own site

1. View Source vs Inspect. Right-click and choose View Page Source. That is the raw HTML bots get first. Inspect shows the page after JavaScript has run. Search the source for a sentence from your main content. If it is only in Inspect, JavaScript is adding it.
2. Turn off JavaScript in Chrome. Open DevTools, press Ctrl+Shift+P (Cmd+Shift+P on a Mac), type "Disable JavaScript," and reload. What is left is roughly what a non-rendering bot sees.
3. Use curl with a bot user agent. In a terminal, run something like curl -A "GPTBot" https://yoursite.com/page and look for your headings and text. This also shows whether a firewall blocks that user agent. Some hosts check IP addresses too, so a pass here is a good sign, not proof.
4. Google Search Console URL Inspection. Inspect a URL and choose View crawled page to see the HTML Google stored, or run a live test to see a screenshot, the HTML and any JavaScript errors.
5. Rich Results Test. Google's Rich Results Test renders the page like Googlebot and lets you view the tested HTML, which is handy for pages you do not own in Search Console.
6. Check your server logs. Search raw access logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Applebot. If they show up and get 200 responses, they can reach you. If they never show up, or get 403s, something is blocking them.

How to fix it

• Render on the server or at build time. Frameworks like Next.js, Nuxt, Astro and SvelteKit can send complete HTML and then hydrate it. This is how we build, and it is why we like Next.js for business websites.
• Pre-render if you cannot rebuild. Prerendering tools or services can generate HTML snapshots of a client-side app. Just make sure bots and people get the same content.
• Put critical content and links in the first HTML. Headings, body copy, prices, service areas, contact details and navigation should not wait for a script. Client-side features are fine for extras like filters or chat widgets.
• Use real <a href> links for anything you want crawled.
• Put structured data in the HTML, not injected later by a tag manager. See llms.txt, schema and Open Graph.
• Treat llms.txt as optional. It is a proposal from Jeremy Howard, first published in September 2024, not an adopted standard. It can point tools to your content, but it cannot fix pages that are empty without JavaScript.
• Skip dynamic rendering for new builds. Serving bots a pre-rendered copy while people get the JavaScript version still works, and Bing once recommended it. But Google's dynamic rendering page now calls it "a workaround and not a recommended solution" and points to server-side rendering, static rendering or hydration instead.

Faster first HTML also helps real visitors. Our guide to improving website speed covers the performance side, and answer engine optimization covers what to write once bots can actually read it.

The short version

Google will usually wait for your JavaScript. Most AI crawlers, based on the best data available, will not. If the words and links that matter are in the HTML your server sends first, you do not have to guess which bots render and which do not.

Not sure what bots see on your site? Get in touch and we will check it for you.