What GPTBot, ClaudeBot and PerplexityBot actually read on your site
AI crawlers do not run JavaScript. Here is what they extract from a page, why client-rendered sites disappear, and how to check your own pages.
AI assistants answer questions about your category every day. Whether your brand appears in those answers depends on something unglamorous: what the crawler could read from your raw HTML.
The crawlers that matter
- GPTBot — OpenAI's content crawler, feeds training and browsing.
- OAI-SearchBot — powers ChatGPT search results and citations.
- ClaudeBot — Anthropic's crawler.
- PerplexityBot — indexes for Perplexity's answer engine.
- Google-Extended — controls whether your content informs Gemini and AI Overviews.
- Bingbot — still the substrate for Copilot answers.
They fetch HTML. That is it.
None of these crawlers run a JavaScript engine on your page the way a browser does. They issue an HTTP request, take the HTML that comes back, and parse it. If your content is injected client-side after hydration, the crawler sees an empty shell — a title, a nav, and a div with a spinner.
curl -A "GPTBot" https://yoursite.com/pricing | wc -w
Run that against your most important page. If the word count is under a couple of hundred, an AI assistant cannot describe what you sell.
What they extract, in priority order
- The <title> and meta description — the page's one-line identity.
- The H1 and heading hierarchy — the structure of the argument.
- Body text in the server-rendered HTML — the substance.
- JSON-LD structured data — unambiguous facts about entities, prices, FAQs and authorship.
- Link anchor text — how pages relate to one another.
The five failures we see most
- Client-side-only content: the page is real to humans and empty to crawlers.
- No structured data: the assistant has to guess what your product costs and does.
- Headings used for styling rather than meaning, so the outline is nonsense.
- Key claims locked inside images, carousels or accordions rendered on interaction.
- robots.txt blocking AI crawlers by accident, usually copied from a template.
Answer coverage beats keyword density
Assistants assemble answers to questions. A page that states a question and answers it in the following paragraph is directly quotable. A page that circles the topic is not. Write the question as a heading, answer it in the first two sentences underneath, then elaborate.
You can check any page against all of this in about ten seconds with our free AI visibility check — it fetches your URL without JavaScript, exactly like these crawlers do, and scores what is readable.
