Every few months a new acronym promises a new discipline: GEO, AEO, LLMO. Underneath, the mechanics are familiar. An AI answer engine either searches an index it built by crawling the web, or fetches pages live when a user asks a question. Either way, it needs to reach your page, read it as text and be allowed to quote it.
That makes AI search readiness mostly a technical SEO problem, with a few new parts: more crawlers to manage, crawlers that don’t run JavaScript, and controls that decide whether your content can be shown or used for training at all.
This checklist covers all of it, in the order problems usually block visibility. Each section explains what to check, why, and how to find it across a whole site.
What is an AI search readiness audit?
An AI search readiness audit checks whether AI search systems (Google AI Overviews and AI Mode, ChatGPT search, Claude, Perplexity) can crawl, index, read and cite your pages. It covers the classic technical SEO foundations plus AI-specific checks: which AI crawlers you allow, whether content exists without JavaScript, snippet controls, and whether AI bots actually visit.
It’s not a separate discipline from technical SEO. Google is explicit about this for its own AI features: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary” (AI features and your website). Its May 2026 guide to optimizing for generative AI features goes further: “optimizing for generative AI search is optimizing for the search experience, and thus still SEO.” The other AI search engines document their own crawlers, but the same rule holds: they can only cite what they can fetch and read.
How do AI search engines find your pages?
Each major AI company runs separate user agents for separate jobs. The distinction that matters most is between search crawlers, which decide whether you can be cited in answers, and training crawlers, which collect content for future models. You can allow one and block the other.
| Company | Search / citation | Model training | User-triggered fetches |
|---|---|---|---|
Googlebot (AI Overviews and AI Mode use the Search index) |
Google-Extended (a robots.txt token only, no separate crawler) |
— | |
| OpenAI | OAI-SearchBot |
GPTBot |
ChatGPT-User |
| Anthropic | Claude-SearchBot |
ClaudeBot |
Claude-User |
| Perplexity | PerplexityBot |
Not used for foundation models, per Perplexity | Perplexity-User |
What each company says about its crawlers:
- Google:
Google-Extendedcontrols whether content can be used “for training future generations of Gemini models”, and “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google’s common crawlers). Blocking it does not remove you from AI Overviews; only Search controls do that. - OpenAI: OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features”, and “sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” GPTBot is for training. For ChatGPT-User, “robots.txt rules may not apply” because a user started the request (OpenAI crawlers).
- Anthropic: blocking Claude-SearchBot “prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results”; ClaudeBot collects content that could contribute to training (Anthropic’s crawler policy). For its search, user-request fetches and citations together, see how to appear in Claude web search.
- Perplexity: PerplexityBot is designed “to surface and link websites in search results on Perplexity”, while Perplexity-User “generally ignores robots.txt rules” because users trigger it (Perplexity crawlers).
For a Perplexity-specific access and citation check, see Perplexity search SEO: it covers the two agents, WAF verification and readable page content without claiming that crawler access guarantees a citation.
The practical consequence: a blanket “block all AI bots” rule copied from a forum usually blocks the search crawlers too, and removes you from the answers you were hoping to appear in.
The checklist at a glance
| # | Area | The question | Blocks visibility when it fails? |
|---|---|---|---|
| 1 | Crawler access | Do robots.txt, the CDN and the firewall let the right bots in? | Yes, completely |
| 2 | Crawl evidence | Do server logs show AI crawlers fetching your pages, successfully? | It tells you whether 1 worked |
| 3 | Indexability | Do important pages return 200, allow indexing and point canonicals at themselves? | Yes |
| 4 | Snippet controls | Are nosnippet, max-snippet or data-nosnippet limiting quotes by accident? |
Partly |
| 5 | Rendering | Is the content in the raw HTML, or only after JavaScript runs? | Yes, for most AI crawlers |
| 6 | Content structure | Can a machine find the answer on the page and quote it? | Reduces citations |
| 7 | Structured data | Is it valid, and does it match the visible text? | Reduces rich results |
| 8 | Discovery | Can crawlers find every important page through links and sitemaps? | Delays or prevents crawling |
| 9 | Reliability and speed | Do pages respond quickly and without errors, to bots too? | Reduces crawling |
| 10 | Measurement | Can you see AI-driven impressions, visits and crawl activity? | No, but you’re flying blind |
1. Can AI crawlers reach the site?
This is the check that most often fails silently, because the block usually isn’t in your CMS. There are three layers to check.
robots.txt
Read your robots.txt with the table above in mind. A typical policy that allows AI search but opts out of training looks like this:
# AI search and citations: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/
Disallow: /account/
# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /cart/
Disallow: /account/
Whether to opt out of training is a business decision, not an SEO one. The full list of AI crawlers and the rules for each policy are in how to block AI crawlers. What matters for the audit is that the file says what you intend. Watch for:
User-agent: *withDisallow: /left over from staging, which blocks every bot without its own group.- Bot groups you didn’t know about, added by a plugin or a security tool.
- Rules you expected to apply but don’t. Google’s robots.txt documentation says “only one group is valid for a particular crawler”: the one with the most specific matching user agent. A bot with its own group ignores everything under
*, which is why the example repeats the/cart/and/account/rules. - Delays: OpenAI notes it “can take ~24 hours from a site’s robots.txt update for our systems to adjust.”
CDN, firewall and bot protection
Google’s list of best practices for AI features starts with “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure”. That second half is easy to miss. Bot-management features on CDNs and hosting platforms can return 403 errors or challenge pages to crawlers that robots.txt allows.
Cloudflare is the clearest example, because its defaults have changed twice:
- In July 2025 it changed the default “to block AI crawlers unless they pay creators for their content.”
- In July 2026 it sorted AI traffic into Search, Training and Agent. From September 15, 2026, for new domains, “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”
- The same announcement says “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.” In other words, a training opt-out at the CDN level can now keep out the crawler your Google rankings depend on.
If your site is behind a CDN, open its bot settings, write down what each category and crawler is set to, and compare that with your robots.txt. Cloudflare’s AI Crawl Control lets site owners “set allow or block rules for individual crawlers”. If you want to opt out of AI training without touching Search, robots.txt tokens such as Google-Extended and GPTBot express that more precisely than a blanket block.
How to test access
Crawl the site as the bot. In Crawlens, set the crawl’s User-Agent to a custom string containing the bot’s token, for example Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot), and keep Respect robots.txt on. Crawlens matches robots.txt groups against the bot token in the user agent, so:
- URLs that robots.txt disallows for that bot are reported as Blocked by robots.txt.
- URLs where the server or CDN refuses the bot show up as 4xx responses or pages that did not respond.
Compare that crawl with a normal one. Any difference is a rule targeting that bot.
One limit: a crawl from your own machine sends the bot’s name, not its IP address. Some bot protection verifies crawlers by IP (OpenAI publishes its ranges, for example at openai.com/searchbot.json), so the real bot may be treated differently from your test, in either direction. Server logs settle it. For ChatGPT specifically, see how to appear in ChatGPT search.
2. Do AI crawlers actually visit, and what do they get?
Robots.txt says what bots may do. Server logs show what they did. Import your access logs into Crawlens and the log analysis groups requests by crawler, including an AI crawlers group that covers user agents such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Perplexity-User. A daily chart shows AI crawler requests next to Googlebot and Bingbot, and each crawler group shows its requests, unique URLs and how many got a 200 response.
What to look for:
- No AI crawler requests at all while the site is public: something upstream is blocking them, or the logs come from a server behind a cache that answers them first.
- Requests that don’t return 200. In Vercel’s analysis of AI crawler traffic (December 2024), “ChatGPT spends 34.82% of its fetches on 404 pages” and Claude 34.16%, and ChatGPT spent another 14.36% following redirects. Old URLs, broken links and redirect chains waste AI crawls just as they waste Googlebot’s. See how to find broken links and redirect chains.
- Which sections they fetch. If AI crawlers concentrate on your blog and never reach product or documentation pages, look at internal linking and sitemaps (section 8).
Crawlens can verify Googlebot and Bingbot with reverse and forward DNS lookups. AI crawler user agents are taken at their word, so treat a sudden spike from an “AI bot” with some suspicion until you’ve checked the IPs against the operator’s published ranges. For the full method, see log file analysis for SEO.
3. Are the pages indexable?
For Google, this is the hard requirement. “To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet.” AI search engines that build their own index apply the same logic to their own crawls.
The usual blockers, and the Crawlens checks that find them:
| Problem | Check |
|---|---|
| Pages returning 4xx or 5xx | Internal pages return 4xx, 5xx |
noindex left on by mistake |
Noindex pages |
| Canonical pointing elsewhere, at an error or at a noindex page | Canonicalised pages, Canonical points to a non-200 URL, Canonical points to a noindex page |
| Linked pages blocked by robots.txt (indexable without content) | Linked pages blocked by robots.txt |
| Pages with search traffic that are no longer indexable | Pages with search traffic are not indexable |
| Indexable pages Google never shows | Indexable pages with no search impressions |
The last two need Search Console connected, and they’re the most direct signal: a page with zero impressions in Google Search is very unlikely to be cited by Google’s AI features either. For the noindex and robots.txt interaction, see noindex vs robots.txt; for canonicals, canonical tag errors.
4. Are snippet controls limiting quotes by accident?
Google applies its preview controls to AI features: “To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls.” That’s useful when you mean it, and harmful when you don’t:
| Control | Where | Effect |
|---|---|---|
<meta name="robots" content="nosnippet"> |
Page head, or X-Robots-Tag header |
No text snippet shown from the page |
max-snippet:[number] |
Robots meta or header | Caps the length of text shown; a low value limits what can be quoted |
data-nosnippet |
Attribute on an HTML element | Excludes that element’s text from snippets |
There’s also a site-wide switch. Search Console has a setting to exclude a site from Search generative AI features (AI Overviews, AI Mode and Discover’s generative features). Sites are included by default; if yours shows as excluded and nobody decided that, change it back. See how to appear in Google AI Overviews.
Page-level controls often arrive through a theme, an SEO plugin setting or a legal team’s request, and then apply site-wide. To audit them in Crawlens:
- Open any URL and check Meta robots and X-Robots-Tag in its details.
- To check every page, add two custom extractors: a CSS extractor
meta[name="robots"]with the output set to thecontentattribute, and a CSS extractor[data-nosnippet]with All matches on. - Add a custom rule that flags indexable pages where the robots extraction contains
nosnippetormax-snippet, so it shows up in the audit with every crawl.
See custom extraction for SEO for how extractors and rules work.
5. Is the content in the HTML, or only after JavaScript?
This is where AI crawlers differ most from Google. Googlebot renders JavaScript. According to Vercel’s analysis, “none of the major AI crawlers currently render JavaScript”: they fetch JavaScript files (11.50% of ChatGPT’s requests, 23.84% of Claude’s) but don’t execute them. The exceptions it found were Gemini, which “leverages Googlebot’s infrastructure”, and AppleBot, which renders through a browser-based crawler.
So a client-rendered page can rank in Google and still look empty to ChatGPT, Claude and Perplexity crawlers. Vercel’s first recommendation was to server-side render critical content.
Crawlens compares the raw HTML with the rendered page when Render JavaScript is turned on for the crawl, and flags three gaps:
| Check | What it flags | Why it matters for AI search |
|---|---|---|
| Content depends on JavaScript | Pages where the raw HTML holds less than half the words of the rendered page (and the rendered page has at least 50) | Non-rendering crawlers see a nearly empty page |
| Links only in rendered HTML | Pages with links that appear only after rendering | Non-rendering crawlers can’t follow them to discover other pages |
| JavaScript changes SEO tags | Scripts that rewrite the title, description, canonical or robots tags | Non-rendering crawlers see the raw version, which may say something different |
Open any flagged URL to see its word count in HTML next to its rendered word count. The fix is server-side rendering, static generation or pre-rendering for the pages you want cited. The JavaScript SEO guide covers the rendering choices; the raw-versus-rendered HTML audit covers the comparison.
6. Can a machine find and quote the answer?
Once a crawler has the text, the page still has to be easy to use as a source. Google’s own list of best practices includes “making sure that important content is available in textual form”, and notes that AI Overviews and AI Mode may use “query fan-out”, “issuing multiple related searches across subtopics and data sources.” A page that clearly covers a topic’s subtopics gives fan-out searches more to match.
None of the following is a documented ranking requirement. They’re editorial habits that make content easier for both people and machines to extract:
- One clear H1 stating the topic, and descriptive
##headings. Many headings can be the questions readers ask. - The answer first. Put a direct answer in the first sentence or two under each question, then the detail.
- Tables for comparisons, thresholds and specs, and numbered lists for processes.
- Facts with sources, dates and named authors, so a claim can be checked.
- Text alternatives: alt text on meaningful images, transcripts or summaries for video, data in HTML rather than only inside a chart image.
What a crawl can check here:
| Check | Signal |
|---|---|
| Missing H1, Multiple H1 headings | Unclear main topic |
| Thin content (under 200 words) | Too little text to quote from |
| Duplicate content, Near-duplicate content | Several URLs competing to be the source |
| Images missing alt text | Visual information with no text alternative |
| Missing page title, Duplicate page titles | Weak description of what the page is |
See thin content, duplicate content and the image SEO audit for how to fix each.
7. Is structured data valid and does it match the page?
Google lists “making sure your structured data matches the visible text on the page” among its best practices for AI features. It doesn’t require any special schema type for them. Structured data still earns its place through rich results: Google’s structured data introduction cites Rotten Tomatoes measuring a 25% higher click-through rate on pages with structured data.
Check three things:
- Validity. A JSON-LD block with a syntax error is ignored entirely. Crawlens flags Invalid JSON-LD.
- Completeness. Structured data missing required properties isn’t eligible for rich results.
- Consistency. Prices, ratings, dates and authors in the markup should match what the page shows. Use custom extraction to pull the visible value and compare it with the markup on templated pages.
More in structured data errors.
8. Can crawlers find every important page?
Google’s best practices include “making your content easily findable through internal links on your website”. The same applies to every crawler that discovers pages by following links, and doubly so for crawlers that can’t follow links added by JavaScript.
| Check | Problem |
|---|---|
| Orphan pages | In the sitemap, but no internal link points to them |
| Pages more than 4 clicks from the start | Hard to reach, crawled less |
| Pages with only one internal link | Easy to lose |
| Non-indexable URLs in sitemaps | The sitemap sends crawlers to the wrong URLs |
| Sitemaps that could not be read | The sitemap is ignored |
| URLs with impressions the crawl did not find | Google shows pages your link structure doesn’t reach |
Sitemaps deserve a second look for AI search. Bing’s webmaster team wrote in Keeping Content Discoverable with Sitemaps in AI Powered Search that lastmod “should reflect the true last modification time of the page content, not the sitemap file itself”, in ISO 8601 format with date and time, and that changefreq and priority “are ignored by Bing”. It recommends pairing sitemaps with IndexNow for fast URL-level submission. Check your generator: many CMSs stamp every URL with the time the sitemap was built. More in Bing and Copilot SEO.
Related guides: internal linking audit, orphan pages, XML sitemap audit.
9. Do pages respond quickly and reliably, to bots too?
Crawlers that fetch pages live, at the moment a user asks, can’t wait long. Crawlers that build an index slow down when a server struggles: Google’s HTTP status code documentation says 5xx and 429 responses “prompt Google’s crawlers to temporarily slow down with crawling”. Google also lists “providing a great page experience for users” among its AI feature best practices.
| Check | What it tells you |
|---|---|
| Internal pages return 5xx, Pages did not respond | Errors and timeouts under crawl load |
| Slow server response (over 1 second) | Slow time to first byte |
| Poor LCP, INP and CLS for real users | Page experience from Chrome field data |
| Pages PageSpeed Insights could not test | Often bot protection, which may block other bots too |
The last one is worth a second look in an AI readiness audit. If Google’s own testing service can’t load a page, bot protection is often the reason, and the same rules may be turning away AI crawlers. See the Core Web Vitals audit.
10. Should you add an llms.txt file?
llms.txt is “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website”, proposed by Jeremy Howard in September 2024. It’s a Markdown file at the site root with the site name, a short summary and links to the most useful pages.
Not for Google. Google’s guide to optimizing for generative AI features says you don’t need “AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them”, and that creating one “will neither harm nor help”. None of the crawler documentation quoted above mentions llms.txt either. It’s cheap to publish and can help AI agents that read documentation directly, but it can’t fix any of the problems in sections 1–9. See llms.txt: what it is and whether you need it. For agents that use your site rather than read it, see how to make your website work for AI agents.
How do you measure AI search visibility?
There’s no single report, so combine four sources:
- Search Console. The Generative AI performance report shows impressions in AI Overviews and AI Mode by page, country, device and date (once a site has enough of them). Clicks from those features are included in the regular Performance report under the “Web” search type. Connect Search Console in Crawlens to see clicks and impressions next to the crawl data.
- Analytics. Google Analytics has an AI Assistant channel for visits from sources such as ChatGPT, Gemini, Deepseek, Copilot and Grok, and counts AI Overviews and AI Mode as Organic Search (default channel group). OpenAI’s publisher FAQ adds that “ChatGPT automatically includes the UTM parameter utm_source=chatgpt.com in referral URLs.” See how to track AI search traffic.
- Server logs. AI crawler requests over time show whether access changes worked (section 2).
- Re-crawls. Crawl again after each fix and use Compare to confirm that blocked, non-indexable and JavaScript-dependent pages dropped out of their checks.
You can also hand the audit to an AI assistant: Crawlens’s MCP server lets Claude read your crawls, issues and URLs directly. See running a technical SEO audit with Claude.
The full checklist
Crawler access
- robots.txt allows the search crawlers you want: Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot
- Training crawlers (GPTBot, ClaudeBot, Google-Extended) allowed or blocked by deliberate decision
- No leftover
Disallow: /and no unexpected bot groups - CDN, firewall and bot protection settings match robots.txt
- A CDN-level “block AI training” setting doesn’t also block Googlebot or Bingbot
- Test crawl with each AI search bot’s user agent shows no unexpected blocks or 403s
Crawl evidence
- Server logs show AI crawler requests
- AI crawler requests mostly return 200, not 404 or redirects
- Spikes checked against the operators’ published IP ranges
Indexability
- Important pages return 200, have no noindex and a self-referencing canonical
- No pages with search traffic turned non-indexable
- Indexable pages with zero impressions reviewed
Snippet controls
- No unintended
nosnippetor lowmax-snippet, in meta tags or headers -
data-nosnippetused only where intended
Rendering
- Main content present in the raw HTML
- Navigation and internal links present in the raw HTML
- Title, description, canonical and robots tags not changed by JavaScript
Content structure
- One H1 per page; descriptive, question-led headings
- Direct answer first under each question
- Tables and lists for comparisons, specs and steps
- No thin or duplicate pages competing to be the source
- Text alternatives for images, video and charts
Structured data
- Valid JSON-LD with required properties
- Markup matches the visible content
Discovery
- No orphan pages or important pages deeper than four clicks
- Sitemaps list only indexable 200 URLs
-
lastmodreflects real content changes
Reliability and speed
- No 5xx errors or timeouts under crawl load
- Server responds in under a second
- Core Web Vitals pass for real users
Measurement
- Search Console connected and pages tracked over time
- Analytics segment for AI assistant referrals (utm_source=chatgpt.com and others)
- Re-crawl and compare after each round of fixes
All AI Search guides
- Perplexity search SEO: check crawler access and citation-ready pagesA cautious technical audit for Perplexity visibility: PerplexityBot, user-triggered fetches, WAF access, readable pages and citation checks.
- How to make your website work for AI agentsHow AI agents read and use web pages, and what web.dev recommends so they can complete tasks: semantic HTML, labels, visible controls and stable layouts.
- How to track AI search traffic: ChatGPT, Perplexity and AI OverviewsWhere AI search traffic shows up in Search Console, GA4 and server logs, which visits each one misses, and how to build a simple view of AI visibility.
- Bing and Copilot SEO: sitemaps, IndexNow and AI answer controlsWhat Bing recommends for its search and AI-powered answers: accurate sitemaps and lastmod, IndexNow for fast updates, and controls for AI answers.
- How to block AI crawlers without disappearing from AI searchEvery major AI crawler and its purpose, robots.txt rules to opt out of AI training but stay in AI search answers, and what robots.txt can't stop.
- How to appear in ChatGPT search: what OpenAI says and what to checkWhat OpenAI says about getting into ChatGPT search, the crawler that matters, how to opt out, how to track ChatGPT traffic and what to check on your site.
- How to appear in Claude web search: what Anthropic saysHow Claude finds and cites web pages, what its search and user-fetch bots do, and how to check whether your site is accessible and useful as a source.
- How to appear in Google AI Overviews and AI ModeWhat Google says decides which pages AI Overviews and AI Mode cite, what makes a page eligible, which AEO and GEO tactics you can ignore, and how to measure it.
- llms.txt: what it is, how to write one, and whether you need itWhat the llms.txt proposal is, the exact file format, what Google says about it, where it's genuinely useful, and how to write and maintain one.
- How to use Claude for a technical SEO audit (with MCP)Connect Crawlens to Claude over MCP and turn a long issue list into a prioritised, developer-ready plan built on your own crawl data.
Frequently asked questions
Do I need to do anything special to appear in Google AI Overviews?
No. Google says there are no additional requirements or special optimizations for AI Overviews or AI Mode. A page must be indexed and eligible to show in Google Search with a snippet, and the site must not be opted out of generative AI features in Search Console, which it isn't by default.
Will blocking GPTBot remove my site from ChatGPT search?
No. OpenAI uses GPTBot for training data and OAI-SearchBot for ChatGPT search. Blocking GPTBot opts out of training; sites that block OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links.
Does blocking Google-Extended affect my Google rankings?
No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't a ranking signal. It controls whether content Google crawls can be used to train future Gemini models and for grounding in some other Google systems.
Do AI crawlers render JavaScript?
Mostly not. Vercel's December 2024 analysis of its network found that none of the major AI crawlers it measured rendered JavaScript, while Gemini uses Googlebot's rendering. Content that only appears after scripts run may be invisible to ChatGPT, Claude and Perplexity crawlers.
Do I need an llms.txt file?
Not for Google. Google says Google Search, including AI Overviews and AI Mode, doesn't use llms.txt, and that creating one neither helps nor harms. It can still be useful for documentation and developer tools that people use with AI agents.