Part of the AI Search series. For the full picture, start with Technical SEO and AI search readiness: the complete audit checklist.
“Block AI bots” sounds like one switch. It’s several. Each AI company runs different crawlers for different jobs: one builds a search index so it can cite you, one collects training data, and one fetches pages live when a user asks about them. Block all of them and you’ve opted out of training, but you’ve also opted out of being a source in ChatGPT, Claude and Perplexity answers.
This guide lists the AI crawlers that are documented by the companies that run them, what each one is for, the robots.txt rules for the common policies, and the gaps robots.txt can’t close.
Should you block AI crawlers at all?
Decide per purpose, not per company:
| Purpose | If you allow it | If you block it |
|---|---|---|
| Search / citation | Your pages can appear and be linked in AI answers | You’re not a source in that product’s answers |
| Model training | Your content may be used to train future models | Opted out of future training (it doesn’t remove past data) |
| User-triggered fetch | The assistant can read your page when a user asks about it | Often not fully blockable with robots.txt (see below) |
A common position for publishers and businesses: allow search, opt out of training. The rest of this guide shows how.
Which AI crawlers are there, and what does each do?
| Company | Token | Purpose | Notes from the operator |
|---|---|---|---|
Googlebot |
Search, including AI Overviews and AI Mode | Blocking it removes you from Google Search | |
Google-Extended |
Gemini training and grounding | A robots.txt token only; “does not impact a site’s inclusion in Google Search” | |
| OpenAI | OAI-SearchBot |
ChatGPT search | Blocked sites “will not be shown in ChatGPT search answers” |
| OpenAI | GPTBot |
Training | “Disallowing GPTBot indicates a site’s content should not be used in training” |
| OpenAI | ChatGPT-User |
User actions in ChatGPT | “Robots.txt rules may not apply” |
| Anthropic | Claude-SearchBot |
Claude search results | Blocking “may reduce your site’s visibility” in user search results |
| Anthropic | ClaudeBot |
Training | Blocking signals content “should be excluded from our AI model training datasets” |
| Anthropic | Claude-User |
Fetches when Claude users ask | Blocking prevents retrieval “in response to a user query” |
| Perplexity | PerplexityBot |
Perplexity search results | “Not used to crawl content for AI foundation models” |
| Perplexity | Perplexity-User |
Fetches when users ask | “Generally ignores robots.txt rules” |
| Apple | Applebot |
Spotlight, Siri and Safari search; data may also train Apple’s models | |
| Apple | Applebot-Extended |
Training opt-out | “Does not crawl webpages”; disallowed pages “can still be included in search results” |
| Amazon | Amzn-SearchBot |
Search in Amazon products | “Does not crawl content for generative AI model training” |
| Amazon | Amazonbot |
Improving Amazon products | “May be used to train Amazon AI models” |
| Meta | meta-externalagent |
Training AI models and indexing | Can be blocked in robots.txt |
| Meta | meta-externalfetcher |
Fetches links at a user’s request | “May bypass robots.txt rules” |
| Common Crawl | CCBot |
An “open repository of web crawl data” | Blocked with User-agent: CCBot |
Sources: Google, OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta, Common Crawl. Crawlers change: check these pages before relying on the table.
How do you opt out of AI training but stay in AI search?
Disallow the training tokens and allow the search crawlers. Remember that a crawler with its own group ignores the * group, so repeat any rules you want every crawler to follow.
# AI search and citations: allowed (with the same private areas as everyone else)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
Disallow: /account/
Disallow: /checkout/
# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /account/
Disallow: /checkout/
Sitemap: https://www.example.com/sitemap.xml
Two of those choices involve trade-offs:
- Amazonbot is used “to improve our products and services” and “may be used to train Amazon AI models”, so blocking it covers more than training.
- Applebot stays allowed because it powers Apple’s search features;
Applebot-Extendedis the training opt-out.
To block all AI crawlers, including AI search
Add the search crawlers to the disallow group as well. Keep Googlebot and Bingbot out of it unless you want to leave Google and Bing search too.
To allow everything
You don’t need any AI-specific rules. Just make sure nothing else blocks the crawlers: a stray Disallow: / in a named group, or a CDN rule (below).
For the syntax and how rules are matched, see robots.txt for SEO.
What can’t robots.txt control?
User-triggered fetchers
When someone pastes your URL into an assistant, or the assistant fetches a page to answer a question, the request comes from a user-triggered agent. Making pages usable for those agents is covered in how to make a website work for AI agents. OpenAI says robots.txt “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores robots.txt rules”, and Meta says meta-externalfetcher “may bypass robots.txt rules.” Only server-side blocking stops these, and it also stops the assistant from reading your page for its users.
Google’s AI features
AI Overviews and AI Mode are part of Google Search, so Google-Extended doesn’t control them. Two other controls do:
- Search Console’s generative AI setting excludes a whole site’s “links and content from Search generative AI features”, including AI Overviews, AI Mode and Discover’s generative features. Google says the control “isn’t used as a ranking or inclusion signal affecting other parts of Search”, and that it doesn’t stop AI training (help page).
- Page-level preview controls. Google’s AI features documentation says: “To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls.” Those limit your appearance in regular results too.
More in how to appear in Google AI Overviews.
Other page-level controls
| Control | Who honours it | Effect |
|---|---|---|
| Search Console generative AI setting | Excludes the whole site from AI Overviews, AI Mode and Discover’s generative features | |
nosnippet, max-snippet, data-nosnippet |
Limits text shown in Search, including AI features | |
noarchive |
Amazon | Amazon says it treats it as “do not use the page for model training” |
nocache |
Bing | Bing said (September 2023) content “may be included in Bing Chat answers”, showing only URL, snippet and title |
noarchive |
Bing | Bing said such content “will not be included in Bing Chat answers” and won’t be used to train Microsoft’s generative AI models |
Bing’s controls are described in its announcement for Bing Chat, the product since renamed Copilot. In both cases Bing said content still appears in Bing search results.
Crawlers that don’t identify themselves
Robots.txt is a request, not a lock. A crawler that uses a browser user agent won’t match any group. Rate limits and bot management at the server or CDN are the only defence.
Should you block AI crawlers at the CDN?
CDN rules can enforce what robots.txt only asks, and they catch crawlers that ignore it. They can also overreach. Cloudflare’s July 2026 announcement sorts AI traffic into Search, Training and Agent, and warns that “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.” For new domains from September 15, 2026, “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”
If you use CDN-level AI blocking:
- Check which named crawlers each category includes, and whether Googlebot and Bingbot are among them.
- Prefer robots.txt tokens like
Google-ExtendedandApplebot-Extendedfor training opt-outs from multi-purpose companies; they leave search crawling untouched. - Verify crawlers by IP where the operator publishes ranges (OpenAI publishes them, for example at
openai.com/gptbot.jsonandopenai.com/searchbot.json), so you don’t block real bots or allow fake ones.
How do you check that your rules work?
For a quick check of one URL, the free AI crawler checker reads your live robots.txt and shows which search, AI search, AI training and user-triggered crawlers it allows, and which line blocks each one. To check the whole site:
- Read the file as each bot. Crawl your site with a custom user agent containing the bot’s token, for example
Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot). Crawlens applies the robots.txt group that matches the token in the user agent, so with Respect robots.txt on, everything that bot may not crawl shows as Blocked by robots.txt. Repeat withOAI-SearchBot/1.0and compare: the training bot should be blocked, the search bot allowed. - Watch for CDN blocks in the same crawl. If the search bot’s crawl returns 4xx responses or pages that did not respond that a normal crawl doesn’t, something upstream is filtering by user agent. Your test crawl comes from your own IP, so IP-verified rules may treat the real bot differently.
- Check the logs. Import server logs: Crawlens groups GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User and other AI user agents as AI crawlers, with a daily chart next to Googlebot and Bingbot. After a robots.txt change, blocked training crawlers should stop requesting the disallowed paths within a day or two (OpenAI says “~24 hours”).
- Keep Search healthy. Whatever you block, Googlebot and Bingbot must still reach every page that should rank. A normal crawl with the Googlebot Smartphone user agent confirms it.
For the rest of what AI search needs, from rendering to snippet controls, see the AI search readiness checklist.
Checklist
- Policy decided per purpose: search, training, user fetches
- Training tokens disallowed if you’re opting out: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot (and Amazonbot, with its trade-off)
- Search crawlers allowed if you want AI citations: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot
- Shared rules repeated in every named group
- Googlebot, Bingbot and Applebot not blocked by accident
- Snippet controls (nosnippet, max-snippet, noarchive, nocache) used only on purpose
- CDN AI settings checked, including whether they cover multi-purpose crawlers
- Test crawls as a training bot and a search bot behave as intended
- Server logs confirm the change after a day or two
Frequently asked questions
How do I block ChatGPT from training on my site?
Disallow GPTBot in robots.txt. OpenAI says disallowing GPTBot indicates a site's content shouldn't be used in training. Leave OAI-SearchBot allowed if you still want to appear in ChatGPT search answers.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google-Extended controls use for Gemini model training and grounding in some other Google systems, and Google says it doesn't affect inclusion in Google Search. AI Overviews and AI Mode are part of Search: Search Console has a site-level setting to exclude a site from them, and noindex and snippet controls work per page.
Do AI crawlers respect robots.txt?
The major documented crawlers for search and training say they do. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says meta-externalfetcher may bypass them.
How long does it take for robots.txt changes to apply to AI crawlers?
It varies by company. OpenAI says it can take about 24 hours for its systems to adjust, and Google generally caches robots.txt for up to 24 hours. Check your server logs after a day or two to confirm the change.
Can I block AI crawlers at the CDN instead of robots.txt?
Yes, and CDN rules also stop crawlers that ignore robots.txt. But check what each setting covers: Cloudflare says customers who block Training also block multi-purpose crawlers such as Googlebot, Applebot and Bingbot.