New researchTechnical SEO & AI Search · 2027Explore the framework

How to block AI crawlers without disappearing from AI search

Every major AI crawler and its purpose, robots.txt rules to opt out of AI training but stay in AI search answers, and what robots.txt can't stop.

Part of the AI Search series. For the full picture, start with Technical SEO and AI search readiness: the complete audit checklist.

“Block AI bots” sounds like one switch. It’s several. Each AI company runs different crawlers for different jobs: one builds a search index so it can cite you, one collects training data, and one fetches pages live when a user asks about them. Block all of them and you’ve opted out of training, but you’ve also opted out of being a source in ChatGPT, Claude and Perplexity answers.

This guide lists the AI crawlers that are documented by the companies that run them, what each one is for, the robots.txt rules for the common policies, and the gaps robots.txt can’t close.

Should you block AI crawlers at all?

Decide per purpose, not per company:

Purpose If you allow it If you block it
Search / citation Your pages can appear and be linked in AI answers You’re not a source in that product’s answers
Model training Your content may be used to train future models Opted out of future training (it doesn’t remove past data)
User-triggered fetch The assistant can read your page when a user asks about it Often not fully blockable with robots.txt (see below)

A common position for publishers and businesses: allow search, opt out of training. The rest of this guide shows how.

Which AI crawlers are there, and what does each do?

Company Token Purpose Notes from the operator
Google Googlebot Search, including AI Overviews and AI Mode Blocking it removes you from Google Search
Google Google-Extended Gemini training and grounding A robots.txt token only; “does not impact a site’s inclusion in Google Search”
OpenAI OAI-SearchBot ChatGPT search Blocked sites “will not be shown in ChatGPT search answers”
OpenAI GPTBot Training “Disallowing GPTBot indicates a site’s content should not be used in training”
OpenAI ChatGPT-User User actions in ChatGPT “Robots.txt rules may not apply”
Anthropic Claude-SearchBot Claude search results Blocking “may reduce your site’s visibility” in user search results
Anthropic ClaudeBot Training Blocking signals content “should be excluded from our AI model training datasets”
Anthropic Claude-User Fetches when Claude users ask Blocking prevents retrieval “in response to a user query”
Perplexity PerplexityBot Perplexity search results “Not used to crawl content for AI foundation models”
Perplexity Perplexity-User Fetches when users ask “Generally ignores robots.txt rules”
Apple Applebot Spotlight, Siri and Safari search; data may also train Apple’s models
Apple Applebot-Extended Training opt-out “Does not crawl webpages”; disallowed pages “can still be included in search results”
Amazon Amzn-SearchBot Search in Amazon products “Does not crawl content for generative AI model training”
Amazon Amazonbot Improving Amazon products “May be used to train Amazon AI models”
Meta meta-externalagent Training AI models and indexing Can be blocked in robots.txt
Meta meta-externalfetcher Fetches links at a user’s request “May bypass robots.txt rules”
Common Crawl CCBot An “open repository of web crawl data” Blocked with User-agent: CCBot

Sources: Google, OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta, Common Crawl. Crawlers change: check these pages before relying on the table.

Disallow the training tokens and allow the search crawlers. Remember that a crawler with its own group ignores the * group, so repeat any rules you want every crawler to follow.

# AI search and citations: allowed (with the same private areas as everyone else)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
Disallow: /account/
Disallow: /checkout/

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Disallow: /

# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /account/
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

Two of those choices involve trade-offs:

  • Amazonbot is used “to improve our products and services” and “may be used to train Amazon AI models”, so blocking it covers more than training.
  • Applebot stays allowed because it powers Apple’s search features; Applebot-Extended is the training opt-out.

Add the search crawlers to the disallow group as well. Keep Googlebot and Bingbot out of it unless you want to leave Google and Bing search too.

To allow everything

You don’t need any AI-specific rules. Just make sure nothing else blocks the crawlers: a stray Disallow: / in a named group, or a CDN rule (below).

For the syntax and how rules are matched, see robots.txt for SEO.

What can’t robots.txt control?

User-triggered fetchers

When someone pastes your URL into an assistant, or the assistant fetches a page to answer a question, the request comes from a user-triggered agent. Making pages usable for those agents is covered in how to make a website work for AI agents. OpenAI says robots.txt “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores robots.txt rules”, and Meta says meta-externalfetcher “may bypass robots.txt rules.” Only server-side blocking stops these, and it also stops the assistant from reading your page for its users.

Google’s AI features

AI Overviews and AI Mode are part of Google Search, so Google-Extended doesn’t control them. Two other controls do:

  • Search Console’s generative AI setting excludes a whole site’s “links and content from Search generative AI features”, including AI Overviews, AI Mode and Discover’s generative features. Google says the control “isn’t used as a ranking or inclusion signal affecting other parts of Search”, and that it doesn’t stop AI training (help page).
  • Page-level preview controls. Google’s AI features documentation says: “To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls.” Those limit your appearance in regular results too.

More in how to appear in Google AI Overviews.

Other page-level controls

Control Who honours it Effect
Search Console generative AI setting Google Excludes the whole site from AI Overviews, AI Mode and Discover’s generative features
nosnippet, max-snippet, data-nosnippet Google Limits text shown in Search, including AI features
noarchive Amazon Amazon says it treats it as “do not use the page for model training”
nocache Bing Bing said (September 2023) content “may be included in Bing Chat answers”, showing only URL, snippet and title
noarchive Bing Bing said such content “will not be included in Bing Chat answers” and won’t be used to train Microsoft’s generative AI models

Bing’s controls are described in its announcement for Bing Chat, the product since renamed Copilot. In both cases Bing said content still appears in Bing search results.

Crawlers that don’t identify themselves

Robots.txt is a request, not a lock. A crawler that uses a browser user agent won’t match any group. Rate limits and bot management at the server or CDN are the only defence.

Should you block AI crawlers at the CDN?

CDN rules can enforce what robots.txt only asks, and they catch crawlers that ignore it. They can also overreach. Cloudflare’s July 2026 announcement sorts AI traffic into Search, Training and Agent, and warns that “multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training.” For new domains from September 15, 2026, “Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default.”

If you use CDN-level AI blocking:

  • Check which named crawlers each category includes, and whether Googlebot and Bingbot are among them.
  • Prefer robots.txt tokens like Google-Extended and Applebot-Extended for training opt-outs from multi-purpose companies; they leave search crawling untouched.
  • Verify crawlers by IP where the operator publishes ranges (OpenAI publishes them, for example at openai.com/gptbot.json and openai.com/searchbot.json), so you don’t block real bots or allow fake ones.

How do you check that your rules work?

For a quick check of one URL, the free AI crawler checker reads your live robots.txt and shows which search, AI search, AI training and user-triggered crawlers it allows, and which line blocks each one. To check the whole site:

  1. Read the file as each bot. Crawl your site with a custom user agent containing the bot’s token, for example Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot). Crawlens applies the robots.txt group that matches the token in the user agent, so with Respect robots.txt on, everything that bot may not crawl shows as Blocked by robots.txt. Repeat with OAI-SearchBot/1.0 and compare: the training bot should be blocked, the search bot allowed.
  2. Watch for CDN blocks in the same crawl. If the search bot’s crawl returns 4xx responses or pages that did not respond that a normal crawl doesn’t, something upstream is filtering by user agent. Your test crawl comes from your own IP, so IP-verified rules may treat the real bot differently.
  3. Check the logs. Import server logs: Crawlens groups GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User and other AI user agents as AI crawlers, with a daily chart next to Googlebot and Bingbot. After a robots.txt change, blocked training crawlers should stop requesting the disallowed paths within a day or two (OpenAI says “~24 hours”).
  4. Keep Search healthy. Whatever you block, Googlebot and Bingbot must still reach every page that should rank. A normal crawl with the Googlebot Smartphone user agent confirms it.

For the rest of what AI search needs, from rendering to snippet controls, see the AI search readiness checklist.

Checklist

  • Policy decided per purpose: search, training, user fetches
  • Training tokens disallowed if you’re opting out: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot (and Amazonbot, with its trade-off)
  • Search crawlers allowed if you want AI citations: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amzn-SearchBot
  • Shared rules repeated in every named group
  • Googlebot, Bingbot and Applebot not blocked by accident
  • Snippet controls (nosnippet, max-snippet, noarchive, nocache) used only on purpose
  • CDN AI settings checked, including whether they cover multi-purpose crawlers
  • Test crawls as a training bot and a search bot behave as intended
  • Server logs confirm the change after a day or two

Frequently asked questions

How do I block ChatGPT from training on my site?

Disallow GPTBot in robots.txt. OpenAI says disallowing GPTBot indicates a site's content shouldn't be used in training. Leave OAI-SearchBot allowed if you still want to appear in ChatGPT search answers.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended controls use for Gemini model training and grounding in some other Google systems, and Google says it doesn't affect inclusion in Google Search. AI Overviews and AI Mode are part of Search: Search Console has a site-level setting to exclude a site from them, and noindex and snippet controls work per page.

Do AI crawlers respect robots.txt?

The major documented crawlers for search and training say they do. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says meta-externalfetcher may bypass them.

How long does it take for robots.txt changes to apply to AI crawlers?

It varies by company. OpenAI says it can take about 24 hours for its systems to adjust, and Google generally caches robots.txt for up to 24 hours. Check your server logs after a day or two to confirm the change.

Can I block AI crawlers at the CDN instead of robots.txt?

Yes, and CDN rules also stop crawlers that ignore robots.txt. But check what each setting covers: Cloudflare says customers who block Training also block multi-purpose crawlers such as Googlebot, Applebot and Bingbot.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows