New researchTechnical SEO & AI Search · 2027Explore the framework

Free SEO tool · Google matching rules

Robots.txt Tester & AI Crawler Checker

Enter a URL to fetch its live robots.txt and see whether Googlebot, Bingbot and the main AI crawlers may crawl it, with the line that decides. Edit the file and test again before you deploy.

18 crawlers checked at onceEdit and retest in the browserGoogle's robots.txt rules ↗
Test a URL

Which crawlers can reach this page?

Enter a page URL or a domain. The tester reads robots.txt from the same protocol and host, the way crawlers do.

How it works

How Google reads robots.txt

The tester applies the same rules, from Google's documentation and the Robots Exclusion Protocol standard (RFC 9309).

Which group appliesA crawler follows the group that names its user-agent token, case-insensitively. Several groups for the same crawler are combined. Only when no group names it does it follow User-agent: *; the two are never merged.
Which rule winsThe matching rule with the longest path wins. When an Allow and a Disallow are equally long, Google uses the less restrictive one, Allow.
Wildcards* matches any sequence of characters and a $ at the end anchors the end of the URL. Paths are case-sensitive and must start with /.
Supported fieldsuser-agent, allow, disallow and sitemap. Google ignores others, such as crawl-delay.
ScopeA robots.txt file covers only its own protocol, host and port. blog.example.com needs its own file.
Missing or failing fileA 4xx response (except 429) counts as no file: everything is allowed. A 5xx, 429 or network error makes Google stop crawling the site for 12 hours, then use its last good copy for up to 30 days.
Redirects and sizeGoogle follows at least five redirects for robots.txt, then treats it as missing. It reads the first 500 KiB and ignores the rest.
CachingGoogle generally caches robots.txt for up to 24 hours, so a change can take a day to apply.

Sources: How Google interprets the robots.txt specification and RFC 9309.

Crawler list

The crawlers this tool checks

Each token is documented by the company that runs it. Crawlers change, so check the operator's page before relying on a rule.

TokenOperatorUsed forSource
GooglebotGoogleSearch enginesAlso AI Overviews and AI ModeDocumentation ↗
BingbotMicrosoftSearch enginesDocumentation ↗
ApplebotAppleSearch enginesSpotlight, Siri and Safari; data may also train Apple modelsDocumentation ↗
OAI-SearchBotOpenAIAI searchDocumentation ↗
Claude-SearchBotAnthropicAI searchDocumentation ↗
PerplexityBotPerplexityAI searchDocumentation ↗
Amzn-SearchBotAmazonAI searchDocumentation ↗
GPTBotOpenAIAI trainingDocumentation ↗
ClaudeBotAnthropicAI trainingDocumentation ↗
Google-ExtendedGoogleAI trainingA robots.txt token, not a separate crawlerDocumentation ↗
Applebot-ExtendedAppleAI trainingA robots.txt token, not a separate crawlerDocumentation ↗
meta-externalagentMetaAI trainingDocumentation ↗
AmazonbotAmazonAI trainingImproves Amazon products; may train Amazon AI modelsDocumentation ↗
CCBotCommon CrawlAI trainingDocumentation ↗
ChatGPT-UserOpenAIUser-triggered fetchesMay ignore robots.txtDocumentation ↗
Claude-UserAnthropicUser-triggered fetchesDocumentation ↗
Perplexity-UserPerplexityUser-triggered fetchesMay ignore robots.txtDocumentation ↗
meta-externalfetcherMetaUser-triggered fetchesMay ignore robots.txtDocumentation ↗

Next step

Fix what the test turns up

A blocked URL is only a problem when you wanted it crawled. These guides cover the usual cases.

Rules

Write robots.txt correctly

Syntax, matching and the mistakes that block more than you meant, such as a stray Disallow: /.

robots.txt for SEO
AI

Choose which AI crawlers to allow

Opt out of AI training while staying citable in ChatGPT, Claude and Perplexity answers.

How to block AI crawlers
Index

Keep a page out of search

Robots.txt stops crawling, not indexing. To remove a page from results, use noindex instead.

Noindex vs robots.txt

Good to know

Robots.txt tester FAQ

Sources: Google Search Central and RFC 9309.

How do I test if a URL is blocked by robots.txt?

Enter the URL above: the tester fetches the live robots.txt for that host and shows, for each crawler, whether the URL is allowed and which line decides. For Googlebot specifically, Search Console's URL Inspection tool also reports whether a URL is blocked.

Does blocking a page in robots.txt remove it from Google?

No. Google says a disallowed URL can still be indexed without being crawled if other pages link to it, usually shown without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or header.

Which rule wins when Allow and Disallow conflict?

The rule with the longest matching path. If an Allow and a Disallow rule are the same length, Google uses the less restrictive Allow rule.

Why does Googlebot ignore my User-agent: * rules?

Because your file has a group for Googlebot. A crawler that finds a group naming it follows only that group and ignores the * group, so repeat any shared rules inside the Googlebot group.

What happens if robots.txt returns a 404 or a 500 error?

A 404 or other 4xx (except 429) counts as no robots.txt, so everything may be crawled. A 5xx or 429 makes Google stop crawling the site for 12 hours; after that it uses the last good copy for up to 30 days.

Test robots.txt across the whole site

Crawlens crawls with the user-agent you pick and can follow its robots.txt rules, flags linked pages that robots.txt blocks, and its server log analysis shows which AI crawlers actually requested your pages.

Download free for Windows