Write robots.txt correctly
Syntax, matching and the mistakes that block more than you meant, such as a stray Disallow: /.
robots.txt for SEOFree SEO tool · Google matching rules
Enter a URL to fetch its live robots.txt and see whether Googlebot, Bingbot and the main AI crawlers may crawl it, with the line that decides. Edit the file and test again before you deploy.
Enter a page URL or a domain. The tester reads robots.txt from the same protocol and host, the way crawlers do.
Results
Testing your edited robots.txt, not the live file.
| Crawler | Result | Deciding rule |
|---|
Edit the file to test a change. Nothing is saved or sent anywhere; results update as you type.
How it works
The tester applies the same rules, from Google's documentation and the Robots Exclusion Protocol standard (RFC 9309).
| Which group applies | A crawler follows the group that names its user-agent token, case-insensitively. Several groups for the same crawler are combined. Only when no group names it does it follow User-agent: *; the two are never merged. |
|---|---|
| Which rule wins | The matching rule with the longest path wins. When an Allow and a Disallow are equally long, Google uses the less restrictive one, Allow. |
| Wildcards | * matches any sequence of characters and a $ at the end anchors the end of the URL. Paths are case-sensitive and must start with /. |
| Supported fields | user-agent, allow, disallow and sitemap. Google ignores others, such as crawl-delay. |
| Scope | A robots.txt file covers only its own protocol, host and port. blog.example.com needs its own file. |
| Missing or failing file | A 4xx response (except 429) counts as no file: everything is allowed. A 5xx, 429 or network error makes Google stop crawling the site for 12 hours, then use its last good copy for up to 30 days. |
| Redirects and size | Google follows at least five redirects for robots.txt, then treats it as missing. It reads the first 500 KiB and ignores the rest. |
| Caching | Google generally caches robots.txt for up to 24 hours, so a change can take a day to apply. |
Sources: How Google interprets the robots.txt specification and RFC 9309.
Crawler list
Each token is documented by the company that runs it. Crawlers change, so check the operator's page before relying on a rule.
| Token | Operator | Used for | Source |
|---|---|---|---|
Googlebot | Search enginesAlso AI Overviews and AI Mode | Documentation ↗ | |
Bingbot | Microsoft | Search engines | Documentation ↗ |
Applebot | Apple | Search enginesSpotlight, Siri and Safari; data may also train Apple models | Documentation ↗ |
OAI-SearchBot | OpenAI | AI search | Documentation ↗ |
Claude-SearchBot | Anthropic | AI search | Documentation ↗ |
PerplexityBot | Perplexity | AI search | Documentation ↗ |
Amzn-SearchBot | Amazon | AI search | Documentation ↗ |
GPTBot | OpenAI | AI training | Documentation ↗ |
ClaudeBot | Anthropic | AI training | Documentation ↗ |
Google-Extended | AI trainingA robots.txt token, not a separate crawler | Documentation ↗ | |
Applebot-Extended | Apple | AI trainingA robots.txt token, not a separate crawler | Documentation ↗ |
meta-externalagent | Meta | AI training | Documentation ↗ |
Amazonbot | Amazon | AI trainingImproves Amazon products; may train Amazon AI models | Documentation ↗ |
CCBot | Common Crawl | AI training | Documentation ↗ |
ChatGPT-User | OpenAI | User-triggered fetchesMay ignore robots.txt | Documentation ↗ |
Claude-User | Anthropic | User-triggered fetches | Documentation ↗ |
Perplexity-User | Perplexity | User-triggered fetchesMay ignore robots.txt | Documentation ↗ |
meta-externalfetcher | Meta | User-triggered fetchesMay ignore robots.txt | Documentation ↗ |
Next step
A blocked URL is only a problem when you wanted it crawled. These guides cover the usual cases.
Syntax, matching and the mistakes that block more than you meant, such as a stray Disallow: /.
robots.txt for SEOOpt out of AI training while staying citable in ChatGPT, Claude and Perplexity answers.
How to block AI crawlersRobots.txt stops crawling, not indexing. To remove a page from results, use noindex instead.
Noindex vs robots.txtGood to know
Sources: Google Search Central and RFC 9309.
Enter the URL above: the tester fetches the live robots.txt for that host and shows, for each crawler, whether the URL is allowed and which line decides. For Googlebot specifically, Search Console's URL Inspection tool also reports whether a URL is blocked.
No. Google says a disallowed URL can still be indexed without being crawled if other pages link to it, usually shown without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or header.
The rule with the longest matching path. If an Allow and a Disallow rule are the same length, Google uses the less restrictive Allow rule.
Because your file has a group for Googlebot. A crawler that finds a group naming it follows only that group and ignores the * group, so repeat any shared rules inside the Googlebot group.
A 404 or other 4xx (except 429) counts as no robots.txt, so everything may be crawled. A 5xx or 429 makes Google stop crawling the site for 12 hours; after that it uses the last good copy for up to 30 days.
Crawlens crawls with the user-agent you pick and can follow its robots.txt rules, flags linked pages that robots.txt blocks, and its server log analysis shows which AI crawlers actually requested your pages.
Download free for Windows