Part of the Crawling & Indexing series. For the full picture, start with Crawling and indexing: how search engines find your pages, and what stops them.
Robots.txt is the smallest file on most sites and one of the few that can take a whole site out of search in one line. It’s also widely misunderstood: people use it to “hide” pages it can’t hide, and miss the rules that are actually blocking them.
This guide covers what robots.txt does and doesn’t do, the syntax, how crawlers decide which rules apply, examples you can adapt, what happens when the file breaks, and how to test it against a real crawl.
What is robots.txt?
Robots.txt is a plain-text file at the root of a host that tells crawlers which URLs they may request. In Google’s words, it “tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests” (introduction to robots.txt).
What it is not:
- Not an indexing control. Google is explicit that “it is not a mechanism for keeping a web page out of Google.” A disallowed URL can still be indexed if other pages link to it, typically with no description. Use
noindexfor that; see noindex vs robots.txt. - Not security. Reputable crawlers obey it; others don’t. Anything private needs authentication.
Where does robots.txt go?
At the root of each host: https://www.example.com/robots.txt. Google’s robots.txt specification says its rules “apply only to the host, protocol, and port number where the robots.txt file is hosted.”
| File | Applies to | Doesn’t apply to |
|---|---|---|
https://example.com/robots.txt |
https://example.com/… |
https://shop.example.com/…, http://example.com/…, https://example.com:8443/… |
https://shop.example.com/robots.txt |
https://shop.example.com/… |
Every other host |
Subdomains are the usual surprise: a blog on blog.example.com or a shop on a platform subdomain needs its own file. Moving that blog into a folder on the main domain is a migration of its own: see subdomain to subfolder migration.
What is the robots.txt syntax?
A file is a set of groups. Each group starts with one or more user-agent lines, followed by the rules for those crawlers.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
Sitemap: https://www.example.com/sitemap.xml
Google supports four fields:
| Field | What it does |
|---|---|
user-agent |
Names the crawler the group applies to; * means any crawler without its own group |
disallow |
A path prefix the crawler shouldn’t request |
allow |
A path prefix the crawler may request, overriding a broader disallow |
sitemap |
The full URL of a sitemap; can appear anywhere in the file |
Details that matter:
- Paths are prefixes and case-sensitive.
Disallow: /blogblocks/blog,/blog/,/blog-tipsand/blog.html, but not/Blog/. - Two wildcards:
*matches any sequence of characters,$marks the end of the URL.Disallow: /*.pdf$blocks URLs ending in.pdf. - Size limit: Google reads the first 500 KiB; “content which is after the maximum file size is ignored.”
crawl-delayisn’t supported by Google, though some other crawlers honour it (Anthropic’s crawlers, for example).- Caching: Google “generally caches the contents of robots.txt file for up to 24 hours”, so changes aren’t instant.
How do crawlers decide which rules apply?
Two decisions, in order.
1. Which group? “Only one group is valid for a particular crawler”: the one with the most specific user agent that matches. A crawler with its own group ignores the * group completely. This is the most common source of surprises:
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: Googlebot
Disallow: /old-catalogue/
Here Googlebot may crawl /admin/ and /cart/, because it follows only its own group. Repeat shared rules in every named group.
2. Which rule within the group? The most specific rule, meaning the longest matching path, wins. When an allow and a disallow rule are equally specific, “Google uses the least restrictive rule.”
| Rules | URL | Result |
|---|---|---|
Disallow: /shop/ + Allow: /shop/sale/ |
/shop/sale/shoes |
Allowed (longer rule) |
Disallow: /shop/ + Allow: /shop/sale/ |
/shop/new/ |
Blocked |
Disallow: /*? |
/shoes?colour=red |
Blocked |
Disallow: /blog |
/blog-tips/ |
Blocked (prefix match) |
Disallow: /blog/ |
/Blog/ |
Allowed (case-sensitive) |
What does a good robots.txt look like?
Most sites need very little. Block areas with no search value that crawlers would otherwise waste time on, and point to your sitemap.
User-agent: *
# Account, checkout and internal search: no search value
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
For faceted navigation, Google’s faceted navigation guide suggests disallowing the filter parameters and allowing what should stay crawlable:
User-agent: Googlebot
Disallow: /*?*color=
Disallow: /*?*size=
Allow: /*?products=all$
For AI crawlers, each company uses separate user agents for search and for model training, so you can allow one and block the other. That’s covered in how to block AI crawlers.
What happens if robots.txt can’t be fetched?
This is where robots.txt can hurt a site that never touched it. Google’s handling depends on the status code:
| Response | What Google does |
|---|---|
| 2xx | Reads and applies the file |
| 3xx | Follows at least five redirect hops, then treats it as a 404 |
| 4xx (except 429) | Treats it “as if a valid robots.txt file didn’t exist”: no restrictions |
| 5xx or 429 | “For the first 12 hours, Google stops crawling the site”, then uses the last good copy for up to 30 days |
| Network or DNS error | Treated like a server error |
After 30 days of errors, “if the site is generally available to Google, Google will behave as if there is no robots.txt file”; if not, “Google will stop crawling the site.” A CDN or firewall that returns 5xx or 403 for /robots.txt to bots is therefore a site-wide problem, not a detail. Search Console’s Crawl Stats report shows robots.txt fetch failures under host status.
What are the most common robots.txt mistakes?
| Mistake | What happens | Fix |
|---|---|---|
Disallow: / copied from staging |
Nothing gets crawled | Check robots.txt in every launch checklist |
| Using robots.txt to remove pages from Google | Pages stay indexed, without content | Allow crawling and add noindex |
Blocking a page that has noindex |
Google never sees the noindex | Unblock it until it drops out |
| Blocking CSS, JavaScript or images pages need | Google can’t render the page properly | Allow resources needed for rendering |
| A named bot group without the shared rules | That bot ignores the * rules |
Repeat the rules in each group |
Prefix too short (/blog instead of /blog/) |
Blocks unrelated URLs | End directory rules with / |
| Wrong case | Rule doesn’t match | Match the URL’s exact case |
| Robots.txt on the main domain only | Subdomains are uncontrolled | Add a file to each host |
| robots.txt returns 5xx | Google pauses crawling | Monitor the file’s status |
How do you test robots.txt?
In Search Console, the robots.txt report shows the robots.txt files Google found for the top 20 hosts of a property, when each was last crawled, the fetch status and any parsing issues. After fixing a problem, you can request a recrawl from the report. URL Inspection shows whether a single URL is blocked.
For one URL and any crawler, the free robots.txt tester fetches the live file and shows whether Googlebot, Bingbot and the main AI crawlers may crawl the URL, with the line that decides. You can edit the file in the page and test a change before you deploy it.
Across the whole site, crawl with robots.txt respected and see what it blocks. In Crawlens:
- With Respect robots.txt on (the default), every disallowed URL shows as Blocked by robots.txt, both in the live crawl breakdown and in the URLs table.
- Linked pages blocked by robots.txt flags blocked URLs that your own pages link to: the ones Google may index without content.
- Robots.txt is evaluated for the selected User-Agent. Crawl with the Googlebot Smartphone preset to see exactly what Googlebot’s group allows, or with a custom user agent to test another crawler’s group.
- To see what you’re hiding, run a second crawl with Respect robots.txt off and compare the two. Anything only the second crawl reaches is blocked for search engines; make sure that’s intended.
- In Sitemap mode, Crawlens reads the sitemaps declared in robots.txt when you don’t list any, which also tests that your
Sitemap:lines work.
For how robots.txt fits with sitemaps, canonicals and noindex, see the crawling and indexing guide.
Checklist
- robots.txt at the root of every host and subdomain, returning 200
- No
Disallow: /outside staging - Only areas with no search value blocked: account, cart, checkout, internal search, endless filters
- CSS, JavaScript and images needed for rendering not blocked
- No pages blocked that rely on noindex
- Shared rules repeated in every named user-agent group
- Paths match the URLs’ case and end directories with
/ -
Sitemap:lines point at working sitemaps - robots.txt report in Search Console shows no fetch errors
- A crawl that respects robots.txt reaches every page that should rank
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. Google says robots.txt is not a mechanism for keeping a web page out of Google: a disallowed page can still be indexed if other pages link to it, usually shown without a description. To keep a page out of the index, allow crawling and use noindex.
Where should robots.txt be located?
At the root of each host, for example https://www.example.com/robots.txt. It applies only to the protocol, host and port it's served from, so a subdomain or an http:// version needs its own file.
Does Google support crawl-delay in robots.txt?
No. Google supports only user-agent, allow, disallow and sitemap, and ignores crawl-delay. Some other crawlers, such as Anthropic's, do honour it.
What happens if robots.txt returns a 404?
Google treats a 4xx response, except 429, as if no robots.txt existed, so it may crawl everything. A 5xx response is the dangerous one: Google stops crawling the site for the first 12 hours and then relies on its last good copy.
Should I block CSS and JavaScript files in robots.txt?
No, not if pages need them to render. Google warns that blocking resources it needs to understand a page can stop it from analysing the page properly.