New researchTechnical SEO & AI Search · 2027Explore the framework

Robots.txt for SEO: syntax, examples and the mistakes that block sites

How robots.txt works, how crawlers pick which rules apply, copy-ready examples, what happens when the file fails to load, and how to test it across a site.

Part of the Crawling & Indexing series. For the full picture, start with Crawling and indexing: how search engines find your pages, and what stops them.

Robots.txt is the smallest file on most sites and one of the few that can take a whole site out of search in one line. It’s also widely misunderstood: people use it to “hide” pages it can’t hide, and miss the rules that are actually blocking them.

This guide covers what robots.txt does and doesn’t do, the syntax, how crawlers decide which rules apply, examples you can adapt, what happens when the file breaks, and how to test it against a real crawl.

What is robots.txt?

Robots.txt is a plain-text file at the root of a host that tells crawlers which URLs they may request. In Google’s words, it “tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests” (introduction to robots.txt).

What it is not:

  • Not an indexing control. Google is explicit that “it is not a mechanism for keeping a web page out of Google.” A disallowed URL can still be indexed if other pages link to it, typically with no description. Use noindex for that; see noindex vs robots.txt.
  • Not security. Reputable crawlers obey it; others don’t. Anything private needs authentication.

Where does robots.txt go?

At the root of each host: https://www.example.com/robots.txt. Google’s robots.txt specification says its rules “apply only to the host, protocol, and port number where the robots.txt file is hosted.”

File Applies to Doesn’t apply to
https://example.com/robots.txt https://example.com/… https://shop.example.com/…, http://example.com/…, https://example.com:8443/…
https://shop.example.com/robots.txt https://shop.example.com/… Every other host

Subdomains are the usual surprise: a blog on blog.example.com or a shop on a platform subdomain needs its own file. Moving that blog into a folder on the main domain is a migration of its own: see subdomain to subfolder migration.

What is the robots.txt syntax?

A file is a set of groups. Each group starts with one or more user-agent lines, followed by the rules for those crawlers.

User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

Sitemap: https://www.example.com/sitemap.xml

Google supports four fields:

Field What it does
user-agent Names the crawler the group applies to; * means any crawler without its own group
disallow A path prefix the crawler shouldn’t request
allow A path prefix the crawler may request, overriding a broader disallow
sitemap The full URL of a sitemap; can appear anywhere in the file

Details that matter:

  • Paths are prefixes and case-sensitive. Disallow: /blog blocks /blog, /blog/, /blog-tips and /blog.html, but not /Blog/.
  • Two wildcards: * matches any sequence of characters, $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf.
  • Size limit: Google reads the first 500 KiB; “content which is after the maximum file size is ignored.”
  • crawl-delay isn’t supported by Google, though some other crawlers honour it (Anthropic’s crawlers, for example).
  • Caching: Google “generally caches the contents of robots.txt file for up to 24 hours”, so changes aren’t instant.

How do crawlers decide which rules apply?

Two decisions, in order.

1. Which group? “Only one group is valid for a particular crawler”: the one with the most specific user agent that matches. A crawler with its own group ignores the * group completely. This is the most common source of surprises:

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: Googlebot
Disallow: /old-catalogue/

Here Googlebot may crawl /admin/ and /cart/, because it follows only its own group. Repeat shared rules in every named group.

2. Which rule within the group? The most specific rule, meaning the longest matching path, wins. When an allow and a disallow rule are equally specific, “Google uses the least restrictive rule.”

Rules URL Result
Disallow: /shop/ + Allow: /shop/sale/ /shop/sale/shoes Allowed (longer rule)
Disallow: /shop/ + Allow: /shop/sale/ /shop/new/ Blocked
Disallow: /*? /shoes?colour=red Blocked
Disallow: /blog /blog-tips/ Blocked (prefix match)
Disallow: /blog/ /Blog/ Allowed (case-sensitive)

What does a good robots.txt look like?

Most sites need very little. Block areas with no search value that crawlers would otherwise waste time on, and point to your sitemap.

User-agent: *
# Account, checkout and internal search: no search value
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

For faceted navigation, Google’s faceted navigation guide suggests disallowing the filter parameters and allowing what should stay crawlable:

User-agent: Googlebot
Disallow: /*?*color=
Disallow: /*?*size=
Allow: /*?products=all$

For AI crawlers, each company uses separate user agents for search and for model training, so you can allow one and block the other. That’s covered in how to block AI crawlers.

What happens if robots.txt can’t be fetched?

This is where robots.txt can hurt a site that never touched it. Google’s handling depends on the status code:

Response What Google does
2xx Reads and applies the file
3xx Follows at least five redirect hops, then treats it as a 404
4xx (except 429) Treats it “as if a valid robots.txt file didn’t exist”: no restrictions
5xx or 429 “For the first 12 hours, Google stops crawling the site”, then uses the last good copy for up to 30 days
Network or DNS error Treated like a server error

After 30 days of errors, “if the site is generally available to Google, Google will behave as if there is no robots.txt file”; if not, “Google will stop crawling the site.” A CDN or firewall that returns 5xx or 403 for /robots.txt to bots is therefore a site-wide problem, not a detail. Search Console’s Crawl Stats report shows robots.txt fetch failures under host status.

What are the most common robots.txt mistakes?

Mistake What happens Fix
Disallow: / copied from staging Nothing gets crawled Check robots.txt in every launch checklist
Using robots.txt to remove pages from Google Pages stay indexed, without content Allow crawling and add noindex
Blocking a page that has noindex Google never sees the noindex Unblock it until it drops out
Blocking CSS, JavaScript or images pages need Google can’t render the page properly Allow resources needed for rendering
A named bot group without the shared rules That bot ignores the * rules Repeat the rules in each group
Prefix too short (/blog instead of /blog/) Blocks unrelated URLs End directory rules with /
Wrong case Rule doesn’t match Match the URL’s exact case
Robots.txt on the main domain only Subdomains are uncontrolled Add a file to each host
robots.txt returns 5xx Google pauses crawling Monitor the file’s status

How do you test robots.txt?

In Search Console, the robots.txt report shows the robots.txt files Google found for the top 20 hosts of a property, when each was last crawled, the fetch status and any parsing issues. After fixing a problem, you can request a recrawl from the report. URL Inspection shows whether a single URL is blocked.

For one URL and any crawler, the free robots.txt tester fetches the live file and shows whether Googlebot, Bingbot and the main AI crawlers may crawl the URL, with the line that decides. You can edit the file in the page and test a change before you deploy it.

Across the whole site, crawl with robots.txt respected and see what it blocks. In Crawlens:

  • With Respect robots.txt on (the default), every disallowed URL shows as Blocked by robots.txt, both in the live crawl breakdown and in the URLs table.
  • Linked pages blocked by robots.txt flags blocked URLs that your own pages link to: the ones Google may index without content.
  • Robots.txt is evaluated for the selected User-Agent. Crawl with the Googlebot Smartphone preset to see exactly what Googlebot’s group allows, or with a custom user agent to test another crawler’s group.
  • To see what you’re hiding, run a second crawl with Respect robots.txt off and compare the two. Anything only the second crawl reaches is blocked for search engines; make sure that’s intended.
  • In Sitemap mode, Crawlens reads the sitemaps declared in robots.txt when you don’t list any, which also tests that your Sitemap: lines work.

For how robots.txt fits with sitemaps, canonicals and noindex, see the crawling and indexing guide.

Checklist

  • robots.txt at the root of every host and subdomain, returning 200
  • No Disallow: / outside staging
  • Only areas with no search value blocked: account, cart, checkout, internal search, endless filters
  • CSS, JavaScript and images needed for rendering not blocked
  • No pages blocked that rely on noindex
  • Shared rules repeated in every named user-agent group
  • Paths match the URLs’ case and end directories with /
  • Sitemap: lines point at working sitemaps
  • robots.txt report in Search Console shows no fetch errors
  • A crawl that respects robots.txt reaches every page that should rank

Frequently asked questions

Does robots.txt stop a page from being indexed?

No. Google says robots.txt is not a mechanism for keeping a web page out of Google: a disallowed page can still be indexed if other pages link to it, usually shown without a description. To keep a page out of the index, allow crawling and use noindex.

Where should robots.txt be located?

At the root of each host, for example https://www.example.com/robots.txt. It applies only to the protocol, host and port it's served from, so a subdomain or an http:// version needs its own file.

Does Google support crawl-delay in robots.txt?

No. Google supports only user-agent, allow, disallow and sitemap, and ignores crawl-delay. Some other crawlers, such as Anthropic's, do honour it.

What happens if robots.txt returns a 404?

Google treats a 4xx response, except 429, as if no robots.txt existed, so it may crawl everything. A 5xx response is the dangerous one: Google stops crawling the site for the first 12 hours and then relies on its last good copy.

Should I block CSS and JavaScript files in robots.txt?

No, not if pages need them to render. Google warns that blocking resources it needs to understand a page can stop it from analysing the page properly.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows