Part of the Crawling & Indexing series. For the full picture, start with Crawling and indexing: how search engines find your pages, and what stops them.
Most indexing rules live in a <meta name="robots"> tag in the page’s head. The X-Robots-Tag puts the same rules in the HTTP response instead. That makes it the only option for files without a head, and a common hidden cause of pages that won’t index: nobody sees it in the page source.
What is the X-Robots-Tag?
Google’s robots meta tag documentation describes it as “an element of the HTTP header response for a given URL.” A response with a noindex rule looks like this:
HTTP/1.1 200 OK
X-Robots-Tag: noindex
Rules can target one crawler: Google’s example is X-Robots-Tag: googlebot: nofollow.
When should you use it?
- Non-HTML files. “To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.”
- Rules for many URLs at once, set in the server or CDN configuration, such as everything under
/internal/or every PDF. - Staging environments, as a server-wide noindex, as long as it’s removed at launch.
For normal HTML pages the meta tag is easier to see and maintain; both carry the same rules.
Which rules does it support?
| Rule | Effect |
|---|---|
noindex |
Keep the URL out of search results |
nofollow |
Don’t follow the links on the page |
none |
Same as noindex, nofollow |
nosnippet |
No text snippet in results |
max-snippet:[number] |
Limit snippet length |
max-image-preview:[setting] |
Limit image preview size |
noimageindex |
Don’t index images on the page |
unavailable_after:[date/time] |
Drop from results after a date |
indexifembedded |
Allow indexing when embedded in an iframe despite noindex |
Snippet rules also apply to Google’s AI features; see how to appear in AI Overviews.
When rules conflict, “the more restrictive rule applies”: with both max-snippet:50 and nosnippet, nosnippet wins. That includes a conflict between the header and the meta tag.
The robots.txt trap
A URL blocked by robots.txt is never fetched, so its header is never read: Google says such URLs’ “indexing or serving rules will not be found and will therefore be ignored.” Blocking a PDF in robots.txt and sending noindex doesn’t remove it; it can stay indexed from links. Allow crawling until it drops out. See noindex vs robots.txt.
How do you find X-Robots-Tag across a site?
Because the header isn’t in the HTML, a page can look indexable in the source and still be noindexed. Crawl the site with Crawlens:
- Noindex pages counts noindex from both the meta tag and the X-Robots-Tag header.
- Each URL’s details show Meta robots and X-Robots-Tag side by side, and the Headers tab shows the full response headers.
- Filter the URLs table by indexability to list every non-indexable URL with its reason.
- With Search Console connected, Pages with search traffic are not indexable catches a header added by mistake to pages that earn clicks.
A sudden site-wide noindex after a deployment is often a server or CDN rule copied from staging. If the header appears only on some URLs, compare their paths and file types with the server configuration.
For how this fits with canonicals, robots.txt and sitemaps, see the crawling and indexing guide.
Checklist
- Non-HTML files that shouldn’t be indexed send
X-Robots-Tag: noindex - No staging-wide X-Robots-Tag left on the live site
- Header and meta tag don’t conflict unintentionally
- URLs relying on the header aren’t blocked in robots.txt
- Pages with search traffic checked for unexpected noindex
Frequently asked questions
What is the X-Robots-Tag?
It's an HTTP response header that tells search engines how to index and serve a URL, using the same rules as the robots meta tag. Because it's sent in the response headers, it also works for files that have no HTML head, such as PDFs and images.
When should I use X-Robots-Tag instead of a meta robots tag?
For non-HTML resources such as PDF, video or image files, which can't carry a meta tag, and when it's easier to set a rule at the server level for many URLs at once, such as a whole folder.
What happens if meta robots and X-Robots-Tag conflict?
Google applies the more restrictive rule. For example, if one says max-snippet:50 and the other nosnippet, nosnippet applies.
Does X-Robots-Tag work on pages blocked by robots.txt?
No. If robots.txt blocks a URL, Google can't fetch it, so the header's rules are never seen. To noindex a URL with the header, it must stay crawlable.