New researchTechnical SEO & AI Search · 2027Explore the framework

X-Robots-Tag: how to use the HTTP header for noindex and more

What the X-Robots-Tag header does, when to use it instead of a meta tag (PDFs, images, whole folders), the rules it supports, and how to find it on a site.

Part of the Crawling & Indexing series. For the full picture, start with Crawling and indexing: how search engines find your pages, and what stops them.

Most indexing rules live in a <meta name="robots"> tag in the page’s head. The X-Robots-Tag puts the same rules in the HTTP response instead. That makes it the only option for files without a head, and a common hidden cause of pages that won’t index: nobody sees it in the page source.

What is the X-Robots-Tag?

Google’s robots meta tag documentation describes it as “an element of the HTTP header response for a given URL.” A response with a noindex rule looks like this:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Rules can target one crawler: Google’s example is X-Robots-Tag: googlebot: nofollow.

When should you use it?

  • Non-HTML files. “To block indexing of non-HTML resources, such as PDF files, video files, or image files, use the X-Robots-Tag response header instead.”
  • Rules for many URLs at once, set in the server or CDN configuration, such as everything under /internal/ or every PDF.
  • Staging environments, as a server-wide noindex, as long as it’s removed at launch.

For normal HTML pages the meta tag is easier to see and maintain; both carry the same rules.

Which rules does it support?

Rule Effect
noindex Keep the URL out of search results
nofollow Don’t follow the links on the page
none Same as noindex, nofollow
nosnippet No text snippet in results
max-snippet:[number] Limit snippet length
max-image-preview:[setting] Limit image preview size
noimageindex Don’t index images on the page
unavailable_after:[date/time] Drop from results after a date
indexifembedded Allow indexing when embedded in an iframe despite noindex

Snippet rules also apply to Google’s AI features; see how to appear in AI Overviews.

When rules conflict, “the more restrictive rule applies”: with both max-snippet:50 and nosnippet, nosnippet wins. That includes a conflict between the header and the meta tag.

The robots.txt trap

A URL blocked by robots.txt is never fetched, so its header is never read: Google says such URLs’ “indexing or serving rules will not be found and will therefore be ignored.” Blocking a PDF in robots.txt and sending noindex doesn’t remove it; it can stay indexed from links. Allow crawling until it drops out. See noindex vs robots.txt.

How do you find X-Robots-Tag across a site?

Because the header isn’t in the HTML, a page can look indexable in the source and still be noindexed. Crawl the site with Crawlens:

  • Noindex pages counts noindex from both the meta tag and the X-Robots-Tag header.
  • Each URL’s details show Meta robots and X-Robots-Tag side by side, and the Headers tab shows the full response headers.
  • Filter the URLs table by indexability to list every non-indexable URL with its reason.
  • With Search Console connected, Pages with search traffic are not indexable catches a header added by mistake to pages that earn clicks.

A sudden site-wide noindex after a deployment is often a server or CDN rule copied from staging. If the header appears only on some URLs, compare their paths and file types with the server configuration.

For how this fits with canonicals, robots.txt and sitemaps, see the crawling and indexing guide.

Checklist

  • Non-HTML files that shouldn’t be indexed send X-Robots-Tag: noindex
  • No staging-wide X-Robots-Tag left on the live site
  • Header and meta tag don’t conflict unintentionally
  • URLs relying on the header aren’t blocked in robots.txt
  • Pages with search traffic checked for unexpected noindex

Frequently asked questions

What is the X-Robots-Tag?

It's an HTTP response header that tells search engines how to index and serve a URL, using the same rules as the robots meta tag. Because it's sent in the response headers, it also works for files that have no HTML head, such as PDFs and images.

When should I use X-Robots-Tag instead of a meta robots tag?

For non-HTML resources such as PDF, video or image files, which can't carry a meta tag, and when it's easier to set a rule at the server level for many URLs at once, such as a whole folder.

What happens if meta robots and X-Robots-Tag conflict?

Google applies the more restrictive rule. For example, if one says max-snippet:50 and the other nosnippet, nosnippet applies.

Does X-Robots-Tag work on pages blocked by robots.txt?

No. If robots.txt blocks a URL, Google can't fetch it, so the header's rules are never seen. To noindex a URL with the header, it must stay crawlable.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows