X-Robots-Tag: how to use the HTTP header for noindex and more
What the X-Robots-Tag header does, when to use it instead of a meta tag (PDFs, images, whole folders), the rules it supports, and how to find it on a site.
Blog
Audit walkthroughs, how-tos and what’s new in Crawlens. Written for people who fix sites, not just report on them. Page 7 of 13.
What the X-Robots-Tag header does, when to use it instead of a meta tag (PDFs, images, whole folders), the rules it supports, and how to find it on a site.
How AI agents read and use web pages, and what web.dev recommends so they can complete tasks: semantic HTML, labels, visible controls and stable layouts.
A deep technical audit for Google, ChatGPT, Claude and Perplexity: crawler access, indexability, rendering, snippet controls, structure and measurement.
Where AI search traffic shows up in Search Console, GA4 and server logs, which visits each one misses, and how to build a simple view of AI visibility.
What Bing recommends for its search and AI-powered answers: accurate sitemaps and lastmod, IndexNow for fast updates, and controls for AI answers.
Every major AI crawler and its purpose, robots.txt rules to opt out of AI training but stay in AI search answers, and what robots.txt can't stop.
Canonicals pointing to redirects, errors or noindex pages, multiple canonicals and missing ones. How to audit rel=canonical across a site and fix what's broken.
A step-by-step domain change: both Search Console properties, a page-to-page redirect map, the Change of Address tool, what to update and what to expect.
What OpenAI says about getting into ChatGPT search, the crawler that matters, how to opt out, how to track ChatGPT traffic and what to check on your site.
Crawl, audit 70 checks, and line them up with Search Console and Core Web Vitals data.
Download free for Windows