Noindex vs robots.txt: which to use, and the mistakes that hide pages
Robots.txt controls crawling; noindex controls indexing. How each works, why combining them backfires, and how to find pages hidden or exposed by mistake.
Blog / Crawling & Indexing
How search engines discover, crawl and index your pages, and what quietly stops them: robots.txt rules, noindex and canonicals, XML sitemaps, internal links and orphan pages, and the pages Google crawls but won’t index. 15 articles. Page 2 of 2.
Robots.txt controls crawling; noindex controls indexing. How each works, why combining them backfires, and how to find pages hidden or exposed by mistake.
Orphan pages have no internal links pointing to them, so users and crawlers struggle to find them. How to find them with sitemaps, Search Console and logs.
How robots.txt works, how crawlers pick which rules apply, copy-ready examples, what happens when the file fails to load, and how to test it across a site.
Why Google reports pages that return 200 as soft 404s, the usual causes (empty pages, error messages, homepage redirects) and how to find and fix them.
Sitemaps full of redirects, errors and noindex pages lose Google's trust. How to audit XML sitemaps against a crawl and keep only indexable URLs.
Crawl, audit 70 checks, and line them up with Search Console and Core Web Vitals data.
Download free for Windows