Crawling and indexing: how search engines find your pages, and what stops them
How crawling and indexing work, what blocks each step, how to read Search Console's Page indexing report, and how to audit crawlability across a whole site.
Blog / Crawling & Indexing
How search engines discover, crawl and index your pages, and what quietly stops them: robots.txt rules, noindex and canonicals, XML sitemaps, internal links and orphan pages, and the pages Google crawls but won’t index. 15 articles.
How crawling and indexing work, what blocks each step, how to read Search Console's Page indexing report, and how to audit crawlability across a whole site.
The Google Indexing API is limited to qualifying job posting and livestream pages. Learn its real use cases, notification types and alternatives.
How Google treats each HTTP status code: which can be indexed, which pass canonical signals, which slow crawling, and how to audit them across a site.
Read host status, response codes, crawl purpose and Googlebot type in Search Console Crawl Stats, then know when URL inspection or logs are needed.
Learn which page changes deserve a new XML sitemap lastmod date, how to catch fake timestamps at scale, and what Google actually does with the tag.
Audit tracking, sorting, session and filter parameters without blocking valuable pages. Learn when to use canonical tags, redirects, robots.txt or a clean URL.
What the X-Robots-Tag header does, when to use it instead of a meta tag (PDFs, images, whole folders), the rules it supports, and how to find it on a site.
Canonicals pointing to redirects, errors or noindex pages, multiple canonicals and missing ones. How to audit rel=canonical across a site and fix what's broken.
What Crawled and Discovered - currently not indexed mean in Search Console, when they're a real problem, how to find the cause, and how to fix it at scale.
How to audit internal links across a site: pages buried too deep, pages with one link, dead ends, internal nofollow and broken links.
Crawl, audit 70 checks, and line them up with Search Console and Core Web Vitals data.
Download free for Windows