New researchTechnical SEO & AI Search · 2027Explore the framework

Crawling and indexing: how search engines find your pages, and what stops them

How crawling and indexing work, what blocks each step, how to read Search Console's Page indexing report, and how to audit crawlability across a whole site.

Every page that ranks went through the same pipeline: a search engine discovered its URL, fetched it, rendered it, decided it was worth keeping and stored it in its index. A failure at any step stops the page from appearing, and most failures make no noise. The page is simply never there.

This guide explains each step, what blocks it, how to read the reports Google gives you, and how to audit crawling and indexing across a whole site. Each section links to a detailed guide.

What is the difference between crawling and indexing?

Crawling is a search engine downloading your page; indexing is the search engine analysing that page and storing it so it can appear in results. Google describes crawling as when it “downloads text, images, and videos from pages it found on the internet with automated programs called crawlers”, followed by indexing, where it analyses the content and works out which URL is the canonical one (how Google Search works).

The two can come apart:

  • Crawled, not indexed: Google fetched the page and chose not to keep it. “Indexing isn’t guaranteed; not every page that Google processes will be indexed.”
  • Indexed, not crawled: a page blocked by robots.txt can still be indexed from links pointing to it, without its content.

How does a page get from your server to the index?

Stage What happens What blocks it
Discovery The search engine learns the URL exists, from links, sitemaps or submissions No internal links, missing from sitemaps, links only added by JavaScript
Crawling The crawler requests the URL robots.txt disallow, server errors, timeouts, bot protection
Rendering The page is run in a browser so scripts can add content Scripts that fail, time out or need user interaction
Indexing The content is analysed and stored, and a canonical URL chosen Non-200 status, noindex, canonical pointing elsewhere, duplicate or low-value content
Serving Indexed pages are ranked for queries Everything else in SEO

Google’s own technical requirements boil the first four stages down to three conditions: “Googlebot isn’t blocked”, “the page works, meaning that Google receives an HTTP 200 (success) status code”, and “the page has indexable content.”

How do search engines discover pages?

Mostly through links. A crawler that starts at your homepage and follows links reaches every page that’s linked, and nothing that isn’t. That makes internal linking the main lever you control:

  • Orphan pages, with no internal links, depend entirely on sitemaps or external links. See orphan pages.
  • Deep pages, many clicks from the homepage, get crawled less often. See the internal linking audit.
  • Links only added by JavaScript are found later by Google and not at all by crawlers that don’t render. See the JavaScript SEO guide for rendering and discovery, and the raw-versus-rendered HTML audit for the comparison.

For single-page apps, discovery also depends on real links and URLs that work on a fresh load. The SPA navigation SEO guide covers client-side routing, History API changes and deep-link checks.

Content revealed only after a user scrolls or clicks can be missed too. The lazy loading SEO guide explains how to keep images, deeper products and infinite-scroll pages discoverable.

XML sitemaps are the second channel. Google says they help most on large sites, new sites with few external links, and sites with lots of media, and that a sitemap “doesn’t guarantee that all the items in your sitemap will be crawled and indexed” (sitemaps overview). A sitemap full of redirects, errors and noindexed URLs is worse than a short accurate one. See the XML sitemap audit.

When the inventory exceeds one file, the sitemap index SEO guide shows how to split it into valid child sitemaps and monitor each group.

Within each file, the sitemap lastmod guide explains when a page’s modification date should change and how to catch generators that stamp every URL with the build date.

What stops a page from being crawled?

Cause How it shows up Crawlens check
robots.txt disallows the URL Blocked by robots.txt Linked pages blocked by robots.txt
Server errors 5xx responses; Google slows down crawling Internal pages return 5xx
Timeouts, DNS or SSL failures No response Pages did not respond
Firewall or bot protection 403 or challenge pages for crawlers only Pages PageSpeed Insights could not test often hints at it
Not linked anywhere Never discovered by following links Orphan pages
Too deep Crawled rarely Pages more than 4 clicks from the start
Links only after rendering Discovered late or not at all Links only in rendered HTML
Slow responses Fewer pages crawled per visit Slow server response (over 1 second)

What stops a page from being indexed?

Cause What to check Crawlens check
noindex in a meta tag or X-Robots-Tag header Intended, or left on from staging? Noindex pages
Canonical points to another URL Is the other URL the right one, and indexable? Canonicalised pages, Canonical points to a non-200 URL, to a noindex page
Status code isn’t 200 4xx, 5xx or redirect Internal pages return 4xx
Duplicate of another page Google picks one URL and drops the others Duplicate content
Too little content Little reason to index it Thin content (under 200 words)
Content only after JavaScript Indexing is slower, and fails if rendering does Content depends on JavaScript

The detailed guides: noindex vs robots.txt, X-Robots-Tag, HTTP status codes, canonical tag errors, duplicate content, thin content and soft 404 errors.

Which controls affect crawling, and which affect indexing?

Mixing these up causes more indexing problems than anything else.

Control Affects crawling Affects indexing Use it to
robots.txt Disallow Yes No (blocked URLs can still be indexed from links) Keep crawlers out of areas with no search value: carts, internal search, endless filters
noindex (meta or header) No Yes Keep a page out of results. The page must stay crawlable so the directive is seen
rel="canonical" No Hint Say which of several similar URLs should be indexed
301 / 308 redirect Moves the crawler Moves signals to the target Retire or move a URL
404 / 410 Crawled less over time Removed from the index Remove a page with no replacement
nofollow on internal links Withholds the link signal No Rarely needed internally
XML sitemap Helps discovery No List the canonical, indexable URLs you want found

The classic mistake is blocking a page in robots.txt to remove it from Google: the crawler can no longer see the noindex, so the URL may stay indexed. Noindex vs robots.txt covers this in detail, and robots.txt for SEO covers the syntax and how crawlers pick which rules apply. To check one URL against the live file for Googlebot and the main AI crawlers, use the free robots.txt tester.

The Google Indexing API guide explains why API notifications are limited to qualifying job and livestream pages, not a substitute for links and sitemaps on ordinary sites.

When query parameters create multiple versions of a page, choose the control by what each version changes. The URL parameters SEO guide walks through tracking, sorting, pagination and filters without blocking useful pages by accident.

Does crawl budget matter for your site?

Probably not, unless the site is large. Google’s crawl budget guide is written for “large sites (1 million+ unique pages) with content that changes moderately often (once a week)”, “medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”, and sites with many URLs reported as Discovered - currently not indexed. For everyone else, Google says, the guide isn’t needed.

When it does matter, the advice is to stop wasting crawls: block URLs that shouldn’t be crawled in robots.txt, “return a 404 or 410 status code for permanently removed pages”, fix soft 404s, “keep your sitemaps up to date”, “avoid long redirect chains”, and make pages fast to load.

The only way to see how crawl budget is actually spent is server logs: which URLs Googlebot requests, how often, and what it gets back. See log file analysis for SEO and crawl budget for the details; the enterprise technical SEO framework connects those findings to large URL inventories and release workflows.

For a quicker view of request trends, host availability, response codes and Googlebot types, read the Search Console Crawl Stats report. It shows patterns and example URLs; logs are still needed for a complete path-level request history.

How do you read Search Console’s Page indexing report?

The Page indexing report explains why Google hasn’t indexed each URL it knows about. The statuses you’ll see most:

Status What Google says What to do
Discovered - currently not indexed “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site” Check server speed and errors; strengthen internal links to the pages
Crawled - currently not indexed “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.” Improve, merge or remove thin and duplicate pages
Duplicate without user-selected canonical Google chose another page as canonical “and so will not serve this page in Search” Add canonicals or consolidate the duplicates
Alternate page with proper canonical tag “This page correctly points to the canonical page, which is indexed, so there is nothing you need to do.” Nothing, if the canonical is intended
URL marked ‘noindex’ Google “encountered a ‘noindex’ directive and therefore did not index it” Remove the noindex if the page should rank
URL blocked by robots.txt “This page was blocked by your site’s robots.txt file.” Allow it if the page should rank
Indexed, though blocked by robots.txt Indexed despite the block, because “someone else links to your page” Allow crawling and add noindex, or allow it to be indexed properly

The report tells you what Google decided, sampled and delayed. A crawl tells you the current state of every URL. Use both. For the two most common statuses, see crawled and discovered but not indexed.

How do you audit crawling and indexing across a whole site?

  1. Crawl the site from the homepage with robots.txt respected. Everything the crawl can’t reach by links, search engines can’t either. In Crawlens, the URLs table can be filtered by indexability, so you can see every non-indexable URL with its reason (noindex, canonicalised, redirect, error, blocked).
  2. Crawl the sitemaps too. Sitemap mode with Also follow links turned on compares the two: URLs only in the sitemap are orphans, and sitemap URLs that aren’t indexable show up as Non-indexable URLs in sitemaps.
  3. Render JavaScript on client-rendered sites, to see links and content that only exist after scripts run.
  4. Join Search Console. Three checks show where the crawl and Google disagree: URLs with impressions the crawl did not find, Pages with search traffic are not indexable and Indexable pages with no search impressions.
  5. Import server logs to see what Googlebot actually does: Indexable pages Googlebot did not crawl, URLs Googlebot crawls that the crawl did not find and Googlebot gets errors.
  6. Fix, re-crawl and compare. Then confirm in Search Console over the following weeks: Google says crawling “can take anywhere from a few days to a few weeks” (ask Google to recrawl).

AI search engines run their own crawlers with their own rules, and most don’t render JavaScript. The AI search readiness checklist covers what changes for them. For where crawling and indexing fit in a full audit, see the technical SEO audit guide.

Crawling and indexing checklist

Discovery

  • Every important page linked from at least one other page, ideally several
  • Important pages within four clicks of the homepage
  • Navigation and key links present in the raw HTML
  • Sitemaps list only canonical, indexable 200 URLs, and are submitted in Search Console

Crawling

  • robots.txt blocks only areas with no search value
  • No 5xx errors, timeouts or bot-protection blocks for search engine crawlers
  • Server responds in under a second
  • No redirect chains; removed pages return 404 or 410

Indexing

  • No unintended noindex in meta tags or headers
  • Canonicals point at indexable 200 URLs
  • Duplicates consolidated; thin pages improved, merged or removed
  • Main content present without JavaScript, or rendering verified

Verification

  • Page indexing report reviewed, status by status
  • Crawl joined with Search Console data
  • Server logs checked for Googlebot errors and uncrawled pages (large sites)
  • Re-crawl and compare after fixes

All Crawling & Indexing guides

Frequently asked questions

What is the difference between crawling and indexing?

Crawling is when a search engine's bot downloads a page. Indexing is when the search engine analyses that page and stores it so it can be shown in results. A page can be crawled and not indexed, and, if it's blocked by robots.txt but linked from elsewhere, indexed without being crawled.

How long does it take Google to index a new page?

Google says crawling can take anywhere from a few days to a few weeks. Linking to the page from strong internal pages, listing it in your sitemap and requesting indexing in Search Console's URL Inspection tool can help, but none of them guarantees indexing.

Why is my page crawled but not indexed?

"Crawled - currently not indexed" means Google fetched the page and chose not to index it for now. Common reasons are duplicate or near-duplicate content, thin pages, or pages Google doesn't consider valuable enough. Improve or consolidate the page; Google says there's no need to resubmit it.

Does a sitemap guarantee my pages get indexed?

No. Google says a sitemap doesn't guarantee that all the items in it will be crawled and indexed. It helps discovery, especially on large or new sites, but pages still need to be indexable and worth indexing.

Can robots.txt keep a page out of Google?

Not reliably. Robots.txt stops crawling, not indexing: a blocked page can still be indexed from links pointing to it, without its content. To keep a page out of the index, let it be crawled and add a noindex directive.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows