Every page that ranks went through the same pipeline: a search engine discovered its URL, fetched it, rendered it, decided it was worth keeping and stored it in its index. A failure at any step stops the page from appearing, and most failures make no noise. The page is simply never there.
This guide explains each step, what blocks it, how to read the reports Google gives you, and how to audit crawling and indexing across a whole site. Each section links to a detailed guide.
What is the difference between crawling and indexing?
Crawling is a search engine downloading your page; indexing is the search engine analysing that page and storing it so it can appear in results. Google describes crawling as when it “downloads text, images, and videos from pages it found on the internet with automated programs called crawlers”, followed by indexing, where it analyses the content and works out which URL is the canonical one (how Google Search works).
The two can come apart:
- Crawled, not indexed: Google fetched the page and chose not to keep it. “Indexing isn’t guaranteed; not every page that Google processes will be indexed.”
- Indexed, not crawled: a page blocked by robots.txt can still be indexed from links pointing to it, without its content.
How does a page get from your server to the index?
| Stage | What happens | What blocks it |
|---|---|---|
| Discovery | The search engine learns the URL exists, from links, sitemaps or submissions | No internal links, missing from sitemaps, links only added by JavaScript |
| Crawling | The crawler requests the URL | robots.txt disallow, server errors, timeouts, bot protection |
| Rendering | The page is run in a browser so scripts can add content | Scripts that fail, time out or need user interaction |
| Indexing | The content is analysed and stored, and a canonical URL chosen | Non-200 status, noindex, canonical pointing elsewhere, duplicate or low-value content |
| Serving | Indexed pages are ranked for queries | Everything else in SEO |
Google’s own technical requirements boil the first four stages down to three conditions: “Googlebot isn’t blocked”, “the page works, meaning that Google receives an HTTP 200 (success) status code”, and “the page has indexable content.”
How do search engines discover pages?
Mostly through links. A crawler that starts at your homepage and follows links reaches every page that’s linked, and nothing that isn’t. That makes internal linking the main lever you control:
- Orphan pages, with no internal links, depend entirely on sitemaps or external links. See orphan pages.
- Deep pages, many clicks from the homepage, get crawled less often. See the internal linking audit.
- Links only added by JavaScript are found later by Google and not at all by crawlers that don’t render. See the JavaScript SEO guide for rendering and discovery, and the raw-versus-rendered HTML audit for the comparison.
For single-page apps, discovery also depends on real links and URLs that work on a fresh load. The SPA navigation SEO guide covers client-side routing, History API changes and deep-link checks.
Content revealed only after a user scrolls or clicks can be missed too. The lazy loading SEO guide explains how to keep images, deeper products and infinite-scroll pages discoverable.
XML sitemaps are the second channel. Google says they help most on large sites, new sites with few external links, and sites with lots of media, and that a sitemap “doesn’t guarantee that all the items in your sitemap will be crawled and indexed” (sitemaps overview). A sitemap full of redirects, errors and noindexed URLs is worse than a short accurate one. See the XML sitemap audit.
When the inventory exceeds one file, the sitemap index SEO guide shows how to split it into valid child sitemaps and monitor each group.
Within each file, the sitemap lastmod guide explains when a page’s modification date should change and how to catch generators that stamp every URL with the build date.
What stops a page from being crawled?
| Cause | How it shows up | Crawlens check |
|---|---|---|
| robots.txt disallows the URL | Blocked by robots.txt | Linked pages blocked by robots.txt |
| Server errors | 5xx responses; Google slows down crawling | Internal pages return 5xx |
| Timeouts, DNS or SSL failures | No response | Pages did not respond |
| Firewall or bot protection | 403 or challenge pages for crawlers only | Pages PageSpeed Insights could not test often hints at it |
| Not linked anywhere | Never discovered by following links | Orphan pages |
| Too deep | Crawled rarely | Pages more than 4 clicks from the start |
| Links only after rendering | Discovered late or not at all | Links only in rendered HTML |
| Slow responses | Fewer pages crawled per visit | Slow server response (over 1 second) |
What stops a page from being indexed?
| Cause | What to check | Crawlens check |
|---|---|---|
noindex in a meta tag or X-Robots-Tag header |
Intended, or left on from staging? | Noindex pages |
| Canonical points to another URL | Is the other URL the right one, and indexable? | Canonicalised pages, Canonical points to a non-200 URL, to a noindex page |
| Status code isn’t 200 | 4xx, 5xx or redirect | Internal pages return 4xx |
| Duplicate of another page | Google picks one URL and drops the others | Duplicate content |
| Too little content | Little reason to index it | Thin content (under 200 words) |
| Content only after JavaScript | Indexing is slower, and fails if rendering does | Content depends on JavaScript |
The detailed guides: noindex vs robots.txt, X-Robots-Tag, HTTP status codes, canonical tag errors, duplicate content, thin content and soft 404 errors.
Which controls affect crawling, and which affect indexing?
Mixing these up causes more indexing problems than anything else.
| Control | Affects crawling | Affects indexing | Use it to |
|---|---|---|---|
robots.txt Disallow |
Yes | No (blocked URLs can still be indexed from links) | Keep crawlers out of areas with no search value: carts, internal search, endless filters |
noindex (meta or header) |
No | Yes | Keep a page out of results. The page must stay crawlable so the directive is seen |
rel="canonical" |
No | Hint | Say which of several similar URLs should be indexed |
| 301 / 308 redirect | Moves the crawler | Moves signals to the target | Retire or move a URL |
| 404 / 410 | Crawled less over time | Removed from the index | Remove a page with no replacement |
nofollow on internal links |
Withholds the link signal | No | Rarely needed internally |
| XML sitemap | Helps discovery | No | List the canonical, indexable URLs you want found |
The classic mistake is blocking a page in robots.txt to remove it from Google: the crawler can no longer see the noindex, so the URL may stay indexed. Noindex vs robots.txt covers this in detail, and robots.txt for SEO covers the syntax and how crawlers pick which rules apply. To check one URL against the live file for Googlebot and the main AI crawlers, use the free robots.txt tester.
The Google Indexing API guide explains why API notifications are limited to qualifying job and livestream pages, not a substitute for links and sitemaps on ordinary sites.
When query parameters create multiple versions of a page, choose the control by what each version changes. The URL parameters SEO guide walks through tracking, sorting, pagination and filters without blocking useful pages by accident.
Does crawl budget matter for your site?
Probably not, unless the site is large. Google’s crawl budget guide is written for “large sites (1 million+ unique pages) with content that changes moderately often (once a week)”, “medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”, and sites with many URLs reported as Discovered - currently not indexed. For everyone else, Google says, the guide isn’t needed.
When it does matter, the advice is to stop wasting crawls: block URLs that shouldn’t be crawled in robots.txt, “return a 404 or 410 status code for permanently removed pages”, fix soft 404s, “keep your sitemaps up to date”, “avoid long redirect chains”, and make pages fast to load.
The only way to see how crawl budget is actually spent is server logs: which URLs Googlebot requests, how often, and what it gets back. See log file analysis for SEO and crawl budget for the details; the enterprise technical SEO framework connects those findings to large URL inventories and release workflows.
For a quicker view of request trends, host availability, response codes and Googlebot types, read the Search Console Crawl Stats report. It shows patterns and example URLs; logs are still needed for a complete path-level request history.
How do you read Search Console’s Page indexing report?
The Page indexing report explains why Google hasn’t indexed each URL it knows about. The statuses you’ll see most:
| Status | What Google says | What to do |
|---|---|---|
| Discovered - currently not indexed | “The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site” | Check server speed and errors; strengthen internal links to the pages |
| Crawled - currently not indexed | “The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.” | Improve, merge or remove thin and duplicate pages |
| Duplicate without user-selected canonical | Google chose another page as canonical “and so will not serve this page in Search” | Add canonicals or consolidate the duplicates |
| Alternate page with proper canonical tag | “This page correctly points to the canonical page, which is indexed, so there is nothing you need to do.” | Nothing, if the canonical is intended |
| URL marked ‘noindex’ | Google “encountered a ‘noindex’ directive and therefore did not index it” | Remove the noindex if the page should rank |
| URL blocked by robots.txt | “This page was blocked by your site’s robots.txt file.” | Allow it if the page should rank |
| Indexed, though blocked by robots.txt | Indexed despite the block, because “someone else links to your page” | Allow crawling and add noindex, or allow it to be indexed properly |
The report tells you what Google decided, sampled and delayed. A crawl tells you the current state of every URL. Use both. For the two most common statuses, see crawled and discovered but not indexed.
How do you audit crawling and indexing across a whole site?
- Crawl the site from the homepage with robots.txt respected. Everything the crawl can’t reach by links, search engines can’t either. In Crawlens, the URLs table can be filtered by indexability, so you can see every non-indexable URL with its reason (noindex, canonicalised, redirect, error, blocked).
- Crawl the sitemaps too. Sitemap mode with Also follow links turned on compares the two: URLs only in the sitemap are orphans, and sitemap URLs that aren’t indexable show up as Non-indexable URLs in sitemaps.
- Render JavaScript on client-rendered sites, to see links and content that only exist after scripts run.
- Join Search Console. Three checks show where the crawl and Google disagree: URLs with impressions the crawl did not find, Pages with search traffic are not indexable and Indexable pages with no search impressions.
- Import server logs to see what Googlebot actually does: Indexable pages Googlebot did not crawl, URLs Googlebot crawls that the crawl did not find and Googlebot gets errors.
- Fix, re-crawl and compare. Then confirm in Search Console over the following weeks: Google says crawling “can take anywhere from a few days to a few weeks” (ask Google to recrawl).
AI search engines run their own crawlers with their own rules, and most don’t render JavaScript. The AI search readiness checklist covers what changes for them. For where crawling and indexing fit in a full audit, see the technical SEO audit guide.
Crawling and indexing checklist
Discovery
- Every important page linked from at least one other page, ideally several
- Important pages within four clicks of the homepage
- Navigation and key links present in the raw HTML
- Sitemaps list only canonical, indexable 200 URLs, and are submitted in Search Console
Crawling
- robots.txt blocks only areas with no search value
- No 5xx errors, timeouts or bot-protection blocks for search engine crawlers
- Server responds in under a second
- No redirect chains; removed pages return 404 or 410
Indexing
- No unintended noindex in meta tags or headers
- Canonicals point at indexable 200 URLs
- Duplicates consolidated; thin pages improved, merged or removed
- Main content present without JavaScript, or rendering verified
Verification
- Page indexing report reviewed, status by status
- Crawl joined with Search Console data
- Server logs checked for Googlebot errors and uncrawled pages (large sites)
- Re-crawl and compare after fixes
All Crawling & Indexing guides
- Google Indexing API SEO: who can use it and what it cannot doThe Google Indexing API is limited to qualifying job posting and livestream pages. Learn its real use cases, notification types and alternatives.
- HTTP status codes for SEO: what 200, 301, 404, 410 and 5xx mean for GoogleHow Google treats each HTTP status code: which can be indexed, which pass canonical signals, which slow crawling, and how to audit them across a site.
- Search Console Crawl Stats report: what the charts actually tell youRead host status, response codes, crawl purpose and Googlebot type in Search Console Crawl Stats, then know when URL inspection or logs are needed.
- Sitemap lastmod: when to update it and how to audit itLearn which page changes deserve a new XML sitemap lastmod date, how to catch fake timestamps at scale, and what Google actually does with the tag.
- URL parameters and SEO: which URLs should Google crawl and index?Audit tracking, sorting, session and filter parameters without blocking valuable pages. Learn when to use canonical tags, redirects, robots.txt or a clean URL.
- X-Robots-Tag: how to use the HTTP header for noindex and moreWhat the X-Robots-Tag header does, when to use it instead of a meta tag (PDFs, images, whole folders), the rules it supports, and how to find it on a site.
- Canonical tag errors: how to find and fix themCanonicals pointing to redirects, errors or noindex pages, multiple canonicals and missing ones. How to audit rel=canonical across a site and fix what's broken.
- Crawled - currently not indexed: why it happens and how to fix itWhat Crawled and Discovered - currently not indexed mean in Search Console, when they're a real problem, how to find the cause, and how to fix it at scale.
- Internal linking audit: click depth, weak pages and dead endsHow to audit internal links across a site: pages buried too deep, pages with one link, dead ends, internal nofollow and broken links.
- Noindex vs robots.txt: which to use, and the mistakes that hide pagesRobots.txt controls crawling; noindex controls indexing. How each works, why combining them backfires, and how to find pages hidden or exposed by mistake.
- Orphan pages: how to find them and what to do with themOrphan pages have no internal links pointing to them, so users and crawlers struggle to find them. How to find them with sitemaps, Search Console and logs.
- Robots.txt for SEO: syntax, examples and the mistakes that block sitesHow robots.txt works, how crawlers pick which rules apply, copy-ready examples, what happens when the file fails to load, and how to test it across a site.
- Soft 404 errors: what they are and how to fix themWhy Google reports pages that return 200 as soft 404s, the usual causes (empty pages, error messages, homepage redirects) and how to find and fix them.
- XML sitemap audit: what to check and how to fix itSitemaps full of redirects, errors and noindex pages lose Google's trust. How to audit XML sitemaps against a crawl and keep only indexable URLs.
Frequently asked questions
What is the difference between crawling and indexing?
Crawling is when a search engine's bot downloads a page. Indexing is when the search engine analyses that page and stores it so it can be shown in results. A page can be crawled and not indexed, and, if it's blocked by robots.txt but linked from elsewhere, indexed without being crawled.
How long does it take Google to index a new page?
Google says crawling can take anywhere from a few days to a few weeks. Linking to the page from strong internal pages, listing it in your sitemap and requesting indexing in Search Console's URL Inspection tool can help, but none of them guarantees indexing.
Why is my page crawled but not indexed?
"Crawled - currently not indexed" means Google fetched the page and chose not to index it for now. Common reasons are duplicate or near-duplicate content, thin pages, or pages Google doesn't consider valuable enough. Improve or consolidate the page; Google says there's no need to resubmit it.
Does a sitemap guarantee my pages get indexed?
No. Google says a sitemap doesn't guarantee that all the items in it will be crawled and indexed. It helps discovery, especially on large or new sites, but pages still need to be indexable and worth indexing.
Can robots.txt keep a page out of Google?
Not reliably. Robots.txt stops crawling, not indexing: a blocked page can still be indexed from links pointing to it, without its content. To keep a page out of the index, let it be crawled and add a noindex directive.