Part of the Enterprise Technical SEO series. For the full picture, start with Enterprise technical SEO: a scalable audit framework.
Crawl budget gets blamed for a lot of indexing problems it didn’t cause. On most sites Google crawls everything it wants to, and pages that aren’t indexed have a quality or duplication problem, not a crawling one. On large sites it’s different: Googlebot can spend most of its time on filter combinations, old redirects and session parameters while new products wait weeks to be found.
This guide explains what crawl budget is, how to tell whether you have a problem, and how to find and fix the waste.
What is crawl budget?
Google defines crawl budget as “the set of URLs that Google can and wants to crawl” on a site (crawl budget management). Two things set it:
| Part | What it is | What changes it |
|---|---|---|
| Crawl capacity limit | How much crawling your server can take. It “limits the total amount of time your server spends holding connections open for Google.” | Goes up “if the site responds consistently and its response times remain stable or improve”; goes down “if the site slows down or responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429)” |
| Crawl demand | How much Google wants to crawl | Perceived inventory (how many URLs it knows about), popularity, and staleness (how often pages change) |
Capacity is mostly about your server. Demand is mostly about your URLs. Most crawl budget problems are demand being spent on the wrong URLs.
Does crawl budget matter for your site?
Google’s guide is written for three kinds of site:
- “Large sites (1 million+ unique pages) with content that changes moderately often (once a week)”
- “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)”
- Sites with a large share of URLs reported as Discovered - currently not indexed in Search Console
If your site isn’t one of these, Google says you don’t need the guide. A 2,000-page brochure site with indexing problems should look at content quality, duplicates and internal links instead; see crawled and discovered but not indexed.
The signs that crawl budget really is the constraint:
- New or updated pages take weeks to be crawled, even when they’re linked and in the sitemap.
- Many important URLs are Discovered - currently not indexed. Google’s Page indexing report help says this typically means “Google wanted to crawl the URL but this was expected to overload the site.”
- Logs show Googlebot spending most requests on URLs you’d never want indexed.
Where does crawl budget go to waste?
| Source of waste | Example | Fix |
|---|---|---|
| Faceted navigation and parameters | /shoes?color=red&size=9&sort=price in thousands of combinations |
Disallow filter parameters in robots.txt; keep only valuable combinations crawlable |
| Redirects and chains | Old URLs still linked internally, each one a hop | Link straight to the final URL; collapse chains |
| Errors | 404s from dead links, 5xx under load | Fix links; return 404/410 for removed pages; fix server errors |
| Soft 404s | Empty category pages returning 200 | Return a real 404, or give the page real content |
| Duplicates | http/https, www, trailing slash, tracking parameters |
Redirect to one version; canonicals; strip tracking parameters from internal links |
| Non-indexable pages crawled again and again | Noindexed or canonicalised URLs linked from everywhere | Block in robots.txt if they don’t need to be crawled, or stop linking to them |
| Slow responses | Templates that take two seconds to generate | Caching, faster queries, a CDN |
Two details from Google’s guide are worth repeating because they’re commonly done wrong:
- Noindex doesn’t save budget. “Don’t use noindex, as Google will still request, but then drop the page when it sees a noindex meta tag or header in the HTTP response, wasting crawling time.” Use robots.txt for URLs that shouldn’t be crawled at all.
- Faceted navigation is the classic trap. Google’s faceted navigation guide explains that filter URLs “can generate infinite URL spaces”, and because crawlers “can’t determine whether the URLs are going to be useful without crawling first”, they “will typically access a very large number of faceted navigation URLs.” It recommends robots.txt rules for filter parameters, notes that
rel="canonical"andrel="nofollow"are “generally less effective in the long term”, and suggests returning 404 when a filter combination has no results. See faceted navigation SEO.
How do you see where Googlebot spends its time?
Search Console’s Crawl Stats report (help) shows total crawl requests, total download size and average response time, plus host status for robots.txt fetching, DNS and server connectivity. Requests are broken down by response code, file type, purpose (discovery of new URLs or refresh of known ones) and Googlebot type. It’s available for domain properties and root-level URL-prefix properties.
It’s a sample and a summary. To see every URL, use server logs. Import them into Crawlens and the log analysis shows:
| View | What it answers |
|---|---|
| Where Googlebot spends its time | Googlebot requests by directory: which sections take the budget |
| Most crawled | The URLs Googlebot requests most often |
| Budget on non-indexable URLs | Redirects, noindexed and canonicalised URLs that still take Googlebot’s time |
| Errors served to Googlebot | URLs where Googlebot got a 4xx or 5xx |
| Crawled by Google, not found by crawl | Legacy URLs, parameters and orphans Google requests but your links don’t lead to |
| Indexable pages Googlebot ignored | Pages from the crawl with no Googlebot request in the period |
| Crawler requests per day | Googlebot, Bingbot and AI crawler activity over time |
Crawlens can verify Googlebot with reverse and forward DNS lookups, so fake Googlebots don’t distort the numbers. Three of these views also run as audit checks when a crawl and logs are combined: Googlebot gets errors, Indexable pages Googlebot did not crawl and URLs Googlebot crawls that the crawl did not find. The full method is in log file analysis for SEO.
How do you find crawl waste with a crawl?
Logs show what Googlebot did; a crawl shows what your site invites it to do. Run a full crawl and look for:
- Parameter URLs. Search the URLs table for
?to see how many parameter URLs your own links create. Hundreds of thousands of filter URLs from a few thousand products is the signature of a facet problem. - Links to redirects and errors. Internal links point to redirects, Redirect chains and Pages link to broken internal URLs each turn one page view into extra crawler requests.
- Duplicates. Duplicate content and Near-duplicate content show URL variants competing for the same crawls.
- Slow templates. Slow server response (over 1 second) lowers the crawl capacity limit for the whole site.
- Sitemap noise. Non-indexable URLs in sitemaps sends Google to URLs you don’t want crawled.
Use the crawl’s Exclude URLs matching setting to leave out a parameter pattern and see how much smaller the site becomes: that’s roughly how much crawling robots.txt rules for that pattern would save.
How do you fix a crawl budget problem?
Work in this order, from biggest saving to smallest:
- Stop generating crawlable junk. Disallow filter, sort and session parameters in robots.txt (see robots.txt for SEO), and make sure your templates don’t link to them where possible.
- Remove what’s gone. Return 404 or 410 for removed pages and stop linking to them. Fix soft 404s.
- Link to final URLs. Update internal links that point at redirects; collapse chains to one hop. See redirect chains.
- Consolidate duplicates. One URL per page: redirects for protocol and host variants, canonicals for the rest.
- Keep sitemaps clean and current. Only indexable 200 URLs, with accurate
lastmod. See the XML sitemap audit. - Make pages faster to serve. Response time feeds directly into the crawl capacity limit.
- Point demand at what matters. Link to important pages from strong, frequently crawled pages; see the internal linking audit.
Then compare. Import logs from a period after the changes: Googlebot’s share of requests on non-indexable URLs should fall, and more indexable pages should get crawled.
Checklist
- Confirmed the site is large or fast-changing enough for crawl budget to matter
- Crawl Stats report checked: host status, response codes, response time
- Server logs imported and Googlebot verified
- Sections and URLs taking most of Googlebot’s requests identified
- Filter, sort and session parameters disallowed in robots.txt
- Removed pages return 404 or 410; no soft 404s
- Internal links point at final URLs; no redirect chains
- Duplicate URL variants consolidated
- Sitemaps list only indexable URLs
- Server responds in under a second
- Logs compared before and after the changes
Frequently asked questions
What is crawl budget?
Google defines crawl budget as the set of URLs that Google can and wants to crawl on a site. It combines the crawl capacity limit, how much crawling the server can handle, with crawl demand, how much Google wants to crawl based on the site's size, popularity and how often it changes.
Does crawl budget matter for small sites?
Usually not. Google's crawl budget guide is written for sites with over a million unique pages that change weekly, sites with over ten thousand pages that change daily, and sites with many pages stuck as Discovered - currently not indexed. Smaller sites are normally crawled fully.
Does noindex save crawl budget?
No. Google still requests a noindexed page and only then drops it, so the crawl is spent anyway. To stop Google crawling URLs, block them in robots.txt or remove them and return 404 or 410.
How do I see my crawl budget?
Search Console's Crawl Stats report shows total crawl requests, download size, average response time and host status, broken down by response code, file type and purpose. Server logs show the same activity URL by URL.
What wastes crawl budget?
Mostly URLs that shouldn't be crawled: faceted navigation and parameter combinations, duplicates, soft 404s, redirect chains and pages linked internally that return errors. Slow server responses also lower how much Google is willing to crawl.