New researchTechnical SEO & AI Search · 2027Explore the framework

Search Console Crawl Stats report: what the charts actually tell you

Read host status, response codes, crawl purpose and Googlebot type in Search Console Crawl Stats, then know when URL inspection or logs are needed.

Part of the Crawling & Indexing series. For the full picture, start with Crawling and indexing: how search engines find your pages, and what stops them.

A higher line on the Crawl Stats chart can look like good news. It might represent newly discovered pages, repeated redirects, image requests or a flood of server errors. The Search Console Crawl Stats report is most useful when you read the breakdowns before drawing a conclusion from the total.

What does the Search Console Crawl Stats report show?

It summarizes Google crawling activity for a verified root-level property: total requests, total download size, average response time, host status, and breakdowns by response, file type, purpose and Googlebot type. Google’s Crawl Stats help page says the report is aimed at advanced users; sites with fewer than about a thousand pages generally need not worry about this detail. It is available for Domain properties and root-level URL-prefix properties, not path-only properties.

One request is not necessarily one page. A page load may involve HTML, images, JavaScript and CSS; each redirect hop is also a request. The totals describe what Google fetched, not unique canonical pages and not the number indexed. For the latter, use Page Indexing and URL Inspection.

How do you read host status first?

Open host status before treating a crawl decline as a content problem. Google’s report checks robots.txt fetching, DNS resolution and server connectivity over the past 90 days. A recent availability issue calls for investigation of the corresponding failure chart and server logs. The official host-status explanation distinguishes recent issues from older, possibly resolved ones.

If Google cannot reliably reach the host, adding pages to a sitemap is not the first fix. Confirm whether failures line up with deployment windows, firewall changes, rate limits or hosting incidents. Then inspect response trends. This is an availability diagnosis, not an automatic instruction to increase the crawl rate.

What do the response and file-type breakdowns mean?

Use the response chart to find the type and timing of abnormal requests, then inspect example URLs and logs. A rise in 301/308 responses after a planned migration may be expected; a persistent rise in 5xx needs server attention. A 404 trend can reflect intentionally removed URLs or broken internal links, so sample before “fixing” every count. Google’s crawl troubleshooting guide stresses that crawl errors and indexing questions are different.

Breakdown Question it helps answer Follow-up check
Response Are errors or redirects growing? Sample URLs; compare logs and deployment dates
File type Is Google fetching HTML or mostly resources? Check which templates and assets changed
Crawl purpose Is activity discovery or refresh? Compare new URL releases with existing-page recrawls
Googlebot type Which Google crawler made requests? Compare smartphone and other types in context

Average response time includes resources, not only your HTML pages. A change in heavy image or script delivery can shift the number. Do not infer Core Web Vitals from this average; use Core Web Vitals data for user experience.

What can discovery, refresh and Googlebot type tell you?

Discovery describes requests for URLs Google has not crawled before; refresh covers URLs previously crawled. A post-launch discovery rise can be consistent with new pages being found, but the chart does not tell you whether those pages will be indexed. Googlebot type separates crawlers such as Smartphone Googlebot from others. Compare the mix with your site’s configuration rather than treating a single type as a score.

If a sudden increase appears, ask what changed: a sitemap, internal links, a parameter pattern, a migration, or a new resource bundle? Correlate dates with your release log and inspect examples. Our crawl budget guide explains when excess URL spaces warrant deeper action.

When do you need URL Inspection or server logs instead?

Use URL Inspection when the question is about a specific page’s crawlability, rendered content or indexing state. Use server or CDN logs when you need the exact requested path, timestamp, status and user agent across all requests. Google says Crawl Stats’ example URLs are samples and its report does not provide a complete per-path crawl history; the troubleshooting documentation points to logs for that level of detail. The log file analysis guide shows how to join that evidence with a site crawl.

Crawl Stats investigation checklist

  • Use a root-level verified property and note the 90-day context
  • Check host status and recent DNS, robots.txt or connectivity failures
  • Compare request, download and response-time changes with releases
  • Break down changes by response, file type, purpose and Googlebot type
  • Inspect example URLs, remembering they are not an exhaustive list
  • Use URL Inspection for individual pages and logs for full path-level history
  • Verify whether the issue is crawling, indexing or site performance before acting

Crawl Stats is a pattern detector. The fix comes from connecting those patterns to real URLs and server behavior.

Frequently asked questions

Where is the Crawl Stats report in Search Console?

Open a verified root-level Domain or URL-prefix property, then find Crawl Stats under Settings. Google does not provide the report for a URL-prefix property limited to a subpath.

Does a spike in crawl requests mean Google indexed more pages?

No. A crawl request is a fetch, not an indexing decision. Compare Page Indexing and URL Inspection when the question is whether particular pages are in Google's index.

Why are there more crawl requests than pages on my site?

Googlebot fetches images, scripts, stylesheets, redirects and repeated page versions as well as HTML pages. Each request in a redirect chain is counted separately.

Can Crawl Stats show every URL Googlebot visited?

No. The URL list is a sample and Search Console does not offer complete crawl history filtered by path. Use verified server or CDN access logs when you need the exact request trail.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows