An XML sitemap is a list of URLs you’re telling search engines matter. That’s only useful if the list is accurate. Sitemaps tend to drift: pages get redirected, noindexed or deleted, but the sitemap generator keeps listing them. Over time the file fills with URLs that contradict everything else on the site.
This guide covers what belongs in a sitemap, what to check, and how to audit sitemaps against a real crawl.
What does Google expect from a sitemap?
From Google’s guide to building a sitemap:
- Size: a single sitemap is limited to 50 MB uncompressed or 50,000 URLs. Split larger sites into several sitemaps, optionally listed in a sitemap index.
- URLs: use fully qualified, absolute URLs. Google crawls them exactly as listed.
- Tags: Google ignores
<priority>and<changefreq>, and uses<lastmod>only if it’s consistently and verifiably accurate. - Submission: submit in Search Console, or add a
Sitemap:line to robots.txt.
And from the sitemap overview: a sitemap “doesn’t guarantee that all the items in your sitemap will be crawled and indexed.” It’s a hint, not a substitute for internal links.
What should an XML sitemap audit check?
1. Can every sitemap be read?
A sitemap that returns an error, or isn’t valid XML, is ignored. Check every file, including the children of a sitemap index, since one broken child can drop a whole section.
2. Is every URL indexable?
This is where most sitemaps fail. A URL in a sitemap should return 200 and be indexable. Common offenders:
| Problem | Why it’s there |
|---|---|
| Redirect (3xx) | Page moved, generator still uses the old URL |
| Error (4xx/5xx) | Page deleted, sitemap not regenerated |
| Noindex | Page deliberately hidden, but still listed |
| Canonicalised | A parameter or variant URL listed instead of the canonical |
| Blocked by robots.txt | Section disallowed, sitemap not updated |
Each of these sends a mixed signal: “this URL matters” in the sitemap, “this URL doesn’t” everywhere else.
3. Are sitemap pages linked from the site?
A page that’s in the sitemap but has no internal links is an orphan page. Google can discover it, but it gets no internal link signals and visitors can’t find it.
4. Is lastmod honest?
If every URL shows today’s date on every generation, Google learns to ignore it. Only update lastmod when the page meaningfully changes.
How do you audit sitemaps against a crawl?
Comparing a sitemap with reality means crawling the URLs in it and checking their status, indexability and links. In Crawlens:
- Start a crawl in Sitemap mode. Enter your sitemap URLs, or leave the field empty to use the sitemaps listed in robots.txt, or
/sitemap.xml. Sitemap indexes and.xml.gzfiles are supported. - Turn on Also follow links, so the crawl also maps internal links and can find orphans and pages missing from the sitemap.
- Read the sitemap checks in the audit:
| Check | What it flags | Severity |
|---|---|---|
| Sitemaps that could not be read | Sitemap files returning an error or invalid XML | Warning |
| Non-indexable URLs in sitemaps | Listed URLs that redirect, error, are noindexed, canonicalised or blocked, with the reason and the sitemap they came from | Warning |
| Orphan pages | Listed URLs that no page links to | Warning |
Because each flagged URL shows which sitemap it came from, you can trace problems back to the generator that produced them.
How do you fix sitemap problems?
Fix the generator, not the file. Most sitemaps are generated by the CMS or a plugin. Configure it to include only indexable, canonical URLs, and to drop pages that redirect, are noindexed or are deleted. Hand-edited sitemaps drift again within weeks.
Split by type. Separate sitemaps for products, categories, articles and so on make problems easier to spot, both in a crawl and in Search Console’s Sitemaps report.
Link what you list. Every page worth listing is worth linking. Add orphan pages to navigation or related content, or drop them from the sitemap if they no longer matter.
Re-crawl after changes. Run the sitemap crawl again and compare with the previous one; the issues should show up as fixed.
Checklist
- Every sitemap and sitemap index child returns 200 with valid XML
- Under 50,000 URLs and 50 MB (uncompressed) per sitemap
- Only absolute, canonical, indexable URLs returning 200
- No redirects, errors, noindexed, canonicalised or blocked URLs
- Every listed page linked from the site
-
lastmodonly changes when content does; no reliance on priority or changefreq - Sitemap submitted in Search Console and listed in robots.txt
Frequently asked questions
What should an XML sitemap include?
Only the canonical, indexable URLs you want in search, each returning 200, written as full absolute URLs. Leave out redirects, errors, noindexed pages, canonicalised duplicates and URLs blocked by robots.txt.
How big can a sitemap be?
Google limits a single sitemap to 50 MB uncompressed or 50,000 URLs. Larger sites split URLs across several sitemaps and can submit a sitemap index that lists them.
Does Google use priority and changefreq?
No. Google ignores priority and changefreq. It uses lastmod if the value is consistently and verifiably accurate.
Does a sitemap guarantee indexing?
No. Google says a sitemap helps it discover URLs but doesn't guarantee that they'll be crawled or indexed. Internal links still matter.
How do I tell Google about my sitemap?
Submit it in Search Console's Sitemaps report, or add a Sitemap line with its full URL to robots.txt.