Crawling and indexability
Status codes, redirects, canonicals, noindex, robots.txt and XML sitemaps, so the pages you want found can be crawled and indexed.
Technical SEO audit service
Find out whether search engines and AI answer engines can reach, read and quote your important pages. We crawl the whole site, check the access rules classic audits skip, and hand you a short, evidence-backed list of what to fix first.
Site size, data access and the depth of each area are agreed before the audit begins.
Working view
Search & AI readiness audit
What the audit covers
Six areas, checked in the order problems usually block visibility. We weight them to your site: a JavaScript-heavy store and a documentation site fail in different places.
Status codes, redirects, canonicals, noindex, robots.txt and XML sitemaps, so the pages you want found can be crawled and indexed.
Which search, training and user-triggered AI crawlers your robots.txt, CDN and firewall let in, compared with the policy you actually want.
Compare the raw HTML with the rendered page to find content, links and SEO tags that only exist after scripts run.
Find nosnippet, max-snippet or data-nosnippet rules that limit quoting by accident, and markup that does not match the visible page.
Check that important pages answer their question in visible text, with clear headings, ownership and sources an engine can attribute.
Bring in Core Web Vitals field data, Search Console and server logs where you can share them, to confirm what crawlers and users really get.
The full checklist
What the audit looks at in each area, and where the evidence comes from. Areas that do not apply to your site, such as hreflang on a single-language site, are skipped.
| Check | What we look for | Evidence |
|---|---|---|
| robots.txt by user agent | Search crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) and training tokens (GPTBot, ClaudeBot, Google-Extended) are allowed or blocked as you intend. | Manual review |
| CDN and firewall rules | Bot management or WAF settings that challenge or block crawlers before robots.txt is ever read. | Manual reviewLive test |
| Linked pages blocked by robots.txt | Pages your own links point to that crawlers are not allowed to fetch. | Crawl |
| Crawler activity in logs | Which crawlers request which URLs, and the status codes they actually receive. | Server logs |
| Pages Googlebot skips | Indexable pages Googlebot did not request in the log period. | Server logs |
| Check | What we look for | Evidence |
|---|---|---|
| Noindex pages | Noindex on pages that should rank, and indexable pages that should not be. | Crawl |
| Canonical tags | Missing or multiple canonicals, and canonicals that point to error, redirected or noindex URLs. | Crawl |
| Canonicalised pages | Pages that name another URL as canonical, and whether that choice is right. | Crawl |
| Non-indexable URLs in sitemaps | Redirected, noindex or error URLs listed in XML sitemaps. | Crawl |
| Search traffic on non-indexable pages | Pages that earn impressions or clicks but are now noindex, canonicalised or erroring. | Search Console |
| Check | What we look for | Evidence |
|---|---|---|
| Client and server errors | Internal pages returning 4xx or 5xx, and pages that do not respond. | Crawl |
| Redirect chains and loops | Chains of two or more hops, loops and temporary (302/303/307) redirects that should be permanent. | Crawl |
| Links to redirects and broken URLs | Internal links that should point straight at the final URL. | Crawl |
| HTTPS | Pages served over HTTP, HTTP versions that do not redirect to HTTPS, and mixed content. | Crawl |
| Errors on pages with traffic | Landing pages with search or analytics visits that now return errors. | Search ConsoleAnalytics |
| Check | What we look for | Evidence |
|---|---|---|
| Click depth | Important pages more than four clicks from the start URL. | Crawl |
| Orphan and weakly linked pages | Pages in the sitemap that nothing links to, pages with a single internal link, and pages with no internal links out. | Crawl |
| Internal nofollow | Internal links marked nofollow that hold back discovery. | Crawl |
| XML sitemaps | Sitemaps that cannot be read, and important pages missing from them. | Crawl |
| URLs the crawl misses | URLs with impressions, Googlebot hits or analytics visits that the link crawl did not find. | Search ConsoleServer logsAnalytics |
| Check | What we look for | Evidence |
|---|---|---|
| Content that depends on JavaScript | Main text that appears only after scripts run, which most AI crawlers will not see. | Rendered crawl |
| Links only in rendered HTML | Navigation and internal links that do not exist in the raw HTML. | Rendered crawl |
| JavaScript changing SEO tags | Titles, canonicals or robots meta tags that differ between raw and rendered HTML. | Rendered crawl |
| Render failures | Pages that fail to render in a browser. | Rendered crawl |
| What a non-rendering crawler gets | The raw HTML of each key template, read the way an AI crawler without a browser reads it. | Manual review |
| Check | What we look for | Evidence |
|---|---|---|
| Titles and meta descriptions | Missing, duplicate, too long or too short titles; missing, duplicate or poorly sized descriptions. | Crawl |
| Headings | Missing or multiple H1s, and whether the heading outline matches the questions the page answers. | CrawlManual review |
| Duplicate and thin content | Exact and near-duplicate pages, and pages under 200 words that may not stand on their own. | Crawl |
| Answers in visible text | Key questions answered plainly near the top, with sources, ownership and author details where they matter. | Manual review |
| Images | Missing alt text and broken images. | Crawl |
| Check | What we look for | Evidence |
|---|---|---|
| JSON-LD validity | Invalid JSON-LD and structured data missing required properties. | Crawl |
| Markup matches the page | Structured data describes visible content and a supported use, with no blanket or AI-only markup. | Manual review |
| Snippet controls | nosnippet, max-snippet and data-nosnippet rules that limit quoting by accident. | Manual review |
| Entity consistency | Organization, product and author details agree across pages and markup. | Manual review |
| Check | What we look for | Evidence |
|---|---|---|
| Field Core Web Vitals | Poor or needs-improvement LCP, INP and CLS for real users. | PageSpeed Insights |
| Lab performance | Low Lighthouse performance scores on key templates. | PageSpeed Insights |
| Server response | Responses slower than one second, and HTML larger than 2 MB. | Crawl |
| Check | What we look for | Evidence |
|---|---|---|
| Hreflang codes | Invalid language or region codes. | Crawl |
| Hreflang links | Missing return links, missing self-references and missing x-default. | Crawl |
| Hreflang targets | Alternates that point to non-indexable URLs. | Crawl |
| Check | What we look for | Evidence |
|---|---|---|
| Pages close to page one | Queries ranking in positions 11 to 20, where a fix may move the page up. | Search Console |
| Low click-through rate | Pages with a lower CTR than their position would suggest. | Search Console |
| Indexable pages with no impressions | Pages search engines can index but do not show. | Search Console |
| Check | What we look for | Evidence |
|---|---|---|
| Google AI features in Search Console | The site is not excluded from AI Overviews and AI Mode by accident, and which pages get impressions in the generative AI performance report. | Search Console |
| Copilot and Bing AI citations | Citations, cited pages and grounding queries in the AI Performance report of Bing Webmaster Tools. | Bing Webmaster Tools |
| AI answer sampling | An agreed set of real questions run in Google AI Mode, ChatGPT search and Perplexity: whether you are cited, and who is cited instead. | Manual review |
| AI referral traffic | Visits from AI assistants are visible and kept apart from other referrals in your analytics. | Analytics |
| Self-contained passages | Each key section still makes sense when quoted on its own: it states the answer and its context without leaning on the paragraph before. | Manual review |
| Citable evidence | Original data, named sources and clear authorship that give an engine a reason to cite this page over another. | Manual review |
| Freshness signals | Visible updated dates agree with dateModified and sitemap lastmod, and changed pages are announced through IndexNow where you use it. | Manual review |
| llms.txt | Whether one is worth having. Google Search does not use it; it can help documentation and developer tools that people use with AI agents. | Manual review |
Crawl and Rendered crawl checks run in Crawlens audit rules. Search Console, Bing Webmaster Tools, analytics, PageSpeed Insights and log checks run only with the access you choose to share. Manual review is done by a person. AI answers change between runs, so each sampled answer is recorded with its date, tool and exact question. Reports used: Google's generative AI performance report and Bing's AI Performance report.
AI search readiness
AI answers are built from pages a system could crawl and read. Google says its AI features need no special optimization beyond being indexed and eligible for a snippet. Other answer engines run their own crawlers, often without JavaScript, and each can be allowed or blocked separately.
That is what Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) come down to in practice: pages that AI systems can reach, read, quote and attribute. The audit checks that foundation, then samples real AI answers for your topics to see who gets cited today.
What we check beyond a classic audit
Our full method is public in the AI search readiness checklist. We verify access and readability; no one can guarantee an AI citation.
How an audit runs
A long issue export is not an audit. The useful part is deciding which findings matter for this site and who needs to act on them.
Confirm goals, the sections of the site that matter most, your AI crawler policy and which data sources you can share.
Crawl the site with Crawlens, check rendering and access rules, and review the findings against Search Console and log data.
Rank the fixes by impact and effort, attach affected URLs and evidence, and go through them with the people who will make the changes.
What you get
Exact deliverables are set in the scope. Every finding comes with the pages it affects, why it matters and a suggested fix, so nothing depends on trusting a score.
Typical audit outputs
Prefer to run the crawl yourself? Crawlens is free, and the technical SEO audit guide walks through the process. For help after the audit, see managed SEO.
Before we start
Have a question about your site? Get in touch.
It is a technical SEO audit that also checks whether AI search systems can crawl, read and quote your pages. Beyond indexability, rendering and site structure, it reviews AI crawler access in robots.txt, the CDN and the firewall, and snippet controls that limit quoting.
Yes, in the practical sense. AEO and GEO are names for making content easy for answer engines to find, read and cite. The audit covers that technical foundation, samples real AI answers for your topics and reviews whether key pages are worth citing, without tactics that have no documented effect, such as AI-only markup.
The free app finds technical issues across the site. The audit adds the parts a crawler cannot decide: which findings matter for your goals, what your AI crawler policy should be, checks at the CDN and firewall, and a prioritized plan your team can work through.
A crawl needs only your public URLs. Read access to Search Console and analytics, and server logs if you can export them, make the findings much more reliable. We agree on access before the audit starts.
No audit can guarantee that. Search engines and AI systems decide which sources to show. The audit removes technical reasons a page cannot be crawled, read or quoted, which is the part you control.
The audit itself ends with a prioritized plan and a walkthrough. Your team can implement it, or we can agree ongoing support through managed SEO.
Start with your site
Send us your site, the sections that matter most and any recent changes. We will suggest an audit scope before any work begins.
Request an audit