New researchTechnical SEO & AI Search · 2027Explore the framework

Technical SEO audit service

A technical SEO audit that includes AI search

Find out whether search engines and AI answer engines can reach, read and quote your important pages. We crawl the whole site, check the access rules classic audits skip, and hand you a short, evidence-backed list of what to fix first.

Site size, data access and the depth of each area are agreed before the audit begins.

Illustrative workflowCrawlens / Services

Working view

Search & AI readiness audit

  1. 01
    Crawl the siteStatus codes, canonicals, index controls and links
  2. 02
    Check AI crawler accessrobots.txt, CDN and firewall against your policy
  3. 03
    Compare raw and renderedContent and tags that need JavaScript
  4. 04
    Rank the fixesImpact, affected URLs and who changes what

What the audit covers

From crawl access to quotable answers

Six areas, checked in the order problems usually block visibility. We weight them to your site: a JavaScript-heavy store and a documentation site fail in different places.

01

Crawling and indexability

Status codes, redirects, canonicals, noindex, robots.txt and XML sitemaps, so the pages you want found can be crawled and indexed.

02

AI crawler access

Which search, training and user-triggered AI crawlers your robots.txt, CDN and firewall let in, compared with the policy you actually want.

03

Rendering and JavaScript

Compare the raw HTML with the rendered page to find content, links and SEO tags that only exist after scripts run.

04

Snippet controls and structured data

Find nosnippet, max-snippet or data-nosnippet rules that limit quoting by accident, and markup that does not match the visible page.

05

Content structure

Check that important pages answer their question in visible text, with clear headings, ownership and sources an engine can attribute.

06

Performance and evidence

Bring in Core Web Vitals field data, Search Console and server logs where you can share them, to confirm what crawlers and users really get.

The full checklist

Every check, by category

What the audit looks at in each area, and where the evidence comes from. Areas that do not apply to your site, such as hreflang on a single-language site, are skipped.

01

Crawler access

Can search engines and AI crawlers reach the site at all?
5 checks
CheckWhat we look forEvidence
robots.txt by user agentSearch crawlers (Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) and training tokens (GPTBot, ClaudeBot, Google-Extended) are allowed or blocked as you intend.Manual review
CDN and firewall rulesBot management or WAF settings that challenge or block crawlers before robots.txt is ever read.Manual reviewLive test
Linked pages blocked by robots.txtPages your own links point to that crawlers are not allowed to fetch.Crawl
Crawler activity in logsWhich crawlers request which URLs, and the status codes they actually receive.Server logs
Pages Googlebot skipsIndexable pages Googlebot did not request in the log period.Server logs
02

Indexability and canonicals

Will the pages you want found be indexed, under the right URL?
5 checks
CheckWhat we look forEvidence
Noindex pagesNoindex on pages that should rank, and indexable pages that should not be.Crawl
Canonical tagsMissing or multiple canonicals, and canonicals that point to error, redirected or noindex URLs.Crawl
Canonicalised pagesPages that name another URL as canonical, and whether that choice is right.Crawl
Non-indexable URLs in sitemapsRedirected, noindex or error URLs listed in XML sitemaps.Crawl
Search traffic on non-indexable pagesPages that earn impressions or clicks but are now noindex, canonicalised or erroring.Search Console
03

Response codes and redirects

Do URLs answer correctly and quickly, for users and bots?
5 checks
CheckWhat we look forEvidence
Client and server errorsInternal pages returning 4xx or 5xx, and pages that do not respond.Crawl
Redirect chains and loopsChains of two or more hops, loops and temporary (302/303/307) redirects that should be permanent.Crawl
Links to redirects and broken URLsInternal links that should point straight at the final URL.Crawl
HTTPSPages served over HTTP, HTTP versions that do not redirect to HTTPS, and mixed content.Crawl
Errors on pages with trafficLanding pages with search or analytics visits that now return errors.Search ConsoleAnalytics
04

Site structure and discovery

Can crawlers find every important page through links and sitemaps?
5 checks
CheckWhat we look forEvidence
Click depthImportant pages more than four clicks from the start URL.Crawl
Orphan and weakly linked pagesPages in the sitemap that nothing links to, pages with a single internal link, and pages with no internal links out.Crawl
Internal nofollowInternal links marked nofollow that hold back discovery.Crawl
XML sitemapsSitemaps that cannot be read, and important pages missing from them.Crawl
URLs the crawl missesURLs with impressions, Googlebot hits or analytics visits that the link crawl did not find.Search ConsoleServer logsAnalytics
05

Rendering and JavaScript

Does the content exist without JavaScript?
5 checks
CheckWhat we look forEvidence
Content that depends on JavaScriptMain text that appears only after scripts run, which most AI crawlers will not see.Rendered crawl
Links only in rendered HTMLNavigation and internal links that do not exist in the raw HTML.Rendered crawl
JavaScript changing SEO tagsTitles, canonicals or robots meta tags that differ between raw and rendered HTML.Rendered crawl
Render failuresPages that fail to render in a browser.Rendered crawl
What a non-rendering crawler getsThe raw HTML of each key template, read the way an AI crawler without a browser reads it.Manual review
06

Content and page structure

Can a person, and a machine, find and quote the answer?
5 checks
CheckWhat we look forEvidence
Titles and meta descriptionsMissing, duplicate, too long or too short titles; missing, duplicate or poorly sized descriptions.Crawl
HeadingsMissing or multiple H1s, and whether the heading outline matches the questions the page answers.CrawlManual review
Duplicate and thin contentExact and near-duplicate pages, and pages under 200 words that may not stand on their own.Crawl
Answers in visible textKey questions answered plainly near the top, with sources, ownership and author details where they matter.Manual review
ImagesMissing alt text and broken images.Crawl
07

Structured data and snippet controls

Is the markup valid, and can engines quote the page?
4 checks
CheckWhat we look forEvidence
JSON-LD validityInvalid JSON-LD and structured data missing required properties.Crawl
Markup matches the pageStructured data describes visible content and a supported use, with no blanket or AI-only markup.Manual review
Snippet controlsnosnippet, max-snippet and data-nosnippet rules that limit quoting by accident.Manual review
Entity consistencyOrganization, product and author details agree across pages and markup.Manual review
08

Performance and Core Web Vitals

Do pages load fast enough for users and crawlers?
3 checks
CheckWhat we look forEvidence
Field Core Web VitalsPoor or needs-improvement LCP, INP and CLS for real users.PageSpeed Insights
Lab performanceLow Lighthouse performance scores on key templates.PageSpeed Insights
Server responseResponses slower than one second, and HTML larger than 2 MB.Crawl
09

International SEO

For multi-language or multi-region sites only.
3 checks
CheckWhat we look forEvidence
Hreflang codesInvalid language or region codes.Crawl
Hreflang linksMissing return links, missing self-references and missing x-default.Crawl
Hreflang targetsAlternates that point to non-indexable URLs.Crawl
10

Search performance and measurement

What the search data says, and what you can measure next.
3 checks
CheckWhat we look forEvidence
Pages close to page oneQueries ranking in positions 11 to 20, where a fix may move the page up.Search Console
Low click-through ratePages with a lower CTR than their position would suggest.Search Console
Indexable pages with no impressionsPages search engines can index but do not show.Search Console
11

AI answer visibility (AEO and GEO)

Are you cited in AI answers, and is each key page worth citing?
8 checks
CheckWhat we look forEvidence
Google AI features in Search ConsoleThe site is not excluded from AI Overviews and AI Mode by accident, and which pages get impressions in the generative AI performance report.Search Console
Copilot and Bing AI citationsCitations, cited pages and grounding queries in the AI Performance report of Bing Webmaster Tools.Bing Webmaster Tools
AI answer samplingAn agreed set of real questions run in Google AI Mode, ChatGPT search and Perplexity: whether you are cited, and who is cited instead.Manual review
AI referral trafficVisits from AI assistants are visible and kept apart from other referrals in your analytics.Analytics
Self-contained passagesEach key section still makes sense when quoted on its own: it states the answer and its context without leaning on the paragraph before.Manual review
Citable evidenceOriginal data, named sources and clear authorship that give an engine a reason to cite this page over another.Manual review
Freshness signalsVisible updated dates agree with dateModified and sitemap lastmod, and changed pages are announced through IndexNow where you use it.Manual review
llms.txtWhether one is worth having. Google Search does not use it; it can help documentation and developer tools that people use with AI agents.Manual review

Crawl and Rendered crawl checks run in Crawlens audit rules. Search Console, Bing Webmaster Tools, analytics, PageSpeed Insights and log checks run only with the access you choose to share. Manual review is done by a person. AI answers change between runs, so each sampled answer is recorded with its date, tool and exact question. Reports used: Google's generative AI performance report and Bing's AI Performance report.

AI search readiness

AI search is technical SEO, with more crawlers

AI answers are built from pages a system could crawl and read. Google says its AI features need no special optimization beyond being indexed and eligible for a snippet. Other answer engines run their own crawlers, often without JavaScript, and each can be allowed or blocked separately.

That is what Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO) come down to in practice: pages that AI systems can reach, read, quote and attribute. The audit checks that foundation, then samples real AI answers for your topics to see who gets cited today.

What we check beyond a classic audit

  • Search and training crawlers are allowed or blocked on purpose, not by a copied rule
  • Key content and links exist without JavaScript
  • Snippet controls do not stop engines quoting your pages
  • Server logs show AI crawlers fetching pages successfully, when logs are available

Our full method is public in the AI search readiness checklist. We verify access and readability; no one can guarantee an AI citation.

How an audit runs

Scope, crawl, then prioritize

A long issue export is not an audit. The useful part is deciding which findings matter for this site and who needs to act on them.

  1. 1

    Agree the scope

    Confirm goals, the sections of the site that matter most, your AI crawler policy and which data sources you can share.

  2. 2

    Crawl and review

    Crawl the site with Crawlens, check rendering and access rules, and review the findings against Search Console and log data.

  3. 3

    Prioritize and walk through

    Rank the fixes by impact and effort, attach affected URLs and evidence, and go through them with the people who will make the changes.

What you get

Findings your developers can act on

Exact deliverables are set in the scope. Every finding comes with the pages it affects, why it matters and a suggested fix, so nothing depends on trusting a score.

Typical audit outputs

  • A prioritized list of findings with severity and affected URLs
  • An AI crawler access summary: which bots can reach the site, and where they are blocked
  • Raw versus rendered HTML findings for key templates
  • Implementation notes for developers and content owners
  • An AI answer snapshot: the questions sampled, whether you are cited, and who is cited instead
  • A re-check of shipped fixes, where agreed

Prefer to run the crawl yourself? Crawlens is free, and the technical SEO audit guide walks through the process. For help after the audit, see managed SEO.

Before we start

Technical SEO & AI search audit FAQ

Have a question about your site? Get in touch.

What is a technical SEO and AI search audit?

It is a technical SEO audit that also checks whether AI search systems can crawl, read and quote your pages. Beyond indexability, rendering and site structure, it reviews AI crawler access in robots.txt, the CDN and the firewall, and snippet controls that limit quoting.

Is this an AEO or GEO audit?

Yes, in the practical sense. AEO and GEO are names for making content easy for answer engines to find, read and cite. The audit covers that technical foundation, samples real AI answers for your topics and reviews whether key pages are worth citing, without tactics that have no documented effect, such as AI-only markup.

How is this different from running Crawlens myself?

The free app finds technical issues across the site. The audit adds the parts a crawler cannot decide: which findings matter for your goals, what your AI crawler policy should be, checks at the CDN and firewall, and a prioritized plan your team can work through.

What access do you need?

A crawl needs only your public URLs. Read access to Search Console and analytics, and server logs if you can export them, make the findings much more reliable. We agree on access before the audit starts.

Will the audit get my site cited in AI answers?

No audit can guarantee that. Search engines and AI systems decide which sources to show. The audit removes technical reasons a page cannot be crawled, read or quoted, which is the part you control.

Do you fix the issues too?

The audit itself ends with a prioritized plan and a walkthrough. Your team can implement it, or we can agree ongoing support through managed SEO.

Start with your site

Want to know what is blocking your pages?

Send us your site, the sections that matter most and any recent changes. We will suggest an audit scope before any work begins.

Request an audit