# Crawlens > Crawlens is a free desktop app for Windows (current version 0.20.1) that crawls websites, up to 100,000 URLs per crawl, and audits technical SEO: 70 built-in checks, JavaScript rendering, Core Web Vitals, Google Search Console and GA4 data joined to every URL, server log analysis, and an MCP server so AI assistants (Claude, Codex, Cursor, VS Code) can read the audits. Crawl data stays on the user's computer. ## Product - [Crawlens home](https://crawlens.cc/): features, changelog and download ## Docs - [Your first crawl](https://crawlens.cc/docs/first-crawl/): install, data folder, crawl modes and settings, health score, issues, exports - [Audit rules reference](https://crawlens.cc/docs/audit-rules/): all 70 checks with severity, why each matters, how to fix it and what data it needs; health score formula - [Connect Crawlens to Claude & other AI tools (MCP)](https://crawlens.cc/docs/mcp/): enable sharing, per-app setup, the 8 MCP tools and 3 prompts, privacy and troubleshooting - [Technical SEO glossary](https://crawlens.cc/glossary/): plain-English definitions of the technical SEO terms used in audits, with links to full guides ## About - [About Crawlens](https://crawlens.cc/about/): what it is, principles, founder - [Contact](https://crawlens.cc/contact/) - [Crawlens on LinkedIn](https://www.linkedin.com/company/crawlens/) - [Crawlens on GitHub](https://github.com/Crawlens-cc) ## Comparisons - [A free Screaming Frog alternative for technical SEO audits](https://crawlens.cc/compare/screaming-frog-alternative/): Crawlens is a free alternative to Screaming Frog for Windows. Screaming Frog's free version stops at 500 URLs and locks JavaScript rendering, Search Console, GA and PageSpeed integrations behind a $279/year licence. Crawlens includes them free for crawls up to 100,000 URLs. Screaming Frog still wins on Mac/Linux support, unlimited crawl size and breadth of features. - [A free Sitebulb alternative for Windows](https://crawlens.cc/compare/sitebulb-alternative/): Crawlens is a free alternative to Sitebulb Desktop on Windows. Sitebulb is paid after a 14-day trial: Lite crawls 10,000 URLs per audit, and features like audit comparison, scheduling, PDF reports, structured data and hreflang need Pro. Crawlens includes those free up to 100,000 URLs. Sitebulb is stronger on Mac support, crawl size on Pro, visual reports and number of checks. ## Blog - [Astro SEO: a step-by-step setup guide](https://crawlens.cc/blog/astro-seo/): Astro renders full HTML at build time, but SEO tags are your job. Set site and trailingSlash, write one SEO component for the head, filter drafts in one helper, configure @astrojs/sitemap with real lastmod dates, add hreflang and JSON-LD yourself, put redirects in your host's config, and run checks on dist/ after every build. - [Next.js SEO: a step-by-step setup guide for the App Router](https://crawlens.cc/blog/nextjs-seo/): Next.js renders pages on the server, but it doesn't choose canonicals or sitemap entries for you. Set metadataBase, build every page's metadata with one helper, never put a canonical in the root layout, return real 404s before streaming starts, generate sitemap.ts from content dates, and check the running build before you deploy. - [Best static site generator for SEO and AI search: Astro, Next.js, Hugo and more compared](https://crawlens.cc/blog/static-site-generator-seo/): For SEO and AI search, the best static site generator is one that puts every page's content, links and head tags in the HTML at build time. Astro, Hugo and Eleventy ship no client JavaScript by default; Next.js, Nuxt, SvelteKit, VitePress and Docusaurus also prerender HTML but hydrate it. Pick by the SEO plumbing you'd otherwise build yourself. - [Static site SEO: a technical checklist for Astro and other generators](https://crawlens.cc/blog/static-site-seo/): A static site generator outputs fast, crawlable HTML, but it makes no SEO decisions for you. Put every head tag in one layout, align trailing slashes between the framework and the host, build the sitemap from the same data as noindex and hreflang, and add post-build checks so a broken rule fails the deploy. - [Back/forward cache and Core Web Vitals: test return visits](https://crawlens.cc/blog/back-forward-cache-core-web-vitals/): The back/forward cache, or bfcache, lets a browser restore a previous page from memory during history navigation instead of loading it again. That can make return visits feel much faster and changes how real-user page experience is measured. Test representative routes in DevTools, fix actionable blockers, and refresh stale state safely after a restore. - [Enterprise technical SEO: a scalable audit framework](https://crawlens.cc/blog/enterprise-technical-seo/): Enterprise technical SEO applies crawl, indexing and performance checks across many URLs, templates, markets and teams. Start with a trustworthy URL inventory and group pages by business value and template. Compare crawler results, server logs, sitemaps and Search Console data, then fix shared causes through owners and release checks. Scale the decision process before scaling crawl volume. - [Google Discover Performance report: what the data means](https://crawlens.cc/blog/google-discover-performance-report/): The Google Discover Performance report shows clicks, impressions and CTR for eligible content shown in Discover, separately from Search results. It appears only after a property reaches Google's minimum Discover-impression threshold. Compare pages and periods, account for preliminary data and canonical URLs, and investigate large changes without treating Discover's interest-driven traffic like keyword rankings. - [JavaScript SEO: the complete rendering and indexing guide](https://crawlens.cc/blog/javascript-seo/): JavaScript SEO makes content, links and page signals discoverable when a site uses scripts to render or navigate. Google can execute JavaScript, but it first crawls the server response and rendering may happen later. Give important pages stable URLs, crawlable links and correct HTTP signals, then compare raw and rendered HTML and test representative pages in Search Console. - [Merchant Center crawler blocked: diagnose product access](https://crawlens.cc/blog/merchant-center-crawler-blocked/): When Merchant Center reports that a product page cannot be crawled, test the exact submitted landing URL and image as Google would reach them. Check robots.txt, mobile access, redirects, server responses and CDN or bot-protection logs before changing the feed. A browser success or a spoofed Googlebot user agent alone cannot prove genuine Google crawlers are allowed through. - [Monitor redirects after a site migration: an ongoing QA plan](https://crawlens.cc/blog/monitor-redirects-after-migration/): To monitor redirects after a site migration, keep the old-to-new URL map and test it repeatedly after launch. Check that valuable old URLs reach intended live destinations through permanent redirects, while new internal links and sitemaps use final URLs. Pair crawl results with Search Console and traffic trends, and keep redirects for at least a year. - [Planned website outage SEO: keep a short shutdown safe](https://crawlens.cc/blog/planned-site-outage-seo/): For a planned website outage, keep public pages online with limited functionality whenever possible. If the whole site must be unavailable briefly, return a real 503 Service Unavailable response with a sensible Retry-After header, keep robots.txt accessible, and restore normal 200 responses promptly. Do not use a 200 maintenance page, sitewide noindex or permanent error as a shortcut. - [Technical international SEO: the complete implementation guide](https://crawlens.cc/blog/technical-international-seo/): Technical international SEO gives each language or regional page a stable, crawlable URL and tells search engines which pages are alternatives. Choose a maintainable URL structure, publish genuinely localized content, keep each version indexable, and implement reciprocal hreflang across equivalent pages. Then test canonicals, redirects, sitemaps and search performance separately for each locale. - [Branded vs non-branded search traffic: how to separate them in Search Console](https://crawlens.cc/blog/branded-vs-non-branded-traffic/): Branded searches include your name; non-branded searches are the topics you want to be found for. Search Console's branded queries filter splits them automatically for eligible sites. Where it isn't available, or for your own definition of brand, use a regex query filter. Track non-branded clicks to see whether SEO work is reaching new people. - [ccTLD SEO migration: move regional sites without losing locale signals](https://crawlens.cc/blog/cctld-seo-migration/): A ccTLD SEO migration changes page URLs and a country signal. Map each old regional page to its equivalent destination, keep locale versions distinct, redirect directly, and update canonicals, links, sitemaps and reciprocal hreflang. Check that users and Googlebot can reach every locale without automatic IP redirects. Monitor each country and language group separately after launch. - [CSS animations and layout shifts: find the CLS culprit](https://crawlens.cc/blog/css-animation-layout-shift/): CSS animation does not automatically count as a layout shift. Animating layout properties such as top, left, width or height can move other visible elements and raise CLS when the shift is unexpected. Reproduce the interaction in DevTools, identify which element moved and why, then prefer transform or opacity for visual motion while preserving space and usability. - [Ecommerce category page SEO: make collections useful and crawlable](https://crawlens.cc/blog/ecommerce-category-page-seo/): An ecommerce category page should help shoppers choose a product and help crawlers reach those products. Give each category a distinct purpose, descriptive title and useful guidance. Link through real HTML anchors to subcategories and products, provide stable pagination, and control low-value filter combinations. Audit empty and duplicate categories before adding copy: crawlability and relevance matter more than word count. - [Ecommerce product reviews SEO: useful ratings without markup mistakes](https://crawlens.cc/blog/ecommerce-product-reviews-seo/): Ecommerce product reviews help shoppers when they describe real experience and remain visible on the product page. Review or AggregateRating markup must represent the displayed product and match its ratings. Validate the implementation, moderate abuse without cherry-picking sentiment, and distinguish merchant customer reviews from editorial review pages. Structured data can make a page eligible for enhancements, not guarantee stars. - [First-party JavaScript performance: reduce main-thread work](https://crawlens.cc/blog/first-party-javascript-performance/): First-party JavaScript performance improves when the code your team ships does less work on the main thread. Start with real-user INP to find slow templates and interactions, then trace loading, parsing, execution and event handlers in the browser. Remove unused code, split noncritical work and shorten callbacks before fine-tuning individual functions. Recheck field data after release. - [Font Loading and Core Web Vitals: Diagnose and Fix](https://crawlens.cc/blog/font-loading-core-web-vitals/): Font loading can delay visible text and move it when a fallback face is replaced, affecting paint timing and sometimes Cumulative Layout Shift. Diagnose the actual font requests and affected elements before changing CSS. Reduce unused font files, prioritize only critical faces, choose font-display deliberately, and validate the result with lab and field data. - [GA4 landing page analysis for SEO: find pages that earn traffic but lose outcomes](https://crawlens.cc/blog/ga4-landing-page-seo-analysis/): The GA4 Landing page report shows the first page of each session. For SEO, filter to Organic Search and compare pages by sessions, engagement and key events, not sessions alone. Pair the results with Search Console impressions, clicks and queries to separate discovery problems from post-click problems. Check tracking and attribution before treating a low-conversion page as weak content. - [Google Free Listings SEO: Eligibility and Diagnostics](https://crawlens.cc/blog/google-free-listings-seo/): Google free listings can show eligible products across Google without an ad charge, but permission to appear does not guarantee placement. Check the Free listings marketing method, product status and Merchant Center Needs attention tab. Then fix the specific account, feed, product-page or policy issue and confirm that Google can crawl a consistent landing page. - [Google Indexing API SEO: who can use it and what it cannot do](https://crawlens.cc/blog/google-indexing-api/): Google's Indexing API is not a shortcut for indexing ordinary articles, products or landing pages. Its documented eligibility is limited to pages with JobPosting markup or BroadcastEvent embedded in VideoObject. Eligible publishers can notify Google when those URLs are added, updated or removed; the API prompts crawling but does not guarantee indexing. Other sites should use crawlable links and sitemaps. - [Google manual actions SEO: diagnose the notice and request review](https://crawlens.cc/blog/google-manual-actions-seo/): A Google manual action is a human-reviewed spam-policy finding shown in Search Console, not a label for every traffic drop. Read the action type and affected patterns, remove the violation across all affected pages, document evidence, and request review only after fixes are complete. A clean manual-actions report does not rule out algorithmic or technical causes. - [Google Search Essentials audit: technical checks and policy review](https://crawlens.cc/blog/google-search-essentials-audit/): A Google Search Essentials audit has two parts: technical eligibility and human policy review. Verify that Googlebot can access important pages, they return a successful status and contain indexable content. Then assess whether the site's content and tactics serve users or risk violating Google's spam policies. Automated crawling can find evidence, but it cannot certify content quality or policy compliance. - [Hosting Migration SEO Checklist Without URL Changes](https://crawlens.cc/blog/hosting-migration-seo/): A hosting migration changes the infrastructure serving a site while its public URLs stay the same. Prepare the new host or CDN, compare representative responses and crawl access, then switch DNS and watch both old and new server logs. Keep the old environment until users and Googlebot reliably reach the new one. Redirect mapping is not central here. - [HTTP status codes for SEO: what 200, 301, 404, 410 and 5xx mean for Google](https://crawlens.cc/blog/http-status-codes-seo/): Only pages that return 200 can be indexed. Permanent redirects (301, 308) pass the canonical signal to their target; temporary ones (302, 307) don't. 404 and 410 remove a URL from the index without hurting the rest of the site, while 5xx and 429 make Google slow down crawling, and persistent server errors get URLs dropped. - [HTTP to HTTPS migration SEO: redirect every URL and verify the move](https://crawlens.cc/blog/http-to-https-migration/): An HTTP to HTTPS migration changes every affected URL's protocol. Install and test the certificate, map each HTTP URL to its HTTPS equivalent with a permanent redirect, update canonical tags, internal links and sitemaps, then crawl both versions after launch. The Search Console Change of Address tool is not needed for a protocol-only move. - [Image optimization for Core Web Vitals: improve LCP without causing CLS](https://crawlens.cc/blog/image-core-web-vitals/): Images affect Core Web Vitals in two ways: a late or oversized hero can slow LCP, and an image without reserved space can cause CLS. Identify the LCP element first. Serve responsive sizes, keep the hero discoverable and eager, and reserve dimensions for every image. Lazy-load images below the first screen, and verify lab and real-user results after deployment. - [Internal site search pages SEO: index, noindex or create a category?](https://crawlens.cc/blog/internal-site-search-seo/): Most internal site search result URLs are useful to shoppers but poor search landing pages: queries can be endless, temporary or empty. Keep product discovery rooted in crawlable category links. Noindex low-value result pages when appropriate, and turn recurring, valuable searches into stable curated pages with distinct content and products. Audit the actual URL patterns before applying a blanket rule. - [International URL mapping for SEO migrations](https://crawlens.cc/blog/international-url-mapping-seo/): International URL mapping pairs each old localized page with the new page serving the same language, region and intent. Keep the mapping at page level, record missing translations, and test redirects before launch. After a move, update canonicals, hreflang, internal links and sitemaps to the new URLs, then verify that every locale remains directly accessible and monitor markets separately. - [Lazy loading and SEO: keep content and links visible to Google](https://crawlens.cc/blog/lazy-loading-seo/): Lazy loading defers content until needed, but Google does not scroll or click to reveal a page. Important content must load when visible, and deeper items need crawlable links to persistent URLs. Keep the first screen eager, give images real source URLs, and test rendered HTML with Search Console. Use pagination behind infinite scroll so products remain discoverable. - [Long Animation Frames and INP: find the work behind slow clicks](https://crawlens.cc/blog/long-animation-frames-inp/): Long Animation Frames (LoAF) records browser frames delayed beyond 50 milliseconds and can attribute some of their script and rendering work. Use it alongside interaction-level INP data to identify what delayed a click or tap, reproduce the path, and reduce the expensive work. A LoAF entry is a diagnostic clue, not the INP score itself. - [Merchant Center automatic item updates: setup and limits](https://crawlens.cc/blog/merchant-center-automatic-item-updates/): Merchant Center automatic item updates use product-page structured data and other extracted information to correct occasional differences in price, sale price, availability or condition for ads and free listings. Check Automations settings and make the visible offer, markup and submitted data agree. This is a safety net for sporadic mismatches, not a replacement for timely feed or API updates. - [Merchant Center feed vs structured data: fix product mismatches](https://crawlens.cc/blog/merchant-center-feed-structured-data/): When a Merchant Center feed disagrees with product-page structured data or the visible offer, identify the stale source. Compare the variant URL, currency, price, availability and condition shown to shoppers with submitted data and rendered schema. Update the source of truth, sync the data, and recheck affected items. Automatic updates are a safety net, not a substitute for accurate feeds. - [Merchant shipping and returns SEO: keep policy data consistent](https://crawlens.cc/blog/merchant-shipping-returns/): Shipping and return information can appear in Google's merchant experiences when a store provides eligible policy data. Put accurate policies where shoppers can find them, align Merchant Center settings with product and checkout pages, and use supported structured data where appropriate. Test the markup and review diagnostics; providing data creates eligibility, not a guarantee that an enhancement will show. - [Merge two websites SEO checklist: preserve the pages that matter](https://crawlens.cc/blog/merge-two-websites-seo/): To merge two websites without needless SEO loss, inventory both sites before changing URLs. Decide which pages move, combine, or have no useful replacement. Map each old URL to a relevant destination, use direct permanent redirects, update links and page signals, and compare combined performance across the old properties and destination after launch. Move sites in stages when practical. - [Mobile-first indexing audit: find what Googlebot misses on mobile](https://crawlens.cc/blog/mobile-first-indexing-audit/): A mobile-first indexing audit checks what Smartphone Googlebot can actually access and render, not just whether a page looks good on a phone. Compare mobile and desktop primary content, links, metadata, structured data and images. Test robots rules, user-interaction loading and separate mobile URL signals. Prioritize missing or blocked content on pages that matter to search. - [Mobile page speed: diagnose slow pages with field and lab data](https://crawlens.cc/blog/mobile-page-speed/): To improve mobile page speed, check real-user mobile Core Web Vitals for the URL or origin, then use a controlled lab run to diagnose the slow template. Separate loading problems from interaction and layout problems: LCP, INP and CLS need different fixes. Test on representative devices, make one measurable change, and monitor the field trend after launch. - [Perplexity search SEO: check crawler access and citation-ready pages](https://crawlens.cc/blog/perplexity-search-seo/): Perplexity search visibility starts with pages its search crawler can reach and understand. Perplexity documents PerplexityBot for surfacing linked websites and a separate Perplexity-User fetcher for user requests. Check robots.txt, WAF rules, server responses and readable page content, then test representative questions and citations. Allowing a crawler is necessary for discovery but does not guarantee citation. - [Render-blocking resources SEO: fix CSS and scripts delaying LCP](https://crawlens.cc/blog/render-blocking-resources/): Render-blocking resources delay the browser's first useful paint when critical CSS or synchronous scripts must load before rendering. Diagnose them in a performance trace, keep above-the-fold styling available, defer nonessential scripts and styles, and verify the LCP element appears sooner in field data. Removing every blocking file is not the goal; preserving correct rendering is. - [Search Console 24-Hour Data: Read Fresh Trends Safely](https://crawlens.cc/blog/search-console-24-hour-data/): The Search Console 24-hour view shows recent Performance data by hour, including clicks, impressions and the pages or queries behind them. The newest points are preliminary and can change, so a single low hour is not proof of an SEO problem. Compare equivalent periods, check indexing and analytics, then revisit the finalized trend. - [Search Console BigQuery export: keep and analyze large SEO datasets](https://crawlens.cc/blog/search-console-bigquery-export/): Search Console bulk export sends daily performance data to BigQuery for large-scale analysis beyond UI and API row limits. It includes URL- and property-level tables but still omits anonymized queries. Set up the Google Cloud project and permissions, choose an export destination, check the export log, and use partition filters and pre-aggregated tables to manage query costs. - [Search Console Crawl Stats report: what the charts actually tell you](https://crawlens.cc/blog/search-console-crawl-stats/): Search Console's Crawl Stats report shows how Googlebot has requested your site over time, not which pages are indexed. Start with host availability, then compare request trends, response codes, file types, crawl purpose and Googlebot type. Use the sample URLs to investigate patterns, but turn to URL Inspection or server logs for a specific page's status or full crawl history. - [Search Console Insights: turn trend cards into content decisions](https://crawlens.cc/blog/search-console-insights/): Search Console Insights summarizes search performance trends. Use its clicks, impressions, top and trending content, queries and countries to choose what deserves investigation. A trending card is a lead, not a diagnosis: open the corresponding Performance report with the same date range, check page and query context, and then decide whether to update, expand or leave content alone. - [Search Console query groups: what the Insights cards really mean](https://crawlens.cc/blog/search-console-query-groups/): Search Console query groups combine similar searches inside Insights for properties with enough query data. A group's name is its top query, while its click total covers the whole group. Visible examples may be truncated and groups can change over time. Use them to identify audience themes, then drill into Performance before drawing conclusions about individual queries or creating pages. - [Search Console Search Appearance: read the report correctly](https://crawlens.cc/blog/search-console-search-appearance/): Search Appearance in Search Console groups results by the feature or result type shown, such as product snippets, merchant listings or videos. Filter a feature, compare periods and pages, then check eligibility and indexing separately. The table is page-aggregated, features can overlap, and filtered clicks do not prove the user clicked the enhanced part of a result. - [Google Search Console Security Issues: Audit and Fix](https://crawlens.cc/blog/search-console-security-issues/): The Search Console Security Issues report shows when Google detects hacked content, malware or deceptive behavior that may harm visitors. Treat its example URLs as clues, not a complete inventory. Confirm the issue, find and remove its root cause across the site, test the cleanup, then request a review in that report. - [SEO regression testing: catch search-breaking releases](https://crawlens.cc/blog/seo-regression-testing/): SEO regression testing compares a representative set of important URLs before and after a website release to catch changes that unintentionally block or alter search access. Check responses, robots and noindex directives, canonicals, internal links, rendered content, sitemaps and structured data according to what the release touched. Fix launch-breaking changes first, then monitor Search Console as Google recrawls. - [Sitemap index SEO: split large XML sitemaps without losing coverage](https://crawlens.cc/blog/sitemap-index-seo/): A sitemap index is an XML file that lists separate sitemap files. Use one when a sitemap exceeds Google's 50,000 URL or 50 MB uncompressed limit, or when separate sections need easier monitoring. Keep each child sitemap within the limits, list canonical URLs, submit the index, and check each child in Search Console. - [Sitemap lastmod: when to update it and how to audit it](https://crawlens.cc/blog/sitemap-lastmod-seo/): A sitemap lastmod date should reflect the last significant change to a URL, such as updated main content, structured data or links. Do not set every date to today whenever the sitemap is generated. Audit against real page changes and fix the CMS or deployment logic that produces it. Accurate lastmod is a crawl hint, not an indexing guarantee. - [SPA navigation SEO: make client-side routes crawlable](https://crawlens.cc/blog/spa-navigation-seo/): For SPA navigation SEO, every important view needs a working URL and a crawlable link. Use anchors with href values and History API routes instead of hash-only navigation or click handlers on other elements. Test a fresh load of each route, its HTTP response, rendered content, canonical and links, because Googlebot may land on a deep URL directly. - [Staging site SEO: keep previews private without shipping noindex](https://crawlens.cc/blog/staging-site-seo/): Staging site SEO has two jobs: keep private previews out of public search, then ensure those controls do not reach production. Password-protect nonpublic staging content, test how previews are exposed, and maintain a launch checklist for robots, noindex, canonicals and server responses. A robots.txt disallow is not a privacy control and can hide a noindex rule from Google. - [Subdomain to subfolder SEO migration: move URLs without losing pages](https://crawlens.cc/blog/subdomain-to-subfolder-seo-migration/): A subdomain-to-subfolder migration changes every moved URL, even when the articles stay the same. Map each old page to its closest new equivalent, serve a direct permanent redirect, update canonical and internal links, and submit the new sitemap. Keep both hosts verifiable and monitor crawl, indexing and search traffic by URL group. Do not assume a wildcard redirect covers exceptions. - [Third-party scripts and Core Web Vitals: find the tags slowing your pages](https://crawlens.cc/blog/third-party-scripts-core-web-vitals/): Third-party scripts can delay rendering and occupy the browser's main thread, affecting LCP and INP. Identify costly domains and tasks with PageSpeed Insights and Chrome DevTools. Remove tags with no clear value, reduce duplicates, or change when non-essential code loads. Test business events and real-user metrics afterward: async and defer avoid parser blocking but do not eliminate execution cost. - [URL parameters and SEO: which URLs should Google crawl and index?](https://crawlens.cc/blog/url-parameters-seo/): URL parameters are not automatically bad for SEO. The question is whether each variant deserves its own search result. Keep useful, distinct pages crawlable; consolidate tracking and sorting duplicates to a preferred URL. For large, unnecessary URL spaces, control crawling deliberately. Audit internal links, canonicals and Google-selected URLs before blocking anything, because a blocked page's canonical cannot be read. - [X-Robots-Tag: how to use the HTTP header for noindex and more](https://crawlens.cc/blog/x-robots-tag/): X-Robots-Tag is an HTTP response header that carries the same rules as the robots meta tag, such as noindex, nofollow and nosnippet. Use it for non-HTML files like PDFs and images, or to apply rules server-wide. It's invisible in the page source, so check headers when a page won't index, and remember robots.txt-blocked URLs never show it. - [How to make your website work for AI agents](https://crawlens.cc/blog/ai-agent-friendly-website/): AI agents use websites through screenshots, the HTML and the accessibility tree. web.dev recommends real buttons and links instead of clickable divs, labels tied to form fields, visible interactive elements, no overlays hiding controls, and stable layouts. Most of it is good accessibility practice, and much of it can be checked with a crawl. - [Technical SEO and AI search readiness: the complete audit checklist](https://crawlens.cc/blog/ai-search-readiness-checklist/): AI search runs on the same foundations as classic SEO: a page has to be crawlable, indexable and readable as text before any AI system can cite it. Audit which AI crawlers your robots.txt and CDN allow, whether your content exists without JavaScript, whether snippet controls block quoting, and whether logs show the bots actually visiting. - [How to track AI search traffic: ChatGPT, Perplexity and AI Overviews](https://crawlens.cc/blog/ai-search-traffic/): Track AI search in three places. Search Console's Generative AI performance report shows impressions in AI Overviews and AI Mode, whose clicks count as Organic Search. GA4's AI Assistant channel groups visits from ChatGPT, Gemini, Copilot and others. Server logs show AI crawlers fetching your pages. Visits without a referrer still land in Direct. - [Bing and Copilot SEO: sitemaps, IndexNow and AI answer controls](https://crawlens.cc/blog/bing-copilot-seo/): Bing's AI-powered search relies on its index, so the basics matter: an accurate XML sitemap with real lastmod dates, IndexNow to notify Bing of added, updated or deleted URLs, and Bingbot free to crawl. Bing's NOCACHE and NOARCHIVE tags control how content is used in its AI answers without removing it from search results. - [How to block AI crawlers without disappearing from AI search](https://crawlens.cc/blog/block-ai-crawlers/): AI companies run separate crawlers for search, model training and user-triggered fetches. To opt out of training but stay citable, disallow the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and allow the search ones (OAI-SearchBot, Claude-SearchBot, PerplexityBot). User-triggered fetchers may ignore robots.txt, and blocking Googlebot removes you from Google entirely. - [Canonical tag errors: how to find and fix them](https://crawlens.cc/blog/canonical-tag-errors/): A canonical tag tells Google which URL is the main version of a page. It breaks when it points to a redirect, an error or a noindex page, when a page declares more than one, or when it disagrees with redirects and sitemaps. Point every canonical at a final, 200, indexable URL, and keep one canonical per page. - [How to change your domain name without losing SEO](https://crawlens.cc/blog/change-domain-seo/): To change domains without losing search traffic, redirect every old URL with a server-side 301 to its exact equivalent on the new domain, verify both domains in Search Console and submit a Change of Address, and update internal links, canonicals, hreflang and sitemaps. Keep redirects and the old domain at least a year; expect a few weeks of fluctuation. - [How to appear in ChatGPT search: what OpenAI says and what to check](https://crawlens.cc/blog/chatgpt-search-seo/): OpenAI says any public website can appear in ChatGPT search. To be included in its summaries and snippets, don't block OAI-SearchBot, which is separate from the GPTBot training crawler. Keep content in the HTML, since most AI crawlers don't render JavaScript. Use noindex to opt out, and track visits through the utm_source=chatgpt.com parameter. - [How to appear in Claude web search: what Anthropic says](https://crawlens.cc/blog/claude-web-search-seo/): To be available as a source in Claude web search, let Anthropic's search and user-request bots access the pages you want read, serve useful content in the page HTML, and check that those requests succeed. Claude-SearchBot supports search results; Claude-User fetches pages for user questions. Anthropic does not promise that accessible pages will be cited. - [CMS migration SEO: how to switch platforms without losing traffic](https://crawlens.cc/blog/cms-migration-seo/): A CMS migration changes more than the design: URL formats, templates, titles and meta tags, canonicals, structured data, internal links and often how pages render. Crawl the old site as a benchmark, crawl the new one on staging and compare template by template, keep URLs where you can, redirect the rest page to page, and verify again after launch. - [Content decay: how to find pages losing traffic and refresh them](https://crawlens.cc/blog/content-decay/): Content decay is a page's gradual loss of search traffic as competitors publish, information ages or search intent shifts. Find it by comparing the last three months with the same period a year earlier in Search Console, grouped by page. Rule out technical changes and seasonality, then refresh the decaying pages with substantial updates. - [Core Web Vitals: what they are, how they affect SEO and how to improve them](https://crawlens.cc/blog/core-web-vitals/): Core Web Vitals are three metrics for real-user experience: Largest Contentful Paint (loading, good within 2.5 s), Interaction to Next Paint (responsiveness, 200 ms or less) and Cumulative Layout Shift (visual stability, 0.1 or less), judged at the 75th percentile of Chrome users. Google uses them in ranking, but relevance still comes first. - [How to check Core Web Vitals for every page on your site](https://crawlens.cc/blog/core-web-vitals-audit/): To check Core Web Vitals across a site, combine real-user data from the Chrome UX Report with Lighthouse lab tests. Judge pages on field data at the 75th percentile: good means LCP within 2.5 s, INP under 200 ms and CLS under 0.1. Use lab data to find the cause, then fix the templates behind the worst pages. - [Crawl budget: what it is, when it matters and how to stop wasting it](https://crawlens.cc/blog/crawl-budget/): Crawl budget is the set of URLs Google can and wants to crawl on your site. It only matters for large or fast-changing sites, roughly a million pages, or ten thousand that change daily. If yours is one, find where Googlebot's requests go in server logs, then cut the waste: parameters, redirects, errors, duplicates and slow responses. - [Crawled - currently not indexed: why it happens and how to fix it](https://crawlens.cc/blog/crawled-currently-not-indexed/): Crawled - currently not indexed means Google fetched the page and decided not to index it; Discovered - currently not indexed means it hasn't fetched it yet. The first usually points to duplicate, thin or weakly linked pages; the second to crawl capacity. Group the URLs by template, find the cause and fix the pattern; don't just request indexing. - [Crawling and indexing: how search engines find your pages, and what stops them](https://crawlens.cc/blog/crawling-indexing/): Crawling is a search engine fetching your page; indexing is it analysing the page and storing it so it can appear in results. A page must be discovered, allowed by robots.txt, return 200, carry no noindex and be the canonical version before it can be indexed. Audit both by crawling the site and comparing it with Search Console and logs. - [Cumulative Layout Shift (CLS): causes and how to fix them](https://crawlens.cc/blog/cumulative-layout-shift/): Cumulative Layout Shift measures how much visible content moves unexpectedly while people use a page; good is 0.1 or less at the 75th percentile. The usual causes are images and embeds without dimensions, ads and injected content without reserved space, and web fonts. Reserve space for everything that loads late, and animate with transforms. - [Custom extraction for SEO: 8 things to pull from every page](https://crawlens.cc/blog/custom-extraction-seo/): Custom extraction pulls any value from a page's HTML during a crawl, using a CSS selector or a regular expression: prices, authors, publish dates, analytics IDs, schema fields. The values become columns you can filter and export. Pair them with custom rules to flag pages automatically, such as indexable pages missing the analytics tag. - [E-commerce technical SEO: the complete guide for online stores](https://crawlens.cc/blog/ecommerce-technical-seo/): E-commerce technical SEO is about making a large, constantly changing catalogue crawlable and unambiguous: category, subcategory and product pages linked with plain links, facets and sort orders kept from multiplying URLs, one indexable URL per product, product structured data and a Merchant Center feed, and a clear policy for out-of-stock and discontinued products. - [Faceted navigation SEO: how to stop filters from wrecking your crawl](https://crawlens.cc/blog/faceted-navigation-seo/): Faceted navigation lets shoppers filter by size, colour, brand or price, but every combination can become a URL, creating near-infinite crawlable pages. Google recommends keeping filter parameters out of the crawl with robots.txt, and notes canonical and nofollow are less effective long term. Index only filtered pages that match real searches, and link to those. - [How to find broken links on your website (and fix them)](https://crawlens.cc/blog/find-broken-links/): To find broken links, crawl the whole site and check the status code of every URL each page links to. Fix internal links to 4xx and 5xx pages first, then pages that still get traffic or search clicks, then broken external links. Update the link itself rather than relying on redirects, and redirect only pages that moved. - [How to find duplicate and near-duplicate content on your site](https://crawlens.cc/blog/find-duplicate-content/): To find duplicate content, crawl the site and compare each page's text: identical text means exact duplicates, and a similarity fingerprint like simhash finds pages that are nearly the same. Duplicates aren't a penalty, but they split signals and let Google pick the version. Redirect or canonicalise true duplicates, and rewrite pages that should stand alone. - [How to appear in Google AI Overviews and AI Mode](https://crawlens.cc/blog/google-ai-overviews/): AI Overviews and AI Mode cite pages from Google's Search index, chosen by its core ranking systems. To be eligible a page must be indexed and allowed a snippet, and the site must not be opted out in Search Console. Google says what helps is original, useful content on a technically sound site, not llms.txt, chunking or special markup. - [How to use Google Search Console for SEO: a practical guide](https://crawlens.cc/blog/google-search-console-seo/): Google Search Console shows how Google crawls, indexes and ranks your site: clicks, impressions, CTR and position by query and page, plus indexing, Core Web Vitals and URL-level diagnostics. Use it to find pages close to page one, snippets that underperform, the cause of traffic drops, pages competing for one query and content losing ground. - [Hreflang errors: the 5 most common mistakes and how to fix them](https://crawlens.cc/blog/hreflang-errors/): Hreflang tells Google which language or regional version of a page to show each user. It breaks when codes are invalid (like en-UK), pages don't link back to each other, alternates redirect or are noindexed, a page doesn't list itself, or there's no x-default. Each version must list every version, including itself, with valid ISO codes. - [HTTPS and mixed content: how to find and fix insecure pages](https://crawlens.cc/blog/https-mixed-content/): A fully secure site serves every page over HTTPS, 301-redirects every HTTP URL to its HTTPS version, and loads every resource over HTTPS. Mixed content, HTTP resources on HTTPS pages, triggers browser warnings or gets blocked. Crawl the site to find HTTP pages, missing redirects and insecure images, then fix templates and hard-coded URLs. - [Image SEO audit: missing alt text and broken images](https://crawlens.cc/blog/image-seo-alt-text-audit/): An image SEO audit finds images with no alt attribute and images that fail to load. Google calls alt text the most important attribute for image metadata, and screen readers depend on it. Describe meaningful images in context, give decorative ones an empty alt, fix broken image URLs, and fix the templates that output them. - [How to improve organic CTR: find pages with a low click-through rate](https://crawlens.cc/blog/improve-organic-ctr/): To improve organic CTR, find pages that rank on page one but get far fewer clicks than typical for their position, then fix what searchers see: a title that matches the query, a specific meta description, and structured data for rich results. Check the queries first, because a low CTR is sometimes the wrong query, not the wrong snippet. - [Interaction to Next Paint (INP): how to find and fix slow interactions](https://crawlens.cc/blog/interaction-to-next-paint/): Interaction to Next Paint measures how long a page takes to visually respond to clicks, taps and key presses across a whole visit; good is 200 ms or less at the 75th percentile. Every interaction has an input delay, a processing duration and a presentation delay. Long JavaScript tasks, third-party scripts and large DOMs are the usual causes. - [Internal linking audit: click depth, weak pages and dead ends](https://crawlens.cc/blog/internal-linking-audit/): An internal linking audit checks how pages connect: how many clicks they are from the homepage, how many internal links point to them, and whether links are broken, nofollowed or missing. Pages you want to rank should be a few clicks from the start, linked from several relevant pages, and never only reachable through a sitemap. - [Keyword cannibalization: how to find it in Search Console and fix it](https://crawlens.cc/blog/keyword-cannibalization/): Keyword cannibalization is when several of your pages compete for the same query, so Google splits impressions between them or keeps switching which one ranks. Find it in Search Console by filtering a query and checking which pages get impressions over time. Fix it by merging pages with the same intent, differentiating pages with different intents, or consolidating duplicates. - [Largest Contentful Paint (LCP): how to find and fix a slow LCP](https://crawlens.cc/blog/largest-contentful-paint/): Largest Contentful Paint is the time until the largest visible image or text block renders; good is 2.5 seconds or less for 75% of visits. It splits into time to first byte, load delay, load duration and render delay. Most slow LCPs come from a late-discovered or lazy-loaded hero image, an oversized file, render-blocking code or a slow server. - [llms.txt: what it is, how to write one, and whether you need it](https://crawlens.cc/blog/llms-txt/): llms.txt is a proposed Markdown file at a site's root that gives AI agents a short, curated guide to the site's most useful pages. Google says Google Search, including AI Overviews, ignores it, so it won't help rankings. It's worth a few minutes for documentation, developer tools and APIs that people use with AI agents. - [Log file analysis for SEO: see what Googlebot really crawls](https://crawlens.cc/blog/log-file-analysis-seo/): Log file analysis means reading your web server's access logs to see exactly which URLs search engine bots request and what they get back. For SEO, verify which requests really come from Googlebot, then look for errors Googlebot hits, important pages it never crawls, and URLs it wastes time on. - [Noindex vs robots.txt: which to use, and the mistakes that hide pages](https://crawlens.cc/blog/noindex-vs-robots-txt/): Robots.txt tells crawlers which URLs they may fetch; noindex tells search engines not to index a page. They aren't interchangeable. A page blocked in robots.txt can still be indexed from links, and a noindex in a blocked page is never seen. To keep a page out of search, allow crawling and use noindex. - [Why did my organic traffic drop? How to find the cause](https://crawlens.cc/blog/organic-traffic-drop/): To diagnose an organic traffic drop, first confirm it in Search Console against the previous period and last year, then find where it's concentrated by page, query, device and country. Check for technical changes such as errors, noindex, redirects and robots.txt, then for Google updates, seasonality and falling demand. The shape of the drop usually points to the cause. - [Orphan pages: how to find them and what to do with them](https://crawlens.cc/blog/orphan-pages/): An orphan page is a page on your site that no other page links to. A normal crawl can't find it, so you need a second list of URLs to compare against: your XML sitemap, Search Console, Analytics or server logs. Then link to the orphans that should rank, and redirect or remove the rest. - [Out-of-stock and discontinued products: what to do for SEO](https://crawlens.cc/blog/out-of-stock-products-seo/): Keep temporarily out-of-stock product pages live and indexable, say so on the page and set availability to OutOfStock in structured data. For discontinued products, redirect to a close replacement if one exists, otherwise return 404 or 410 and remove internal links. Avoid redirecting everything to the homepage, which Google may treat as a soft 404. - [Pagination SEO: how to paginate category pages so Google finds every product](https://crawlens.cc/blog/pagination-seo/): Give every paginated page its own URL, such as ?page=2, link pages to each other with plain links, and give each page a self-referencing canonical rather than pointing them all at page one. Google no longer uses rel=next and rel=prev. Load more buttons and infinite scroll need crawlable links or a sitemap, because Google's crawlers don't click buttons. - [Product structured data: product snippets, merchant listings and how to check them](https://crawlens.cc/blog/product-structured-data/): Google uses Product structured data in two ways: product snippets for pages where people can't buy, such as reviews, and merchant listings for pages where they can. Merchant listings require name, image and an offer with price and currency, and recommend availability, shipping and returns. Google recommends pairing markup with a Merchant Center feed. - [Product variants and SEO: URLs, canonicals and structured data](https://crawlens.cc/blog/product-variants-seo/): Product variants can live on one URL, with options selected by parameters, or on separate URLs per variant. Google recommends a distinct URL that preselects each variant, the parameter-free URL as canonical for optional parameters, and ProductGroup structured data with a unique ID for every variant. Index separate variant pages only when people search for them. - [Redirect chains: how to find and fix them](https://crawlens.cc/blog/redirect-chains/): A redirect chain is a URL that redirects more than once before reaching the final page, like A → B → C. Google follows up to 10 hops, but every hop adds latency and room for error. Fix chains by pointing each old URL straight at the final destination with one 301, then updating internal links to the final URL. - [Redirect mapping for a site migration: how to build and test it](https://crawlens.cc/blog/redirect-mapping/): A redirect map lists every old URL and the new URL it should permanently redirect to. Build it from a full crawl plus every URL with traffic or links, map page to page rather than to the homepage, and decide deliberately what to do with pages that have no equivalent. Test it by crawling the old URL list after launch. - [Robots.txt for SEO: syntax, examples and the mistakes that block sites](https://crawlens.cc/blog/robots-txt/): Robots.txt is a text file at the root of a host that tells crawlers which URLs they may request. It controls crawling, not indexing, so it can't remove a page from Google. Each crawler follows the single most specific group that names it, the longest matching rule wins, and a robots.txt returning 5xx can stop Google crawling the site. - [Site migration SEO checklist: crawl before, during and after launch](https://crawlens.cc/blog/site-migration-seo-checklist/): A safe site migration is three crawls: one of the old site to benchmark every URL, one of the new site before launch to catch problems early, and repeated crawls after launch to verify redirects and spot regressions. Use permanent 301 redirects from every old URL straight to its new equivalent, keep them for at least a year, and watch pages with search traffic first. - [Soft 404 errors: what they are and how to fix them](https://crawlens.cc/blog/soft-404-errors/): A soft 404 is a page that tells visitors it doesn't exist, or has almost nothing on it, while returning a 200 status. Google reports these in Search Console and doesn't index them. Fix them by returning a real 404 or 410 for missing pages, adding real content to thin ones, and redirecting removed pages only to a genuine equivalent. - [Structured data errors: how to find and fix them across a site](https://crawlens.cc/blog/structured-data-errors/): Structured data errors come in two kinds: JSON-LD that won't parse, which Google ignores completely, and items missing properties Google requires, which makes them ineligible for rich results. Test templates with the Rich Results Test, crawl the site to catch every page, and fix the template that produced the error. - [Technical SEO audit: the complete guide and checklist](https://crawlens.cc/blog/technical-seo-audit/): A technical SEO audit checks whether search engines can crawl, render and index a site, and whether anything technical holds its pages back. Crawl the whole site, add Search Console and performance data, fix what blocks indexing first, then what wastes crawling or weakens pages, prioritising pages with traffic. Re-crawl to confirm, then audit on a schedule. - [Technical SEO audit report: what to include and how to present it](https://crawlens.cc/blog/technical-seo-audit-report/): A useful technical SEO audit report leads with a short summary and the few issues that matter most, ranked by impact on pages with traffic. For each issue it gives the affected URLs, why it matters, how to fix it and how to verify the fix. Re-crawl after fixes and report progress against the previous audit. - [Thin content in SEO: how to find it and what to do with it](https://crawlens.cc/blog/thin-content-seo/): Thin content is a page that gives searchers little or nothing of value: near-empty templates, tag pages, copied product descriptions, auto-generated pages. Google has no preferred word count, so use low word count to find candidates, then judge each page. Improve pages that should rank, merge overlapping ones, noindex utility pages and remove the rest. - [Time to First Byte (TTFB): what's good and how to reduce it](https://crawlens.cc/blog/time-to-first-byte/): Time to First Byte is the time from starting navigation until the first byte of the response arrives, including redirects, DNS, connection and server time. Good is 0.8 seconds or less. It isn't a Core Web Vital, but every loading metric, including LCP, waits for it. Reduce it with better hosting, caching, a CDN and no unnecessary redirects. - [Title tag and meta description audit: find and fix every problem](https://crawlens.cc/blog/title-tags-meta-descriptions-audit/): A title and meta description audit checks every indexable page for missing, duplicate, truncated or vague titles and descriptions, plus missing or multiple H1s. There's no hard length limit, but titles around 30–60 characters and descriptions around 70–160 usually display in full. Fix templates first, then rewrite pages that rank on page one but get few clicks. - [Traffic dropped after a site migration? How to find and fix the cause](https://crawlens.cc/blog/traffic-drop-after-migration/): Some fluctuation after a migration is normal, but a deep or lasting drop usually means old URLs don't redirect correctly, new pages aren't indexable, or content and links were lost in the new templates. Compare old and new URLs in Search Console, crawl the old URL list to test every redirect, and fix pages that had traffic first. - [XML sitemap audit: what to check and how to fix it](https://crawlens.cc/blog/xml-sitemap-audit/): An XML sitemap should list only canonical, indexable URLs that return 200. An audit checks that every sitemap can be read, flags URLs that redirect, error, are noindexed, canonicalised or blocked, and finds sitemap pages nothing links to. Crawl the sitemap alongside your internal links, then fix the generator rather than editing files by hand. - [How to use Claude for a technical SEO audit (with MCP)](https://crawlens.cc/blog/claude-technical-seo-audit-mcp/): To use Claude for a technical SEO audit, connect it to your crawl data instead of pasting exports. Crawlens includes a read-only MCP server: Claude can read the health score, issues, affected URLs and Search Console data, then prioritise fixes and write developer-ready explanations that name your actual URLs. - [Raw vs rendered HTML for SEO: what to compare and fix](https://crawlens.cc/blog/javascript-seo-raw-vs-rendered-html/): Raw HTML is what your server sends; rendered HTML is what exists after JavaScript runs. Google renders pages, but later, and many other crawlers never do. Compare the two to catch SEO tags changed by JavaScript, links and content that only exist after rendering, and pages that fail to render. - [Striking distance keywords: how to find your page-two wins](https://crawlens.cc/blog/striking-distance-keywords/): Striking distance pages rank on page two of Google, around positions 11–20, for queries that already bring impressions. To find them, filter Search Console to those positions, keep pages with at least 100 impressions, and remove any that aren't indexable in a fresh crawl. Then improve the title, internal links and content. ## Optional - [Crawlens services](https://crawlens.cc/services/): hands-on technical SEO and AI search audits, SEO-ready website development, managed SEO and analytics tracking support from the Crawlens team - [Technical SEO & AI search audit](https://crawlens.cc/services/technical-seo-audit/): full-site crawl plus AEO/GEO checks: AI crawler access, rendering, snippet controls and AI answer visibility, with prioritized fixes - [SEO-ready websites](https://crawlens.cc/services/seo-website-development/): website architecture, technical SEO, answer-ready content and launch QA - [Managed SEO](https://crawlens.cc/services/managed-seo/): technical priorities, search performance and an agreed ongoing roadmap - [Analytics & tracking](https://crawlens.cc/services/analytics-tracking/): measurement plans, GA4 and tag validation, and reporting handoff