Your first crawl

From install to a prioritised list of issues in about ten minutes. You'll crawl a site, read its health score, open the issues that matter and export a report.

1. Install Crawlens

Crawlens runs on Windows. Download the installer (113.7 MB) and run it. You can choose the install folder during setup.

If Windows SmartScreen shows Windows protected your PC, choose More info, then Run anyway.

2. Choose a data folder

On first launch, Crawlens asks where to keep your projects and crawl data. Large sites can produce several GB, so pick a drive with room to spare.

• Keep the Default location, or Choose a folder on any local or external drive (for example D:\Crawlens).
• Network drives aren't supported: the database can be corrupted over a network connection.
• You can move it later in Settings → About → Data folder.

3. Create a project

Each website gets its own project, which keeps its crawls, issues and reports together.

1. In Projects, click New project.
2. Enter the Website URL: the homepage, or the section you want to audit. Crawls start here by default.
3. Give it a Project name and click Create project.

4. Start a crawl

Open the project and click Start crawl (or press Ctrl+N). Pick a Crawl mode:

For a first crawl, keep the defaults: URL limit 100,000, no max depth. A few settings worth knowing (defaults come from Settings):

• Render JavaScript: turn it on for React, Vue or other client-rendered sites. Each page is loaded in Chromium and compared with the raw HTML. Several times slower.
• Check images (on by default) and Check external links (off by default): after the crawl, request every internal image or external link to find broken ones.
• Respect robots.txt: skips URLs disallowed for the selected user agent. On by default.
• Concurrent requests and Delay between requests: lower them for small or fragile servers.
• Advanced settings: Include only URLs matching / Exclude URLs matching (one regular expression per line) and Crawl subdomains.

Click Start crawl.

5. Watch it run

The live panel shows URLs crawled and queued, speed in URLs per second, and a breakdown by response: 2xx Success, 3xx Redirect, 4xx Client error, 5xx Server error, No response and Blocked by robots.txt.

• Pause and Resume at any time, or Stop to keep what's been crawled so far.
• If Crawlens closes mid-crawl, the project shows Resume crawl next time. It carries on where it left off; pages already crawled aren't fetched again.

When crawling ends, Crawlens runs the audit automatically (Running audit…).

6. Read the audit

The project Overview shows the latest crawl:

• Health score (0–100): Good at 90+, Needs work at 70–89, Poor below 70. It's based on the share of URLs with critical issues and warnings; see how it's calculated.
• Issues found, split into critical, warnings and notices.
• Top issues, most severe first. Start with the critical ones.

Below the issues, a summary shows how many checks passed and how many were not available for this crawl. Click Show why: some of the 70 checks need data a basic crawl doesn't collect, like JavaScript rendering, Search Console or server logs. The audit rules reference lists what each check needs.

7. Explore issues and URLs

• Issues (Ctrl+2): every failing check with why it matters, how to fix it and the affected URLs.
• URLs (Ctrl+3): every crawled URL. Filter by status, indexability, type or issue, search by URL or title, and choose columns.
• URL details: click any row to see its SEO fields, issues, links in and out, redirects, headers and structured data. Use the arrow keys to move through rows.

8. Export and share

• Export report → Excel workbook (Ctrl+E): a summary, all URLs, and every issue with its affected URLs.
• Export report → PDF report: a client-ready report with an executive summary, progress since the previous crawl, issues and how to fix them, Core Web Vitals and methodology. Add your logo and colours in Settings for white-label reports.
• Export CSV on the URLs table or an issue: exactly the rows you're looking at.

Next steps

• Add Google data. Connect Google in Settings, then fetch Search Console and Analytics data on the project overview to see which issues affect pages with traffic.
• Measure Core Web Vitals. Add a PageSpeed Insights API key in Settings, then use Measure Core Web Vitals on the overview.
• Import server logs on the Log files page to see what Googlebot really crawls. Read log file analysis for SEO.
• Schedule crawls in Schedule & alerts and get an email when the health score drops.
• Compare crawls after fixes ship, to see what got better and what got worse.
• Ask your AI assistant about the audit: connect Claude, Codex, Cursor or VS Code.

Keyboard shortcuts