New researchTechnical SEO & AI Search · 2027Explore the framework

How to appear in Claude web search: what Anthropic says

How Claude finds and cites web pages, what its search and user-fetch bots do, and how to check whether your site is accessible and useful as a source.

Part of the AI Search series. For the full picture, start with Technical SEO and AI search readiness: the complete audit checklist.

Claude can search the live web when a question needs current information and show links to the sources it used. That creates a simple question for site owners: can Claude reach the page that answers the question, and can it read the answer once it gets there? Anthropic documents how Claude web search works and which of its web bots do which jobs.

This guide is about your site’s availability as a source in Claude’s web answers. It is separate from using Claude with Crawlens over MCP, where Claude reads a private audit you chose to connect. For the full cross-platform audit, see the AI search readiness checklist.

Make the page accessible to Anthropic’s search and user-request bots, give it a clear answer that Claude can retrieve, and keep the page working when those bots visit. Anthropic says Claude may search the live web, process multiple sources and include direct citations and source links in its response. It does not say that allowing a bot or adding a particular tag guarantees a citation.

There are two possible moments when your site is read: Anthropic’s search bot may visit as it improves search results, and its user-request bot may fetch a page in response to a person’s question. Treat a successful bot request as evidence of access, not evidence that your page was chosen as a source.

Which Claude bot should you allow?

Anthropic lists three different bots. They have different consequences when blocked:

Bot Anthropic’s stated purpose If you block it
Claude-SearchBot Navigates the web to improve the relevance and accuracy of search responses Anthropic says this may reduce your visibility and accuracy in user search results
Claude-User Retrieves content when a person asks Claude a question Anthropic says Claude cannot retrieve that content in response to the request, which may reduce user-directed search visibility
ClaudeBot Collects web content that could contribute to model training Blocking it signals that future material should be excluded from Anthropic’s training datasets

Decide separately whether you want search access, user-request access and training. You can permit the first two while disallowing ClaudeBot in robots.txt. Copying a blanket “block AI” rule can close off all three. The AI crawler policy guide covers the full set of companies and example rules.

Also inspect your CDN and firewall. A permissive robots.txt file does not help when bot protection returns a challenge or 403 response. Anthropic publishes an IP list for its bots; use it to investigate real log entries rather than trusting a user-agent string alone. A local test that merely sets its user agent to Claude-SearchBot cannot prove how IP-based protection will treat Anthropic’s actual bot.

What can Claude read on the page?

The safest starting point is the page a normal visitor receives: it should contain the answer in text, a descriptive heading, and links to supporting detail. If the initial HTML is an empty shell and a script inserts the main content later, test what a fetcher receives before assuming Claude can use it. The raw versus rendered HTML guide explains how to compare the two.

Write for a reader who wants to verify an answer. For example, a support page titled “Can I return a refurbished laptop?” should answer that question near the top, state the relevant time limit and conditions, and link to the current policy. A vague heading followed by “contact us for details” offers little that Claude can cite. This is a content-quality judgment, not a documented Claude ranking factor.

Keep the URL stable and the facts current. When a policy changes, update the source page rather than leaving several conflicting versions online. Claude’s web-search help says it processes multiple sources and lets users open them; a citation is useful only if the destination still supports the claim. The same page notes that web search may vary with connectivity and that website links occasionally fail, so even a correct page will not appear in every answer.

How do you check whether Claude can access your pages?

Use the site’s configuration, a crawl and actual server logs as separate checks:

  1. Read robots.txt for each Anthropic bot. Check Claude-SearchBot, Claude-User and ClaudeBot as separate user agents. Anthropic says its bots honor robots.txt directives; make sure the policy matches your intent.
  2. Fetch a representative page. Confirm the URL returns the intended page with a 200 response, not a login, error, redirect loop or bot challenge. Crawlens can surface client errors, server errors and pages that did not respond across the site.
  3. Compare raw and rendered content. If the answer exists only after JavaScript runs, inspect the initial HTML. Crawlens flags content that depends on JavaScript when rendering is enabled for the crawl. That flag identifies pages to review; it does not measure Claude’s own rendering behavior.
  4. Inspect access logs. Look for requests from Claude-SearchBot and Claude-User, the requested URLs, status codes and response sizes. Verify suspicious traffic against Anthropic’s published IP list. Crawlens’s log analysis can group AI-crawler requests; inspect raw log lines when you need a specific bot.

If logs show no requests, do not conclude that Claude has rejected the site. It may not have selected those pages to fetch, and a CDN may serve requests without passing them to the origin log. If logs show repeated 403, 404 or 5xx responses, you have a concrete access problem to fix.

How can you tell whether Claude cited you?

Run a small, repeatable set of real questions where your page should be a useful source, then inspect Claude’s cited links. Record the question, answer date, cited URL and whether the linked page actually supports the statement. Anthropic says Claude’s web answers include direct citations and source links; checking those links is more meaningful than counting bot requests.

Do not treat a single prompt as a ranking report. Claude may choose different sources as the question, available information or connection changes. Server logs measure fetches, while analytics can measure visits after someone clicks a source link. Neither alone counts every mention or citation. The AI search traffic guide explains those measurement limits in more detail.

Claude web search checklist

  • Claude-SearchBot access reflects your search-visibility choice.
  • Claude-User can retrieve pages you want Claude to read for user questions.
  • ClaudeBot training access is a separate, deliberate decision.
  • CDN and firewall return the real page to permitted Anthropic bots.
  • Important pages return 200 and contain the answer in accessible text.
  • Raw HTML is checked when key content depends on JavaScript.
  • Logs are reviewed for actual bot requests and response errors.
  • Claude’s cited links are checked against the page’s current facts.

The useful outcome is a page that readers and Claude can both verify. Give the search and request bots the access you intend, keep the answer clear and current, and use citations and logs to see what happened rather than assuming that a successful crawl means visibility.

Frequently asked questions

How do I get my website cited in Claude web search?

Make the relevant pages accessible to Claude's search and user-request bots, answer the question clearly on the page, and verify that the server returns the intended content. Anthropic documents how Claude searches and cites sources, but it does not offer a way to submit a page for guaranteed citation.

Which Anthropic bot matters for Claude search?

Claude-SearchBot navigates the web to improve search result quality; Claude-User can retrieve a page when a user asks a question or gives Claude a URL. ClaudeBot is for potential model training, a separate purpose.

Can I block ClaudeBot and still appear in Claude search?

Yes, those bots serve different purposes. Anthropic says blocking ClaudeBot signals an opt-out from future training data, while blocking Claude-SearchBot may reduce search visibility and blocking Claude-User prevents retrieval in response to a user query. Check the exact rules you deploy for each bot.

Does Claude need an llms.txt file?

Anthropic's published web-search and crawler guidance does not list llms.txt as a requirement for being cited. A readable page and access for the relevant bots are more direct checks; use llms.txt only if it serves another documented use for your site.

Can server logs prove that Claude cited my page?

No. Logs can show that an Anthropic bot requested a URL and what your server returned, but a fetch is not a citation. Check the answer's source links for a citation and use analytics for visits from readers who follow those links.

· Founder, Crawlens

Dien builds Crawlens, a desktop crawler for technical SEO audits, and writes about the checks it runs: crawling, indexing, rendering and Search Console data.

Audit your own site with Crawlens

Crawl, run 70 checks, and see them next to Search Console and Core Web Vitals data.

Download free for Windows