Part of the AI Search series. For the full picture, start with Technical SEO and AI search readiness: the complete audit checklist.
Few files have been recommended as often, with as little evidence, as llms.txt. It’s described as “robots.txt for AI” (it isn’t) and as a ranking factor for AI search (Google says it isn’t one). It does have a real use, though, and it takes a few minutes to write.
This guide explains what the proposal says, what Google has said about it, where it actually helps, and how to write and maintain one.
What is llms.txt?
llms.txt is a proposal for a Markdown file at /llms.txt that gives AI agents a short, curated guide to a site. The proposal, published by Jeremy Howard in September 2024, describes it as “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website.”
The reasoning is practical: “Agents are best served by concise, expert-level information gathered in a single, accessible location”, instead of parsing full HTML pages with navigation, scripts and layout around the content.
It’s not an access control. Unlike robots.txt, llms.txt doesn’t allow or block anything; it only suggests what to read. To control which AI crawlers can access your site, see how to block AI crawlers.
What does an llms.txt file look like?
The proposal defines the sections in order:
- “An H1 with the name of the project or site. This is the only required section”
- “A blockquote with a short summary of the project, containing key information necessary for understanding the rest of the file”
- “Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information”
- “Zero or more markdown sections delimited by H2 headers, containing ‘file lists’ of URLs where further detail is available”
- In each file list, a Markdown link
[name](url), then optionally:and notes about the page
An ## Optional section marks links an agent can skip when it needs a shorter context. Here’s a shortened version of this site’s own file, crawlens.cc/llms.txt:
# Crawlens
> Crawlens is a free desktop app for Windows that crawls websites, up to
> 100,000 URLs per crawl, and audits technical SEO: 70 built-in checks,
> JavaScript rendering, Core Web Vitals, Google Search Console and GA4 data
> joined to every URL, server log analysis, and an MCP server so AI assistants
> can read the audits. Crawl data stays on the user's computer.
## Docs
- [Your first crawl](https://crawlens.cc/docs/first-crawl/): install, data folder, crawl modes and settings, health score, issues, exports
- [Audit rules reference](https://crawlens.cc/docs/audit-rules/): all 70 checks with severity, why each matters, how to fix it and what data it needs
## About
- [About Crawlens](https://crawlens.cc/about/): what it is, principles, founder
The proposal also suggests offering clean Markdown versions of important pages at the same URL “with .md appended (page.html.md) or with the extension replaced by .md (page.md).”
Does llms.txt help with Google Search or AI Overviews?
No. Google’s guide to optimizing for generative AI features lists llms.txt first among the things you can ignore:
You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.
It adds that creating one is fine: “Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Google may still crawl and index the file like any other text file, which “doesn’t mean that the file is treated in a special way.”
What about other AI search engines? The crawler documentation from OpenAI, Anthropic and Perplexity describes their search and training crawlers in terms of robots.txt, and none of it, at the time of writing, says those crawlers use llms.txt. Visibility in AI search still depends on the basics: crawlable, indexable pages with content in the HTML. See the AI search readiness checklist.
Where is llms.txt genuinely useful?
The proposal’s audience is agents working with a site’s content directly, not search crawlers. That’s most useful for:
- Documentation, APIs and developer tools. People use AI coding agents that read docs on the fly; a curated index helps them find the right page.
- Products with a lot of reference material, where a summary plus links saves an agent from wading through marketing pages.
- Sites that already publish Markdown versions of their pages, which the file can point to.
Some of the companies building AI models publish one for their own developer documentation, for example Anthropic and OpenAI.
For a typical brochure site or online shop, it’s a low-cost extra rather than a priority.
How do you write a good llms.txt?
- Start with the summary. One blockquote that says what the site or product is, who it’s for and any facts an agent must not get wrong (platform, pricing model, limits).
- Pick the pages, don’t dump the sitemap. Group the pages that answer real questions (getting started, reference, pricing, policies) under a few H2 sections.
- Describe each link. The notes after the colon are what an agent uses to choose; make them specific.
- Put secondary material under
## Optional. - Generate it, if you can. Hand-written files go stale. This site builds its llms.txt from the same data as its pages at every deploy, so version numbers and page lists can’t drift.
- Serve it as plain text at
/llms.txt, and make sure robots.txt or bot protection doesn’t block it.
How do you check an llms.txt?
- Every link should work. Paste the URLs from the file into a Crawlens URL list crawl: any link that returns a 4xx or 5xx, or redirects, shows up immediately.
- Every linked page should be indexable and current, the same pages you’d want people to land on.
- The summary should match the site. Re-read it after pricing, platform or product changes.
Checklist
- Decided whether llms.txt is worth it for this site (docs, APIs, developer tools: yes)
- File at
/llms.txt, served as plain text, not blocked - H1 with the site name and a blockquote summary with key facts
- Curated pages under a few H2 sections, each with a specific note
- Secondary links under
## Optional - Generated from site data, or reviewed after every content change
- All links return 200 and point at indexable pages
- No expectation that it affects Google rankings or AI Overviews
Frequently asked questions
What is llms.txt?
llms.txt is a proposed standard, published in September 2024, for a Markdown file at /llms.txt that helps AI agents use a website. It contains the site's name, a short summary and lists of links to the most useful pages, so an agent doesn't have to parse full HTML pages.
Does Google use llms.txt?
No. Google says you don't need AI text files or special markup to appear in Google Search, including its generative AI features, because Google Search doesn't use them. Creating one neither helps nor harms your visibility in Google Search.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which URLs they may request. llms.txt doesn't allow or block anything: it's a reading guide that points AI agents at the pages most worth reading.
Where should llms.txt go?
At the root of the site, at /llms.txt, served as plain text. The proposal also allows one under a subpath, such as /docs/llms.txt, for a section of a site.
Should every website have an llms.txt?
Not necessarily. It costs little, but it doesn't affect Google Search. It's most useful for documentation, APIs and developer tools, where people work with AI coding agents that read docs directly.