llms.txt: what it is, who reads it and how to write one

llms.txt is a markdown file that sits at the root of a website, at /llms.txt, and gives language models and agents a short, curated guide to the site. Jeremy Howard of Answer.AI proposed it in September 2024, and the specification at llmstxt.org is now on its second version. It is cheap to write and easy to get wrong, and the claims made about it range from useful to inflated. This guide covers the format, who reads it, what to expect from it and how to write one that helps.

What it is for

Web pages are built for people. A typical page wraps its information in navigation, ads and JavaScript, and turning that back into clean text is imprecise. Context windows have grown but are still too small for most sites, and every wasted token costs time and money. The proposal’s answer is a single file with brief background, guidance and links to the detailed pages. The file stays small enough to fit in context, and the detail lives behind the links, fetched only when needed.

It complements two older files rather than replacing them. robots.txt tells automated tools what access is acceptable. sitemap.xml lists every indexable page. llms.txt is a curated overview that an agent uses on demand, when it needs information about a topic while helping someone. The specification’s authors expected it to be useful mainly for inference rather than training.

The format

The specification asks for these parts, in this order:

• An optional byte-order mark.
• An H1 with the name of the project or site. This is the only required part.
• A blockquote with a short summary that holds the key facts needed to understand the rest.
• Zero or more markdown sections of any kind except headings, with more detail about how to read the files.
• Zero or more sections under H2 headings. Each holds a list of links written as [name](url), optionally followed by a colon and notes.

A section named Optional has a conventional meaning: it holds secondary links that an agent can skip when it needs a shorter context. The file can sit at the root or at any path, and it covers the URLs beneath it. Where more than one applies, agents should use the most specific. Here is a small example:

# Example Co

> Example Co sends invoices for freelancers and tracks payment.

## Product

- [Pricing](https://example.com/pricing): Plans and what each includes
- [Docs](https://example.com/docs): How to connect a bank account

## Optional

- [Privacy policy](https://example.com/privacy): How customer data is handled

The second version of the proposal also suggests offering clean markdown copies of pages, at the page’s URL with .md added, and pointing to them with standard link relations: rel="alternate" with type text/markdown for the copy, and rel="describedby" for the llms.txt that covers a page.

Who reads it

Here the claims need care. What can be said with some confidence:

• The specification page says thousands of sites publish one, that documentation platforms generate it automatically, and that the AI labs publish one for their own developer documentation. It lists Mintlify, GitBook, Yoast SEO, AIOSEO and Wix among the tools that generate it.
• Coding agents and assistants that are pointed at a documentation site can read it to find the right page quickly. That is where it is used most.

What we could not find is any answer engine documenting llms.txt as an input for choosing which sites to cite. Google’s guide to AI features in Search says there are no additional requirements to appear in AI Overviews or AI Mode beyond being indexed and eligible for a snippet, and it does not mention llms.txt. OpenAI’s crawler documentation names OAI-SearchBot and robots.txt as the controls for ChatGPT search, and llms.txt is not one of them.

How to write a good one

• Name and summary. Use the name people search for. Write one or two sentences on what the product is and who it is for. Facts, not slogans.
• Choose pages, do not list them all. Ten to thirty pages that explain the product: pricing, documentation, integrations, comparisons, policies. The file is a guide, not a sitemap.
• One plain line per link. Say what the page covers in a handful of words. Avoid marketing adjectives.
• Use Optional for the rest. Legal pages, press and other secondary links belong there.
• Link to pages that read cleanly. If you publish markdown copies, link those.
• Serve it properly. Return HTTP 200 at /llms.txt as plain text or markdown, with no HTML wrapper and no login.
• Keep it current. Update it when prices, products or key pages change.
• Test it. The specification suggests asking an agent questions about your content while giving it only the file as a starting point.

Common mistakes

• Listing every URL. That is what sitemap.xml is for.
• Linking pages that robots.txt blocks or that need a login.
• Serving an HTML page with a 200 status at /llms.txt, which looks like a file but is not one.
• Writing summaries that read as advertising instead of facts.
• Forgetting the www variant, so one host has the file and the other does not.

Start with a draft

The llms.txt generator reads your home page and the pages in your navigation and drafts the file for you. Review the wording before you publish, then use the AI crawler check to confirm that it is found at the right address. To find out whether engines mention you at all, run a scan, and read what AEO and GEO mean for the wider picture.