What is llms.txt, and does your website need one?

Short answer: llms.txt is a Markdown file, usually placed at /llms.txt, that gives AI agents a short, curated overview of a website and links to its most important pages. Jeremy Howard proposed it in September 2024 and revised it in August 2026. It has not been adopted as an official web standard. Google says Google Search ignores the file, and the crawler documentation from OpenAI, Anthropic and Perplexity does not name it as something that affects their answers. It is cheap to add and most useful where coding agents read documentation. For an ordinary business website it is optional.
What is llms.txt?
llms.txt is a plain text file written in Markdown that tells an AI agent what a website is about and where its most useful pages are.
The proposal comes from Jeremy Howard, who published it on 3 September 2024. A second version followed in August 2026 (llmstxt.org). The page still describes itself as a proposal. No standards body has adopted it, and nothing obliges an AI service to read it.
The reasoning is practical. Web pages are built for people, with the information wrapped in navigation, ads and JavaScript, and an agent's context window is too small for most whole websites. A short file that points to the pages that matter saves the agent the guessing.
According to the proposal, agents view or search the file and then follow the relevant links. It is used most heavily for software documentation, and the same structure can work for a business outlining its structure and policies.
What does an llms.txt file look like?
An llms.txt file is a short Markdown document with a fixed order of parts. The specification lists them like this:
- An H1 with the name of the project or site. This is the only required part.
- A blockquote with a short summary containing what a reader needs in order to understand the rest of the file.
- Free text, such as paragraphs or lists, with more detail. Any Markdown is allowed here except headings.
- H2 sections with link lists. Each item is a Markdown link, optionally followed by a colon and a note about the page.
Here is an llms.txt example for a fictional heating company:
# Example Heating Ltd
> Example Heating Ltd installs and services heat pumps and boilers for homes in the Northfield area. Quotes are fixed after a home survey.
Pages marked "guide" give general advice and do not replace a survey.
## Services
- [Heat pump installation](https://www.example.com/services/heat-pumps.md): What is included, the usual timeline and how quotes work
- [Boiler servicing](https://www.example.com/services/boiler-service.md): What an annual service covers
## Company
- [About us](https://www.example.com/about.md): Who we are, certifications and service area
- [Contact](https://www.example.com/contact.md): Phone, email and opening hours
## Optional
- [Guide: choosing a heat pump](https://www.example.com/guides/choosing-a-heat-pump.md)- The "Optional" heading has a meaning. By convention it holds secondary links that an agent can skip when it needs a shorter context.
- The links should lead to text an agent can read easily. The proposal recommends offering a clean Markdown version of important pages at the same address with .md added, which is why the example links end in .md. If your site cannot serve Markdown versions, link to the normal pages.
- The file can sit in a subfolder. A file at /docs/llms.txt covers the pages under /docs/.
llms.txt vs robots.txt vs sitemap.xml: what is the difference?
The three files answer different questions. robots.txt says what automated tools may access, sitemap.xml lists the pages a search engine can index, and llms.txt is a curated summary with links for an agent that is looking something up. None of them replaces another.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | States which crawlers may access which parts of the site | Lists the indexable pages for search engines | Gives agents a short overview and links to key pages |
| Format | Plain text rules | XML | Markdown |
| Status | Established and documented by Google, OpenAI, Anthropic and Perplexity | Established and used by search engines | A proposal |
The proposal draws the same line: robots.txt tells automated tools what access is acceptable, while llms.txt is read on demand when an agent needs information while assisting a user. A sitemap is no substitute, because it usually lists far more than fits in a context window.
One practical consequence: you cannot use llms.txt to block or allow an AI crawler. That is still done in robots.txt.
Do Google, ChatGPT and other AI tools use llms.txt?
Google Search does not, by its own account. For ChatGPT, Claude and Perplexity we found no first-party statement that the assistants use a site's llms.txt when they answer. The documented users are mainly coding agents and documentation tools. We checked the sources below on 17 November 2026.
Google Search, AI Overviews and AI Mode
Google's guidance is explicit. Its guide to generative AI features in Search says you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI capabilities, because Google Search does not use them. It adds that it is fine to maintain llms.txt files for other services that use them, and that doing so will neither harm nor help visibility or rankings in Google Search (Google Search Central).
The page on AI features says the same from the other direction: there are no additional requirements for appearing in AI Overviews or AI Mode. To be shown as a supporting link, a page must be indexed and eligible to appear in Search with a snippet (Google Search Central).
Chrome Lighthouse
Another Google product does look at the file. Lighthouse, the audit tool in Chrome, has an Agentic Browsing category that checks whether a machine-readable summary exists at the domain root. The Chrome documentation calls the category experimental and based on proposed standards, and it does not connect the check to Google Search (Chrome for Developers). A pass there tells you the file exists and nothing about rankings.
ChatGPT, Claude and Perplexity
OpenAI, Anthropic and Perplexity each document the bots they run and how to allow or block them in robots.txt. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers (OpenAI). Anthropic says its bots honour robots.txt and that blocking Claude-SearchBot or Claude-User may reduce a site's visibility (Anthropic). Perplexity recommends allowing PerplexityBot in robots.txt (Perplexity).
None of these pages says the company's assistant fetches a website's llms.txt to answer a question, or that having one changes whether a site is cited. Anthropic's page does not mention the file. OpenAI's and Perplexity's pages mention it only as a link to the index of their own documentation.
OpenAI, Anthropic, Perplexity and Google's Gemini API all publish an llms.txt for their developer documentation. That shows the labs find the format useful for their own docs. It does not show that their chat products look for yours.
Coding agents and documentation tools
This is where the proposal reports real use. Its author writes that coding agents use the files reliably and that documentation platforms generate one automatically (llmstxt.org, changes in v2).
The summary: there is no evidence from the companies behind the big assistants that llms.txt gets a business mentioned or cited more often, and Google says outright that it has no effect in Search.
How do you create an llms.txt file?
You create an llms.txt file by writing a short Markdown document by hand or letting your CMS generate one, and publishing it at the root of your domain.
- Pick the pages. Choose the few pages a stranger would need in order to understand what you sell, to whom, where and on what terms. Leave out the rest.
- Write the H1 and the summary. Use your business name as the H1. In the blockquote, say in one or two sentences what you do and for whom.
- Group the links under H2 headings. Add a short note after each link that says what the page answers. Put nice-to-have material under "Optional".
- Save it as llms.txt and upload it to the root so it opens at yourdomain.com/llms.txt. Check in a browser that you get the text file and not an error page or your home page.
- Test it. The proposal suggests giving an agent only your llms.txt as a starting point and asking it questions about your content.
- Keep it current. A file that points to removed pages or old prices is worse than no file.
Using an llms.txt generator
Many platforms now do the work for you. The proposal's own list of integrations includes the WordPress plugins Yoast SEO and AIOSEO, Wix, which generates the file for every Wix site, and the documentation platforms Mintlify and GitBook. Check your plugin or platform settings before writing anything by hand.
Whatever generator you use, read the result. A file that lists every URL on the site repeats the sitemap and loses the benefit of a curated overview.
Is llms.txt worth adding to a business website?
For most business websites llms.txt is worth adding only if it costs you almost nothing. It will not hurt, and it has no documented effect on being recommended by AI assistants.
It makes sense when:
- your CMS or SEO plugin generates the file and you only need to review it
- you publish technical documentation, an API or detailed product manuals that coding agents and other tools are likely to read.
It is not worth prioritising when:
- you would pay for it as a separate project while robots.txt blocks AI crawlers or key pages are not indexed
- someone promises that the file will get you into ChatGPT or Google's AI answers. No source we could open supports that promise.
Our position is the same as in the article GEO vs SEO: treat it as optional housekeeping.
What matters more for AI visibility than llms.txt?
The things with documented effect are crawl access, indexable pages and content that answers real questions. Start there.
- Let the search bots in. Check robots.txt for OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot. OpenAI says a robots.txt change takes about 24 hours to reach its search systems.
- Make sure important pages are indexed. Google requires a page to be indexed and eligible for a snippet before it can be a supporting link in AI Overviews or AI Mode.
- Say plainly what you offer. Each service page should state what you do, for whom and where.
- Keep your facts consistent. Your name, services and contact details should match on your own site, in directories and in reviews.
- Measure. Ask the main assistants the questions your customers ask and note whether you are mentioned. Our AI check gives a quick first snapshot.
How we do this as ongoing work is described on our page about generative engine optimization.








