Websites

What is llms.txt, and does your website need one?

llms.txt is a Markdown text file at yoursite.com/llms.txt that summarises a website for AI models and links to its key pages. Jeremy Howard proposed it in September 2024, and no major AI company has adopted it as a standard. Google says you don't need special AI text files to appear in AI Overviews or AI Mode, and a 2026 study of 83 sites found OpenAI's and Anthropic's crawlers requested it fewer than 10 times each in 12 weeks. It does no harm and takes little time, but it's no substitute for clear, indexed pages.

By MateiPublished · 6 min read
Contents
  1. Key points
  2. What llms.txt is
  3. What it looks like
  4. Do ChatGPT, Google or Claude use it?
  5. Where it is useful
  6. llms.txt, robots.txt and sitemaps
  7. Is it worth making one?
  8. How to make one
  9. Questions
  10. Sources

Key points

  • llms.txt was proposed by Jeremy Howard (Answer.AI) on 3 September 2024 as a summary of a website written for AI models.
  • Google says no new machine-readable files, AI text files or special markup are needed for AI Overviews and AI Mode.
  • In a study of 83 sites over 12 weeks, OpenAI's crawler fetched llms.txt 7 times and robots.txt 3,990 times.
  • The file is most useful for technical documentation that developers hand to AI tools; for a small business site it's optional.

Advice on "SEO for AI" often mentions a file called llms.txt, presented as the way to get ChatGPT to read your site. Below is what it is, what it looks like, what Google and the 2026 data say, and whether it's worth doing for a small business website.

What llms.txt is

llms.txt is a proposal published on 3 September 2024 by Jeremy Howard, co-founder of Answer.AI. The idea: an AI model looking for information on a site has to wade through long HTML pages full of menus, ads and scripts. A short file at a fixed address (/llms.txt) would hand it what matters: what the site is and where the key pages are, ideally in clean Markdown versions.

The format is plain Markdown:

  1. an H1 heading with the site's name, the only required part;
  2. a short summary as a blockquote;
  3. optionally, a few paragraphs of context;
  4. H2 sections with lists of links, each with a short note;
  5. optionally, an "Optional" section for less important links.

What it looks like

For a coffee shop, an llms.txt might look like this:

# Example Coffee

> Specialty coffee shop at 10 Example Street, Austin.
> Open daily 7am–7pm. Coffee, house-made pastries, breakfast.

## Pages

- [Menu](https://examplecoffee.com/menu): drinks and prices, updated monthly
- [Hours and location](https://examplecoffee.com/visit): map, parking, phone
- [Events](https://examplecoffee.com/events): tastings and workshops

## Optional

- [About us](https://examplecoffee.com/about): the shop's story

Do ChatGPT, Google or Claude use it?

In short: there's no evidence that the major AI assistants rely on it.

Crawler llms.txt requests robots.txt requests
GPTBot (OpenAI) 7 3,990
ClaudeBot (Anthropic) 9 3,120
PerplexityBot 0 775

The one exception in the study was Meta's crawler, which fetched the file more often. The authors' conclusion: the data doesn't support treating llms.txt as a reliable channel to any particular engine.

Where it is useful

llms.txt caught on where it was first aimed: technical documentation. Anthropic, Stripe, Vercel and Cloudflare publish llms.txt files for their docs. Developers hand those files to the AI coding tools they work with, so the tools read the right documentation. There, a person asks for the file explicitly; it isn't a crawler discovering it on its own.

llms.txt, robots.txt and sitemaps

File What it does Who uses it
robots.txt Tells crawlers what they may read Every major search engine and known AI crawlers
sitemap.xml Lists the site's pages with their last change date Google, Bing and other search engines
llms.txt Summarises the site and points to key pages Some AI tools, when someone gives it to them; crawlers rarely

The first two matter for every site. If an AI search crawler is blocked in robots.txt, no llms.txt will get you into its answers. How to check: How to show up in ChatGPT and AI Overviews.

Is it worth making one?

For a small business site, llms.txt is optional. It does no harm and takes half an hour, but the right order is different:

  1. The site is indexed in Google and Bing: Why is my website not showing up on Google.
  2. robots.txt doesn't block AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
  3. Every service page opens with the answer: what you do, where, at what price.
  4. Important information is text, not images: the menu, prices, hours.
  5. Your Google Business Profile and reviews are up to date: How to get Google reviews.
  6. Then, if you like, llms.txt.

How to make one

  1. Write the file in the format above: the name, a two-line summary, links to the key pages.
  2. Put it at the root of the site so it opens at yoursite.com/llms.txt.
  3. Keep it in step with your pages: a file that gives different hours from the site does more harm than not having one.
  4. If your site is generated from code, the file can be generated from the same data as the pages, so it never falls behind.

On manvro.com we have an llms.txt generated from the same data as the pages, so it changes when the site does. On the sites we build on our web design page, the list above comes first: indexed pages, clear text, structured data and a sitemap. Which structured data a local business needs is in our schema markup guide.

Questions

What is llms.txt?

A Markdown text file at the root of a website (yoursite.com/llms.txt) that briefly describes the site and lists links to its key pages, so an AI model can find information quickly. It's a 2024 proposal, not an official standard.

Does ChatGPT use llms.txt?

OpenAI hasn't said it does, and its crawler documentation only covers robots.txt. A 2026 study of 83 sites found OpenAI's crawler requested llms.txt 7 times in 12 weeks, against 3,990 requests for robots.txt.

Does llms.txt help with Google?

No. Google's documentation for AI features says you don't need new machine-readable files or AI text files, and Google's Gary Illyes said in 2025 that Google doesn't use llms.txt.

Can having an llms.txt hurt my site?

No. It's one more public file and doesn't change how Google reads your site. Just make sure it says the same as your pages and contains nothing you wouldn't publish on the site.

What's the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they may read and is respected by every major search engine. llms.txt only summarises the site and points to pages, with no access rules, and is rarely fetched.

Sources

  1. The /llms.txt file, Jeremy Howard, Answer.AI
  2. AI features and your website, Google Search Central
  3. Google says normal SEO works for ranking in AI Overviews and LLMS.txt won't be used, Search Engine Land
  4. llms.txt and AI Crawlers: Which Bots Actually Read It (2026), Something Inc.
  5. Overview of OpenAI crawlers, OpenAI

Related articles