“Richard and the crew from Pinecone Agency were incredibly accommodating in our very particular needsIncredibly accommodating”
Tim Zafir

8 min read Updated 11 November 2026

What is llms.txt, and does your website need one?

llms.txt is a short guide to your site written for AI tools. Here's what it does, who reads it, and whether it's worth the hour it takes.

Want help with your project? Book a discovery call
On this page
  1. What is llms.txt?
  2. Which AI companies actually read llms.txt?
  3. What does llms.txt do, and what doesn't it do?
  4. How is llms.txt different from robots.txt and sitemap.xml?
  5. What does an llms.txt file look like?
  6. How do you add llms.txt to a Webflow site?
  7. How do you add llms.txt to a custom site?
  8. What we've seen on the sites we look after
  9. Is llms.txt worth doing?
  10. Frequently asked questions
  11. Sources

The short answer: llms.txt is a plain-text Markdown file at yoursite.com/llms.txt that gives AI tools a short, curated guide to your business and key pages. Jeremy Howard proposed it in September 2024, and it isn't an official standard. Google says Search ignores it, and no major AI company says its crawlers read yours.

Key takeaways

  • Google says llms.txt "will neither harm nor help" your visibility in Google Search.
  • OpenAI, Anthropic and Google publish llms.txt files for their own developer docs. None of them, or Perplexity, says its crawlers read the one on your site.
  • Treat it as a quick tidy-up, not a ranking tactic. It's a four-step upload on Webflow and one file on a custom site. Worth an hour, not a fee.

What is llms.txt?

It's a text file, written in Markdown, that tells an AI tool what your business is and where the useful pages are. It usually sits at the root of your site, though it can also sit under a path, such as /docs/llms.txt. Think of it as a one-page brief for a machine.

Jeremy Howard proposed it on llmstxt.org and the Answer.AI blog on 3 September 2024, and updated it to a second version on 10 August 2026. His reasoning: web pages are wrapped in menus and scripts built for people, and an AI tool has limited room to read them. A short file written for the tool saves it the work.

The format is loose but has an order:

  • A heading with your business or site name. This is the only required part.
  • A one-paragraph summary in a quote block.
  • Sections of links, each with a short note on what the page covers. A section called "Optional" holds links a tool can skip.

Plenty of sites have taken it up. The v2 update on llmstxt.org says "thousands of sites publish an llms.txt file, documentation platforms generate one automatically". That tells you how many people write the file. It doesn't tell you who reads it.

Which AI companies actually read llms.txt?

None of Google, OpenAI, Anthropic or Perplexity has said its AI reads llms.txt on your site. That's where things stand on 1 November 2026.

Here's what each says in its own documentation:

  • Google. Its guide to generative AI features in Search lists llms.txt under things you can ignore: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It adds that keeping one for other services is fine and won't harm or help your rankings. Google's John Mueller was blunter on the Search Off the Record podcast on 1 October 2026: "I would not rely on it. I would not treat that as a priority."
  • OpenAI. Its crawler page explains GPTBot, OAI-SearchBot and ChatGPT-User, and how to control each one in robots.txt. It doesn't say any of them reads llms.txt.
  • Anthropic. Its help article on ClaudeBot, Claude-User and Claude-SearchBot explains robots.txt controls and doesn't mention llms.txt.
  • Perplexity. Its crawler page covers PerplexityBot and Perplexity-User through robots.txt. The only llms.txt on that page is a link to Perplexity's own documentation index.

Don't confuse publishing with reading. OpenAI, Anthropic and Google all publish llms.txt files for their developer docs. That shows they use the format. It doesn't show their crawlers read yours.

One Google product does check for the file. Chrome's Lighthouse has an experimental "Agentic Browsing" category that looks for it. That's about AI agents using a site in a browser, not rankings or citations.

What does llms.txt do, and what doesn't it do?

It gives any tool that chooses to look a short, accurate summary of your business and the pages that matter. It's used most for software documentation, where coding assistants follow it.

It doesn't:

  • block or allow AI crawlers. That's robots.txt.
  • change your Google rankings, by Google's own account.
  • guarantee your business gets cited in ChatGPT, Gemini or Perplexity.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt sets rules, sitemap.xml lists your pages, and llms.txt explains the few that matter.

FileWho it's forWhat it doesStatus
robots.txtCrawlers, including AI crawlersTells them which parts of the site they may crawlInternet standard (RFC 9309, 2022)
sitemap.xmlSearch enginesLists the URLs you want crawled, up to 50,000 per fileProtocol published at sitemaps.org
llms.txtLLMs and AI agentsSummarises the site and links to key pagesProposal, no one is required to read it

Even robots.txt doesn't lock anything. The standard says its rules "are not a form of access authorization". Crawlers are asked to follow them.

What does an llms.txt file look like?

Here's an llms.txt example for a made-up business. Example Plumbing and example.com aren't real. It's only there to show the shape.

# Example Plumbing

> Example Plumbing is a home plumbing business in Melbourne. It fixes hot water systems, leaks and blocked drains for households.

## Services

- [Hot water](https://www.example.com/hot-water): Repairs and replacements, with what affects the price.
- [Blocked drains](https://www.example.com/blocked-drains): What gets cleared, and how long a visit takes.

## About

- [Contact](https://www.example.com/contact): Phone, service area and opening hours.

## Optional

- [Blog](https://www.example.com/blog): Home maintenance guides.

A few rules make a file useful:

  • Describe what you do in plain words. Webflow's own advice is to write it as if you're briefing a new starter, not pasting your homepage slogan.
  • Link only to live, public pages. Leave out anything internal or private.
  • Keep it current. A file that lists an old price or a page that's gone gives an AI tool the wrong facts. Give it an owner, the same person who updates those pages.
  • Test it. llmstxt.org suggests asking an AI agent questions about your business with only your llms.txt as a starting point.

How do you add llms.txt to a Webflow site?

Webflow has a setting for it. As of its help article updated in August 2025:

  1. Go to Site settings, then SEO, then LLMs.txt.
  2. Click Upload file.
  3. Choose your llms.txt file. It must be UTF-8 encoded and under 100 KB.
  4. Click Save changes, then publish the site.

The file then appears at yourdomain.com/llms.txt. It's served on your custom domain only, not your webflow.io staging domain, so check it after you publish.

Still weighing Webflow against a custom build? We cover that in choosing a CMS for the team that uses it.

How do you add llms.txt to a custom site?

Put the file in the folder your framework serves as is, then deploy. On Astro, that's the public folder. Astro copies everything in it into the build untouched, so public/llms.txt becomes yoursite.com/llms.txt. It works the same whether the site is hosted on Cloudflare or Vercel.

To check it, open yoursite.com/llms.txt in a browser. You should see plain text, not your 404 page.

What we've seen on the sites we look after

On Pinecone Agency's own site, AI bots visit often and don't ask for the file. Cloudflare's analytics for 12 October to 10 November 2026 estimate 4,470 requests from bots it classes as AI crawlers, AI search or AI assistants, such as ClaudeBot, Applebot and ChatGPT-User. About 460 of those were for robots.txt. Of an estimated 170 requests for /llms.txt in the same period, none came from those bots. Most were our own checks and SEO audit tools such as Semrush and Lighthouse. (Cloudflare samples this data at about 1 in 10, so a rare visit could be missed.)

We also checked the client sites we look after. On 11 November 2026, 9 of the 21 live sites we checked, ours included, had an llms.txt. Google Analytics for 12 of them, from 1 May to 10 November 2026, puts visits from AI assistants such as ChatGPT, Perplexity and Gemini at 0.38% of sessions on the 7 sites with a file and 0.56% on the 5 without. Twelve very different sites can't prove the file does nothing, but we saw no lift from having one, which is what the larger studies found too.

Is llms.txt worth doing?

Yes, if it takes you an hour and someone will keep it up to date. It's low cost, it can't hurt your Google rankings, and it gives AI agents and developer tools an accurate summary if they look.

It's not worth paying for as an "AI visibility" service, and it won't fix a site that AI search can't read. Google's advice for its AI features is to focus on crawlable, well-structured, people-first content. In practice that means pages that answer buyers' questions clearly, a robots.txt that doesn't block the AI search crawlers you want to be found by, and a way to measure whether you're being cited. We cover that last part in answer engine optimisation: how to tell if it's working.

If you run a software product with documentation, the case is stronger, because that's where coding assistants use these files.

That's also where the better evidence is. Mintlify says /llms.txt was fetched about 10.8 million times in August 2026 across the documentation sites it hosts. In its test of 2,400 agent tasks on 20 documentation sites, agents averaged 2.23 failed requests per task on HTML pages, 1.42 on Markdown and 0.11 on Markdown with an llms.txt map (Mintlify, The 2026 State of Knowledge Report). That helps an agent find its way around a docs site. It doesn't show the file wins citations for a business site. Google's John Mueller has said much the same: it may help an agent that's already on your site, not get you chosen in the first place.

Frequently asked questions

Does ChatGPT read llms.txt?

OpenAI hasn't said it does. Its crawler page covers GPTBot, OAI-SearchBot and ChatGPT-User and how to control them in robots.txt, but it doesn't say any of them reads llms.txt on other people's sites.

Does Google use llms.txt?

Not for Search. Google's guide says Google Search ignores llms.txt files, and that having one "will neither harm nor help" your visibility or rankings. Search doesn't use the file for ranking, but Google can still index it like any other URL. Its guide says Google "may discover, crawl, and index many kinds of files" beyond HTML. In June 2026 Cyrus Shepard posted screenshots of llms.txt files showing up in Google results, and of a Google AI answer that quoted one company's llms.txt and cited it as the source (the question named the file). So if you keep one, make sure it's accurate and matches your pages. Chrome's Lighthouse does look for the file, but only in an experimental audit for AI agents that browse sites.

Is llms.txt supported by major AI platforms?

Not as something they read on your site, as of November 2026. OpenAI, Anthropic and Perplexity document their crawlers through robots.txt and don't say they read llms.txt on other sites. Google says Search ignores it. Several of them publish llms.txt files for their own developer docs.

Do AI crawlers actually request llms.txt?

Hardly ever on a typical business site. Ahrefs checked the May 2026 traffic of 137,210 domains. Of the roughly 38,000 with a valid llms.txt, "97% saw no requests for it whatsoever". Of the fetches the other 3% did get, 19.5% came from named AI tools, led by GPTBot and Claude-Code. The SEO Framework logged 180,219 AI bot hits on its sites between 31 October 2025 and 10 April 2026, and not one went looking for an llms.txt. OtterlyAI counted 84 requests for the file in more than 62,100 AI bot visits over 90 days, about 0.1% (updated 5 February 2026). Our own site had none from verified AI bots between 12 October and 10 November 2026. Documentation sites see more, as covered under "Is llms.txt worth doing?". A request isn't a citation either. SE Ranking found no link between having the file and how often a domain was cited, across about 300,000 domains (15 October 2026). For Dutch-language questions, Timmermans Media found an llms.txt on 22.4% of sites ChatGPT or Gemini cited, and on 25.4% of comparable Google top 10 sites they didn't cite (measured 26 September 2026). If you check your own logs, don't trust a "GPTBot" label on sight. HUMAN Security found "a significant portion" of requests claiming to be ChatGPT, Mistral and Perplexity bots didn't come from those companies. To check a visit, see if its IP address is on the list OpenAI publishes at openai.com/gptbot.json.

How do I create an llms.txt file?

Write a Markdown file called llms.txt. Start with your business name as a heading, add a one-line summary in a quote block, then list your key pages with a short note on each. On Webflow, upload it under Site settings, SEO, LLMs.txt. On Astro, put it in the public folder. Then load yoursite.com/llms.txt to check it works.

Is llms.txt worth it?

For most business sites, it's worth an hour and no more. It won't move your Google rankings, and no major AI company says it reads yours. Add one if someone will keep it current, and don't pay extra for it as an AI visibility service.

Can llms.txt stop AI companies using my content for training?

No. Use robots.txt for that. OpenAI, Anthropic and Perplexity each document user agents you can allow or block there, such as GPTBot and ClaudeBot.

Is llms.txt an official standard?

No. It's a proposal at llmstxt.org, open for community input. robots.txt, by contrast, is an Internet standard, RFC 9309.

What is llms-full.txt?

Some documentation sites also publish the whole of their docs as one file, often called llms-full.txt. The X developer docs do this. It isn't part of the llmstxt.org proposal, and a business website rarely needs one.

Sources

Checked 11 November 2026.

If you're not sure what AI crawlers can see on your site, or whether you're being cited at all, we can look at it with you. We build on Webflow and custom Astro sites, and we run SEO and AEO for both.

Book a discovery call

Want help with your project?

Tell us what you're building and we'll talk through the right approach.

Book a discovery call

Related reading

See how we approach SEO and Digital Growth