Does llms.txt Actually Work? 2026 Adoption and Crawler Data

Few AI search tactics have spread as fast as llms.txt. It is easy to explain, easy to implement and easy to sell: add one file and AI systems will understand your site. Plugins generate it automatically, audit tools flag sites that lack it, and agencies add it to proposals.

The question clients eventually ask is whether it works. This article looks at what llms.txt is, how widely it is used, what the crawler data shows, and when it is still worth publishing. We build an llms.txt generator into Rankwyn, so we have every reason to be honest about its limits.

What is llms.txt?

llms.txt is a proposed standard for a Markdown file at the root of a website, for example example.com/llms.txt. It was put forward in 2024 as a way to give language models a concise, readable overview of a site: what it is, and links to its most useful pages, written in plain Markdown without navigation, scripts or styling.

A typical file looks like this:

# Nordic Roast

> Specialty coffee roaster in Oslo selling single-origin beans online across Norway and Sweden.

## Shop
- [Single-origin coffee](https://nordicroast.co/coffee): All current beans with tasting notes and prices
- [Subscriptions](https://nordicroast.co/subscriptions): Weekly and monthly plans

## Help
- [Shipping and returns](https://nordicroast.co/help/shipping): Delivery times and costs by country

A related variant, llms-full.txt, contains the full text of key pages in one file.

Crucially, llms.txt is a proposal, not an adopted web standard, and it has no access-control role. It does not tell crawlers what they may access. That job belongs to robots.txt.

How many sites use llms.txt?

Adoption is growing quickly from a small base. Different studies sample different sets of sites, so the figures vary, but they tell a consistent story.

So in 2026, somewhere between one in ten and one in twenty of the most visited sites publish an llms.txt file. Among smaller sites, many files exist because a plugin generated them automatically, not because anyone chose to.

Do AI crawlers actually read llms.txt?

Mostly, no. This is the critical finding, and it is consistent across the analyses available.

Studies of server logs report that the major AI crawlers, including GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, overwhelmingly skip llms.txt and crawl HTML pages directly. One widely reported analysis found that 97% of llms.txt files it examined received zero requests from AI systems over the period studied.

There is also no public confirmation from OpenAI, Anthropic, Google or Perplexity that their systems use llms.txt when choosing or citing sources. Their published crawler documentation covers user agents, IP ranges and robots.txt rules. It does not describe llms.txt.

Does llms.txt improve AI citations?

There is no evidence that it does. Analyses comparing sites with and without llms.txt have not found a measurable difference in how often they are cited by ChatGPT, Claude, Gemini or Perplexity. That is consistent with the crawler data: a file that is rarely requested cannot have much influence.

This mirrors what controlled research found for schema markup. Tactics that are concrete and easy to implement tend to be overestimated. See does schema markup get you cited by AI? for the parallel case.

So why does anyone recommend it?

There are a few honest reasons, and a few less honest ones.

Honest reasons:

• It is cheap. Generating a sensible file takes minutes, and keeping it current can be automated.
• It may matter later. If an AI system adopts the convention, sites that already have a good file are ready.
• It is useful for developer tools. Some coding assistants and documentation tools do read llms.txt or llms-full.txt when a developer points them at a site. For API and documentation sites, that is a real use.
• Writing it is a useful exercise. Summarising a site in a few lines forces clarity about what it is and which pages matter most.

Less honest reasons:

• It is easy to sell as a deliverable.
• Tools can flag its absence as an "issue", which creates urgency.

llms.txt compared with robots.txt and sitemap.xml

The three files are often mentioned together, but they do very different jobs.

If you only have time for one, get robots.txt right. See AI crawler user agents.

When llms.txt is worth publishing

Publish one when:

• The site is documentation-heavy, an API reference or a developer product.
• You can generate and update it automatically, so it never goes stale.
• The client wants it, and you have explained what it does and does not do.

Do not prioritise it over the things that do affect AI visibility. Those are crawler access, server-rendered content, answer-first pages and brand corroboration across trusted sites. See our AEO and GEO guide for the full list.

How to write a good llms.txt

If you publish one, make it good.

1. Start with a one-line description. State what the site is in plain words, the same canonical description used on the about page.
2. Group links by purpose. Products, guides, documentation, help, company.
3. Describe each link in a few words. Say what the page contains, not marketing copy.
4. Link to canonical, indexable pages only. No redirects, no parameter URLs, no pages blocked in robots.txt.
5. Keep it short. Aim for the 20 to 50 most useful pages, not a full sitemap.
6. Serve it as plain text. Use text/plain or text/markdown with UTF-8 encoding at the root.
7. Regenerate it when the site changes. A stale file pointing to removed pages is worse than none.
8. Do not block it. If you publish it, make sure robots.txt allows access.

How to check whether AI systems request it on a client's site

The simplest proof for a specific client is their own server logs. Filter requests for /llms.txt and group them by user agent over a month. In most cases you will see few or no requests from AI crawlers, and those you see may be from tools rather than answer engines. Our guide to AI bot log file analysis shows how to do this.

If you see regular requests from a crawler that matters to the client, that is useful information, and a reason to keep the file well maintained.

What to tell clients

Clients ask about llms.txt because they have read about it. A short, direct answer works best:

• "We can add one. It takes little effort and does no harm."
• "There is no evidence today that it makes AI systems cite you more, and the major AI crawlers rarely request it."
• "The work that does move AI visibility is making sure crawlers can read your pages, writing pages that answer questions clearly, and building your reputation on the sites AI engines trust."

Being straight about a popular tactic earns more trust than adding it to a proposal as a headline item.

How Rankwyn handles llms.txt

Rankwyn can generate and keep an llms.txt file in sync for each connected site, and it stages the file for approval like any other change. We include it because it is low-cost hygiene and some clients want it. We do not present it as a citation driver, and our AEO Citation Radar measures actual citations so you can judge any change by its results.

Key takeaways

• llms.txt is a proposed Markdown summary of a site, not an adopted standard and not an access-control file.
• Adoption has grown to roughly 5–9% of top sites, often through plugins.
• Server-log analyses show the major AI crawlers rarely request it, and no study has found a citation benefit.
• It is reasonable to publish one when it can be generated and maintained automatically, particularly for documentation sites.
• Prioritise crawler access, rendering and content quality first.

The bottom line

llms.txt is harmless and inexpensive, and in some developer contexts it is useful. For answer engines in 2026, the evidence says it does not do what it is often sold to do. Publish one if it is easy to maintain, check the client's logs to see who requests it, and put the real effort into the factors that decide whether a page is read and cited. For crawler access in particular, see AI crawler user agents.