What exactly is llms.txt?

llms.txt is a proposal, not an official standard, published by Jeremy Howard, co-founder of Answer.AI, on September 3, 2024 at llmstxt.org. It is a markdown file at the root of a website (/llms.txt) that summarizes what the site is and links to its key pages, in a format designed for language models and their context window limits.

Does Google, ChatGPT or Claude read llms.txt?

There is no evidence of official support. Google has denied it repeatedly: John Mueller compared the file to the keywords meta tag in April 2025, Gary Illyes said in July 2025 that Google does not support it and is not planning to, and Google Search Central documentation states you do not need new AI text files to appear in AI features. OpenAI's crawler documentation only mentions robots.txt, and the Ahrefs study of 137,210 domains found that 97% of published llms.txt files received no requests at all.

Does llms.txt replace robots.txt or my sitemap?

No. robots.txt controls crawler access and is the only control mechanism OpenAI's official crawler documentation names, while sitemap.xml lists URLs for indexing. llms.txt is a different thing: a curated content guide for language models. It blocks nothing, permits nothing, and no major AI engine currently declares that it uses it.

Is it worth creating an llms.txt file at all?

It depends on cost and site type. The evidence says almost nobody reads it, so do not expect traffic from having one. The cost is near zero, though: Yoast SEO generates it for free on WordPress and Mintlify creates it automatically for documentation sites. The most real use case is technical documentation consumed by coding assistants: in the Ahrefs server-log study, the few AI fetches that did happen came mostly from GPTBot and Claude-Code. For a small business without technical docs it is optional and low priority.

How do you write a correct llms.txt?

Follow the specification: an H1 with the site name (the only required section), a blockquote with a one or two sentence summary, H2 sections containing link lists where each markdown link can carry a short note, and a final Optional section for content that can be dropped. The spec also suggests publishing clean markdown versions of key pages at the same URL with .md appended. Live examples include the developer docs of Anthropic and Perplexity.

What is llms-full.txt and how is it different?

llms.txt is a curated index of links; llms-full.txt dumps the site's entire documentation into a single markdown file. Mintlify deployed both automatically for all the documentation sites it hosts in November 2024. On large sites llms-full.txt can exceed the context window of many models, so it works better for deliberate ingestion into a tool than for discovery.

What Is llms.txt and Does Anything Actually Read It?

llms.txt is a proposed markdown file, served at a website’s root, that gives large language models a curated index of the site’s key content. Jeremy Howard of Answer.AI published the spec on September 3, 2024. As of 2026, no major AI engine declares support: Google has rejected it explicitly, OpenAI’s crawler docs never mention it, and Ahrefs found 97% of published files receive zero requests. Here is the format, the evidence, and when it is still worth having.

What is llms.txt and where does it come from?

llms.txt is a proposal by Jeremy Howard, co-founder of Answer.AI and fast.ai, published on September 3, 2024 at llmstxt.org. It is not a ratified web standard; no AI vendor is obliged to read it, and adoption is entirely voluntary.

The problem it targets is a real constraint. Language models operate within context window limits, and the spec starts from there. Howard’s answer is a compact markdown file at /llms.txt, written to help LLMs use a website at inference time: a short summary plus links to the pages that matter, in a format sized for those limits.

The specification goes beyond the single file. It also proposes publishing a clean markdown version of each page at the same URL with .md appended (and index.html.md for URLs without a filename). The use case the spec highlights is development environments, where LLMs need fast access to programming and API documentation.

llms.txt is a recurring entry in generative engine optimization checklists. Whether it has earned that spot is exactly what this guide examines.

What does a correct llms.txt file look like?

The format is minimal, per the official specification (September 2024). One H1 with the site name is the only required section. Then, in order:

A blockquote with a brief summary of the site.
Zero or more markdown sections with context worth knowing.
Zero or more file lists, each delimited by an H2, with links in [name](url) format optionally followed by : note.
A special section named “Optional” for secondary material that can be dropped when a shorter context is needed.

A skeleton with placeholders looks like this:

# Example Company

> One-sentence summary of what the company does and for whom.

Short paragraph of context a language model should know.

## Documentation

- [Quickstart](https://example.com/docs/quickstart.md): install and first steps
- [API reference](https://example.com/docs/api.md): endpoints and authentication

## Optional

- [Company history](https://example.com/about.md)

How is llms.txt different from robots.txt and sitemap.xml?

They do different jobs: robots.txt controls crawler access, sitemap.xml lists URLs for indexing, and llms.txt is a curated content guide with no enforcement power. It blocks nothing and permits nothing.

File	What it does	Official support today
robots.txt	Allows or blocks crawler access	The only control mechanism OpenAI documents (developers.openai.com/api/docs/bots)
sitemap.xml	Lists URLs for discovery and indexing	Long-standing search engine standard
llms.txt	Curated markdown index for LLMs	No major AI engine declares it reads the file (2026)
llms-full.txt	Entire documentation in one markdown file	Auto-generated by platforms such as Mintlify

OpenAI’s official crawler documentation (accessed July 2026) covers GPTBot, OAI-SearchBot, OAI-AdsBot and ChatGPT-User, names robots.txt as the only control mechanism, and never mentions llms.txt. If your actual goal is managing AI bot access, that is a robots.txt job — our guide to AI crawlers covers it.

llms-full.txt is the maximalist sibling: instead of an index, it dumps all documentation into a single markdown file. Its practical limit is size — on large sites it can exceed many models’ context windows, so it suits deliberate ingestion more than discovery.

Who actually publishes llms.txt?

Mostly platforms that generate it for their users, not sites that chose it: 78.1% of Shopify sites publish one through automatic deployment, against 5.61% adoption across the top 10,000 overall, per Casey Burridge’s HTTP Archive analysis (June 2026). The use case with real signal in server logs is narrower — technical documentation, and there too the file often arrives from the docs host rather than the site owner.

Anthropic publishes a real one: docs.anthropic.com/llms.txt redirects (301) to platform.claude.com/docs/llms.txt, a markdown index headed “Anthropic Developer Documentation” with link sections by language and category (checked July 16, 2026).
Perplexity serves docs.perplexity.ai/llms.txt, with an H1 and a blockquote describing its APIs.
Mintlify switched on automatic /llms.txt and /llms-full.txt generation for every docs site it hosts on November 20, 2024, citing Anthropic, Windsurf and Bolt.new. Thousands of documentation sites got the file overnight.

Note the asymmetry: AI labs use llms.txt for their own docs, but none has declared that its engines consume the file on third-party sites. According to Casey Burridge’s HTTP Archive analysis (June 2026), ChatGPT, Claude and Gemini have no llms.txt on their main domains — only on their developer docs.

How many websites have adopted llms.txt?

Between roughly 5% and 28%, depending entirely on the sample you measure. The headline numbers diverge because the methodologies do.

According to Burridge’s HTTP Archive dataset (July 2025 to June 2026), adoption in the top 10,000 sites grew from 1.04% to 5.61% in twelve months — about 5.4x. Broken down by platform, Shopify leads: 78.1% of Shopify sites publish one, driven by automatic platform deployment rather than organic adoption, while WordPress sits around 8.7%. According to Ahrefs (May 2026 logs), 28% of 137,210 domains with traffic publish a valid llms.txt — about 38,000 sites.

Aggregators report yet other figures: Rankability puts adoption at 8.7% of the global top 1,000 (June 2026), another analysis of ~300,000 domains found ~10.13%, and ppc.land reports 8.8x growth in twelve months, from 4,088 to 36,120 files.

The spread is method, not contradiction: different domain samples, validity criteria and data sources. Two signals hold across every study. Adoption is growing fast, and much of it comes from platforms — Shopify, Mintlify, and Yoast SEO, which generates llms.txt automatically as a free WordPress feature while its own product page stops short of guaranteeing that AI platforms use it.

Does Google use llms.txt?

No — and Google has said so at least three times, in escalating levels of formality.

In April 2025, John Mueller wrote on Reddit (r/TechSEO), as reported by Search Engine Journal, that no AI service has said it uses llms.txt, that server logs show they do not even check for it, and that the file reminded him of the keywords meta tag — the classic example of metadata search engines learned to ignore.

In July 2025, Gary Illyes was blunter at the Search Central Deep Dive APAC event:

“Google doesn’t support LLMs.txt and isn’t planning to” — Gary Illyes (Google), as reported by Search Engine Land, July 24, 2025

Per the same report, Illyes added that normal SEO is enough to appear in AI Overviews. And Google Search Central’s official documentation on AI features (updated December 10, 2025) states plainly that you do not need to create new machine-readable files, AI text files or markup to appear in those features.

Mueller also flagged a structural flaw, per Search Engine Journal’s coverage: a site could show one thing in llms.txt and something else to users and search engines. That cloaking risk forces any serious bot to verify the real page anyway — which makes the file redundant.

Does ChatGPT or Claude read llms.txt?

Neither company says so. OpenAI’s crawler documentation names robots.txt as the only mechanism and contains no statement that GPTBot or ChatGPT reads llms.txt. Anthropic publishes the file for its own developer docs but has made no declaration that Claude consumes third-party llms.txt files — an absence of a claim, not a denial.

The one positive signal lives in server logs. Within the minority of llms.txt fetches that Ahrefs (May 2026) attributed to AI tools, GPTBot and Claude-Code stood out. That points to coding assistants pulling technical documentation — not to AI search engines discovering ordinary websites.

What do server logs reveal about llms.txt?

Near-total silence. In the Ahrefs study of 137,210 domains (May 2026 logs), 97% of published llms.txt files received not a single request. Ahrefs’ own conclusion: “AI search bots barely fetch these files, and no AI system goes looking for one.”

The traffic that did arrive was mostly noise. 96% of requests came from bots — SEO audit tools (21.7%), unidentified bots (14.9%), general crawlers (13.1%) and technology profilers like BuiltWith (11.6%). AI tools accounted for 19.5% of those fetches, led by GPTBot and Claude-Code, yet only about 1.1% of requests came from AI retrieval bots — the file’s theoretical audience.

In UpgradePro’s labeling, that makes any traffic benefit from llms.txt a SUPUESTO (assumption), not a MEDIDO (measured) result. Nobody has published logs showing otherwise.

Should you create an llms.txt file?

Create one if it costs you nothing; skip it if it takes real work away from things that are actually read. The evidence supports a cost-based decision, not a hype-based one.

Situation	Verdict
Developer docs, API or SaaS product	Best case: coding assistants (GPTBot, Claude-Code) show real fetches in logs
Docs hosted on Mintlify	Already done automatically since November 2024
WordPress site with Yoast SEO	Enable the free feature; near-zero cost, no expected traffic
SMB brochure or local-service site	Low priority: no engine reads it, and there are no docs to index

Whatever you decide, do not fund it from work with actual evidence behind it. Clean crawlable HTML, answer-first content and structured data that AI systems parse all have a stronger claim on your hours than a file that 97% of the time nobody requests.

What should you do this week?

A 30-minute, evidence-first routine for a small team:

Check your server logs for requests to /llms.txt over the last 90 days. Measure demand before investing — that is the difference between MEDIDO and SUPUESTO.
On WordPress: switch on Yoast’s free llms.txt generation. Cost is near zero, so the negative evidence does not matter.
On Mintlify docs: you already serve /llms.txt and /llms-full.txt; verify they render correctly.
Writing one by hand: follow the spec exactly — H1, blockquote, H2 link lists, an “Optional” section. Twenty minutes, once.
Do not reallocate budget from crawlability, content structure or structured data to llms.txt maintenance.
Recheck quarterly. Adoption grew ~5.4x in a year; if an engine ever announces support, the calculus changes — and your logs will show it first.

And if you want to know where your site actually stands in AI answers today — measured before and after, not assumed — that baseline is exactly what our AI visibility audit delivers.