What is llms.txt, exactly?#
llms.txt is a proposed convention — not a W3C or IETF standard, just a format enough sites adopted that it became a de facto one — for a plain-text, Markdown-formatted file served at a site's root: https://example.com/llms.txt. It gives an AI system a short, curated map of a site instead of forcing it to crawl and guess: a one-line description of what the site is, followed by grouped, linked lists of the pages worth reading and a short note on what each one covers.
The proposal, from Jeremy Howard of Answer.AI, also defines an optional companion file, /llms-full.txt, which inlines the full content of every listed page into one document — useful for models with large enough context windows to skip fetching each page individually.
CrawlReady's own /llms.txt is a working example: it lists the free tools, then every doc and blog post currently published, generated from the same content loaders that build the docs and blog pages — so it can't list a page that's been removed or miss one that's new.
Does llms.txt affect Google or AI Overviews?#
No, and Google says so directly. Its AI features documentation states you don't need to create new machine-readable files, AI text files, or markup for AI Overviews or AI Mode — those features work from the same rendered HTML that ordinary Search indexing uses. Publishing /llms.txt won't move a Google-driven score, and CrawlReady's own Agent Readiness check treats it as one signal among several, not a required file.
Do AI bots actually fetch it?#
Some do, per third-party crawl data rather than the platforms' own documentation. Practitioner crawl telemetry — a self-reported, single-product sample (TrustMRR, n=1M+ AI bot requests over 30 days, surfaced August 2026) — found /llms.txt among the most-requested paths by OpenAI and Anthropic's bots, reportedly used both to answer user queries and, per that data, to gather training data. That's evidence the convention gets picked up in practice, not an official statement from OpenAI or Anthropic that every deployment reads it — treat it as corroborating, not definitive.
That asymmetry is the practical takeaway: llms.txt is a low-cost file to publish (a few dozen lines, no build-time cost if generated from existing content) that has a plausible payoff with OpenAI- and Anthropic-side tooling specifically, and no claimed effect on Google.
llms.txt vs. robots.txt#
Easy to conflate since both are plain-text root files read by crawlers, but they do opposite jobs:
robots.txt | llms.txt | |
|---|---|---|
| Purpose | Permissions — what a crawler may or may not fetch | Curation — which pages matter, and why |
| Format | Directive syntax (User-agent, Allow, Disallow) | Markdown, human-readable |
| Predates AI? | Yes, since 1994 | No, proposed September 2024 |
| Enforced by? | Voluntary, but widely respected by major crawlers | Nothing — a bot that ignores it just crawls normally |
A site can have a strict robots.txt and no llms.txt, a permissive robots.txt and a detailed llms.txt, or any other combination — they don't reference or depend on each other.
What actually goes in the file?#
The convention itself is loose, but a minimal, useful /llms.txt follows this shape:
# Company Name
> One-sentence description of what the site or product does.
- [Page title](https://example.com/page): what this page covers
- [Another page](https://example.com/other): what this one covers
## Docs
- [Getting started](https://example.com/docs/start): setup guide
The parts worth getting right: the description at the top should be accurate and specific rather than marketing copy, since it's the only context a model gets before deciding whether to follow a link; and the list should be generated from real content rather than hand-maintained, or it silently goes stale the first time a page is renamed or removed.
Run a free scan to check whether your site's Agent Readiness signals — llms.txt included — are in place.
