What Is llms.txt and Does Your Site Need One?
A small markdown file at your site root tells AI systems what your site is and which pages matter. Here is exactly how it works and when it is worth adding.
llms.txt is a plain-markdown file at your site root that tells AI systems what your site is and which pages matter, in a format designed for language models. It is a community proposal from llmstxt.org, not an official standard, but it costs one small text file and several AI tools already read it.
The problem it solves
When a language model wants to understand your site, it has two bad options. It can crawl raw HTML, wading through navigation, cookie banners and scripts to find the substance. Or it can rely on what ended up in its training data, which is often stale. Context windows are limited, so every token wasted on boilerplate is a token not spent on your actual content.
llms.txt is the shortcut: a curated, LLM-friendly map that says "this is who we are, and these are the pages worth reading", in clean markdown that models parse natively.
The format, in one example
The whole specification fits in four rules. One # heading with the site name (required). An optional > blockquote with a one-line summary. Optional prose. Then ## sections containing markdown links, each with an optional description:
# CellChart
> Button and coin battery cross-reference: every alias,
size and equivalent for 92 cells.
## Battery families
- [LR44 equivalents](https://example.com/lr44/): all names
for the LR44 and what replaces it
- [CR2032 vs CR2025](https://example.com/cr2032-vs-cr2025/):
the two coin cells people confuse mostA section named ## Optional has special meaning: models may skip it when context is tight. Put your must-read pages in the earlier sections and the nice-to-haves there.
llms.txt vs robots.txt vs sitemap.xml
| File | Audience | Says | Format |
|---|---|---|---|
robots.txt | All crawlers | "You may / may not crawl these paths" | Directives |
sitemap.xml | Search engine crawlers | "Here is every page, exhaustively" | XML |
llms.txt | Language models | "Here is what matters, curated" | Markdown |
The three complement each other. robots.txt controls access, the sitemap aims for completeness, llms.txt aims for curation. A companion file, llms-full.txt, optionally inlines the full text of your key pages for models that want everything in one fetch; start without it.
The honest adoption picture
Be skeptical of anyone selling llms.txt as a ranking hack. Google has said it does not use the file, and no major engine has committed to it publicly. Adoption is real but uneven: documentation platforms like Mintlify generate it for thousands of docs sites, companies like Anthropic and Cloudflare publish one, and a growing set of AI dev tools fetch it when present.
The correct framing is cheap insurance. The file takes minutes to create, harms nothing, and if AI assistants keep growing as a traffic source, the sites with clean machine-readable maps are positioned first. That asymmetry, tiny cost against a plausible upside, is the whole argument, and it is enough.
Common mistakes
- Dumping every URL. That is what the sitemap is for. An llms.txt with 5,000 links is exactly as useful to a model as no file at all. Curate 10 to 30 pages.
- Wrong location. The file lives at the root:
https://yourdomain.com/llms.txt. Subdirectories are not part of the proposal and tools only check the root. - Serving HTML. Some hosts route unknown paths to a 200 error page. AI tools then parse your 404 page as your site description. Check what the URL actually returns.
- Vague descriptions. "Useful resources" tells a model nothing. "Expiry policies for 14 car seat brands, verified against manufacturer pages" is what gets you quoted accurately.
- Writing it once and forgetting it. Dead links in the file send models to 404s. Regenerate when key pages change.
How to add one in five minutes
Our free llms.txt generator reads your sitemap and page metadata and assembles the file for you; review it, edit what reads wrong, and upload it next to robots.txt. Then run the AI readiness check to confirm it is reachable and valid, and to see the rest of your AI-facing surface (robots.txt rules, structured data) at the same time.