SEO glossary · AI search
What is llms.txt?
llms.txt is a proposed standard: a Markdown file at the root of a website (/llms.txt) that gives large language models a short description of the site and a curated list of links to its most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024. It is a convention, and no crawler is required to read it.
What goes in an llms.txt file?
- An H1 with the name of the site or project. This is the only required part.
- A blockquote with a short summary.
- Optional paragraphs of notes, with no headings.
- H2 sections, each a list of links written as [name](url): a note about the page.
- A section named "Optional" for links a reader can skip when it needs a shorter context.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which paths they may fetch. llms.txt grants and blocks nothing: it is a reading guide that points a language model at the pages worth reading, in a format that needs no HTML parsing. A site can have both, and a sitemap as well.
Do AI search engines use llms.txt?
Use is uneven. Google's guidance on AI features says no new machine-readable files are needed to appear in AI Overviews or AI Mode. llms.txt is most useful today for coding assistants and agents that a person points at a site's documentation, and it costs little to publish.
What is llms-full.txt?
llms-full.txt is a companion convention: one file holding the full text of the main pages as Markdown, for tools that prefer to read a single document instead of following links.
How SEOAnything helps
The SEO tools produce starter robots.txt, sitemap.xml and llms.txt files for a site. SEOAnything publishes its own at /llms.txt and /llms-full.txt. Go to SEO tools.
Sources
Related terms
- AI crawlers: AI crawlers are the bots AI companies use to fetch web pages. The three kinds (training, search, user-triggered), their names, and which to allow or block.
- robots.txt: robots.txt is a text file at a site's root that tells crawlers which paths they may fetch. Its syntax, its limits, and why it cannot keep a page out of search.
- Generative engine optimization (GEO): Generative engine optimization (GEO) makes pages easy for AI answer engines such as ChatGPT, Gemini and Perplexity to find, quote and cite. What it involves.
- XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.
Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.