SEO glossary · AI search

What is llms.txt?

llms.txt is a proposed standard: a Markdown file at the root of a website (/llms.txt) that gives large language models a short description of the site and a curated list of links to its most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024. It is a convention, and no crawler is required to read it.

What goes in an llms.txt file?

  • An H1 with the name of the site or project. This is the only required part.
  • A blockquote with a short summary.
  • Optional paragraphs of notes, with no headings.
  • H2 sections, each a list of links written as [name](url): a note about the page.
  • A section named "Optional" for links a reader can skip when it needs a shorter context.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers which paths they may fetch. llms.txt grants and blocks nothing: it is a reading guide that points a language model at the pages worth reading, in a format that needs no HTML parsing. A site can have both, and a sitemap as well.

Do AI search engines use llms.txt?

Use is uneven. Google's guidance on AI features says no new machine-readable files are needed to appear in AI Overviews or AI Mode. llms.txt is most useful today for coding assistants and agents that a person points at a site's documentation, and it costs little to publish.

What is llms-full.txt?

llms-full.txt is a companion convention: one file holding the full text of the main pages as Markdown, for tools that prefer to read a single document instead of following links.

How SEOAnything helps

The SEO tools produce starter robots.txt, sitemap.xml and llms.txt files for a site. SEOAnything publishes its own at /llms.txt and /llms-full.txt. Go to SEO tools.

Sources

Related terms

  • AI crawlers: AI crawlers are the bots AI companies use to fetch web pages. The three kinds (training, search, user-triggered), their names, and which to allow or block.
  • robots.txt: robots.txt is a text file at a site's root that tells crawlers which paths they may fetch. Its syntax, its limits, and why it cannot keep a page out of search.
  • Generative engine optimization (GEO): Generative engine optimization (GEO) makes pages easy for AI answer engines such as ChatGPT, Gemini and Perplexity to find, quote and cite. What it involves.
  • XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.

All terms in the SEO and AI search glossary

Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.