SEO glossary · Crawling and indexing

What is robots.txt?

robots.txt is a plain text file at the root of a site (/robots.txt) that tells crawlers which paths they may fetch. It follows the Robots Exclusion Protocol, published as RFC 9309 in September 2022. It controls crawling, not indexing: a blocked page can still appear in search results when other pages link to it.

How is a robots.txt file written?

  • User-agent names the crawler a group of rules is for; * means every crawler without a group of its own.
  • Disallow and Allow list path prefixes. When both match a URL, the longest matching rule wins.
  • Sitemap gives the full address of a sitemap and can appear anywhere in the file.
  • A crawler obeys only the most specific group that names it, so a named group does not inherit the rules under *.

What limits does Google apply to robots.txt?

Google reads the first 500 kibibytes of a robots.txt file and ignores the rest. It usually caches the file for up to 24 hours. When the file answers with a server error (5xx), Google treats the whole site as blocked for a time; when it answers 404, Google treats the site as having no restrictions.

Why can robots.txt not keep a page out of search?

A Disallow rule stops a crawler from fetching a page, not from listing its address. To keep a page out of search results, leave it crawlable and add a noindex rule, which a crawler can only see when it is allowed to fetch the page. Blocking a page that carries noindex defeats the noindex.

How SEOAnything helps

The robots.txt generator builds a valid file with separate rules for search, assistant and training crawlers, free and in your browser. Go to robots.txt generator.

Sources

Also described at Wikipedia: Robots.txt.

Related terms

  • noindex: noindex is a rule that tells search engines not to show a page in results. How to set it in a meta tag or a header, and why robots.txt must not block the page.
  • AI crawlers: AI crawlers are the bots AI companies use to fetch web pages. The three kinds (training, search, user-triggered), their names, and which to allow or block.
  • XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.
  • Crawl budget: Crawl budget is the number of URLs a search engine will crawl on a site in a given time. Google says most sites need not manage it. Who does, and what helps.

All terms in the SEO and AI search glossary

Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.