SEO glossary · Crawling and indexing
What is robots.txt?
robots.txt is a plain text file at the root of a site (/robots.txt) that tells crawlers which paths they may fetch. It follows the Robots Exclusion Protocol, published as RFC 9309 in September 2022. It controls crawling, not indexing: a blocked page can still appear in search results when other pages link to it.
How is a robots.txt file written?
- User-agent names the crawler a group of rules is for; * means every crawler without a group of its own.
- Disallow and Allow list path prefixes. When both match a URL, the longest matching rule wins.
- Sitemap gives the full address of a sitemap and can appear anywhere in the file.
- A crawler obeys only the most specific group that names it, so a named group does not inherit the rules under *.
What limits does Google apply to robots.txt?
Google reads the first 500 kibibytes of a robots.txt file and ignores the rest. It usually caches the file for up to 24 hours. When the file answers with a server error (5xx), Google treats the whole site as blocked for a time; when it answers 404, Google treats the site as having no restrictions.
Why can robots.txt not keep a page out of search?
A Disallow rule stops a crawler from fetching a page, not from listing its address. To keep a page out of search results, leave it crawlable and add a noindex rule, which a crawler can only see when it is allowed to fetch the page. Blocking a page that carries noindex defeats the noindex.
How SEOAnything helps
The robots.txt generator builds a valid file with separate rules for search, assistant and training crawlers, free and in your browser. Go to robots.txt generator.
Sources
Also described at Wikipedia: Robots.txt.
Related terms
- noindex: noindex is a rule that tells search engines not to show a page in results. How to set it in a meta tag or a header, and why robots.txt must not block the page.
- AI crawlers: AI crawlers are the bots AI companies use to fetch web pages. The three kinds (training, search, user-triggered), their names, and which to allow or block.
- XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.
- Crawl budget: Crawl budget is the number of URLs a search engine will crawl on a site in a given time. Google says most sites need not manage it. Who does, and what helps.
Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.