SEO glossary · Crawling and indexing

What is noindex?

noindex is a rule that tells search engines not to show a page in their results. It is set with a robots meta tag in the page's head, <meta name="robots" content="noindex">, or with an X-Robots-Tag HTTP header. The page must stay crawlable, because a crawler that cannot fetch the page never sees the rule.

When should a page carry noindex?

  • Internal search results and filter combinations that produce endless near-identical pages.
  • Thin or unfinished pages that are public but not ready to rank.
  • Login, account and thank-you pages.
  • Files such as PDFs, using the X-Robots-Tag header, because a file has no head.

How do noindex and robots.txt work together?

They do different jobs and can cancel each other. robots.txt stops crawling; noindex stops indexing. A page that is both blocked in robots.txt and marked noindex can stay in the index, because the crawler is never allowed to read the noindex. To remove a page from results, allow crawling, add noindex, and wait for the page to be crawled again.

How can a page be checked for noindex?

A noindex can come from two places: the HTML and the response headers. Both must be checked, because an X-Robots-Tag header set by a server or a CDN is invisible in the page source. Google Search Console's URL Inspection reports a page blocked this way as "Excluded by noindex tag".

How SEOAnything helps

The meta tag checker shows a page's robots meta tag and the HTTP header checker shows its X-Robots-Tag header, as a crawler gets them. Go to HTTP header checker.

Sources

Also described at Wikipedia: Noindex.

Related terms

  • robots.txt: robots.txt is a text file at a site's root that tells crawlers which paths they may fetch. Its syntax, its limits, and why it cannot keep a page out of search.
  • Canonical tag: A canonical tag names the preferred address of a page that can be reached at several URLs. How search engines use it, and the mistakes that make them ignore it.
  • Crawl budget: Crawl budget is the number of URLs a search engine will crawl on a site in a given time. Google says most sites need not manage it. Who does, and what helps.
  • XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.

All terms in the SEO and AI search glossary

Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.