SEO glossary · Crawling and indexing
What is noindex?
noindex is a rule that tells search engines not to show a page in their results. It is set with a robots meta tag in the page's head, <meta name="robots" content="noindex">, or with an X-Robots-Tag HTTP header. The page must stay crawlable, because a crawler that cannot fetch the page never sees the rule.
When should a page carry noindex?
- Internal search results and filter combinations that produce endless near-identical pages.
- Thin or unfinished pages that are public but not ready to rank.
- Login, account and thank-you pages.
- Files such as PDFs, using the X-Robots-Tag header, because a file has no head.
How do noindex and robots.txt work together?
They do different jobs and can cancel each other. robots.txt stops crawling; noindex stops indexing. A page that is both blocked in robots.txt and marked noindex can stay in the index, because the crawler is never allowed to read the noindex. To remove a page from results, allow crawling, add noindex, and wait for the page to be crawled again.
How can a page be checked for noindex?
A noindex can come from two places: the HTML and the response headers. Both must be checked, because an X-Robots-Tag header set by a server or a CDN is invisible in the page source. Google Search Console's URL Inspection reports a page blocked this way as "Excluded by noindex tag".
How SEOAnything helps
The meta tag checker shows a page's robots meta tag and the HTTP header checker shows its X-Robots-Tag header, as a crawler gets them. Go to HTTP header checker.
Sources
- Google Search Central: block search indexing with noindex
- Google Search Central: robots meta tag and X-Robots-Tag specifications
Also described at Wikipedia: Noindex.
Related terms
- robots.txt: robots.txt is a text file at a site's root that tells crawlers which paths they may fetch. Its syntax, its limits, and why it cannot keep a page out of search.
- Canonical tag: A canonical tag names the preferred address of a page that can be reached at several URLs. How search engines use it, and the mistakes that make them ignore it.
- Crawl budget: Crawl budget is the number of URLs a search engine will crawl on a site in a given time. Google says most sites need not manage it. Who does, and what helps.
- XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.
Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.