SEO glossary · Crawling and indexing
What is crawl budget?
Crawl budget is the set of URLs a search engine can and wants to crawl on a site in a given time. Google defines it by two things: the crawl capacity limit, which is how much crawling the server can take, and crawl demand, which is how much Google wants the site's pages.
Which sites need to manage crawl budget?
Few. Google's guide to crawl budget is written for sites with more than one million unique pages whose content changes about once a week, sites with more than 10,000 pages that change daily, and sites where a large share of URLs sit at "Discovered - currently not indexed" in Search Console. A smaller site whose new pages are crawled the day they are published has no crawl budget problem.
What wastes crawl budget?
- Filter and sort parameters that create endless URL combinations.
- Duplicate pages reachable at several addresses.
- Long redirect chains and soft 404 pages that answer 200.
- Slow responses and server errors, which make Google lower its crawl rate.
What improves crawling?
Serve pages fast, return 404 or 410 for removed pages, keep the sitemap to canonical URLs with honest lastmod dates, and block true crawl traps in robots.txt. Blocking a URL in robots.txt saves crawling but does not remove the URL from the index, and Google does not move the saved crawling to other pages unless the server was at its limit.
How SEOAnything helps
Search Visibility crawls a site within fixed limits and lists redirect chains, duplicate addresses, error pages and pages blocked from indexing, each with its evidence. Go to Search Visibility.
Sources
Related terms
- robots.txt: robots.txt is a text file at a site's root that tells crawlers which paths they may fetch. Its syntax, its limits, and why it cannot keep a page out of search.
- XML sitemap: An XML sitemap lists the URLs a site wants search engines to crawl. Its size limits, which tags Google ignores, and which pages belong in it and which do not.
- Canonical tag: A canonical tag names the preferred address of a page that can be reached at several URLs. How search engines use it, and the mistakes that make them ignore it.
- noindex: noindex is a rule that tells search engines not to show a page in results. How to set it in a meta tag or a header, and why robots.txt must not block the page.
Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.