SEO glossary · AI search
What are AI crawlers?
AI crawlers are the automated programs that AI companies use to fetch web pages. They come in three kinds with different jobs: training crawlers collect pages to train models, search crawlers build the index an assistant searches, and user-triggered fetchers open one page when a person asks about it.
Which AI crawlers are there?
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search), ChatGPT-User (a person's request).
- Anthropic: ClaudeBot (training), Claude-SearchBot (search), Claude-User (a person's request).
- Perplexity: PerplexityBot (search index), Perplexity-User (a person's request).
- Google: Googlebot crawls for Search, AI Overviews included. Google-Extended is a robots.txt token, not a separate crawler: it controls whether pages are used for Gemini models.
- Others: Applebot and Applebot-Extended (Apple), CCBot (Common Crawl), Meta-ExternalAgent and Meta-ExternalFetcher (Meta), Bingbot (Bing and Copilot).
Should a site block AI crawlers?
It depends on the kind. Blocking a training crawler such as GPTBot keeps pages out of future model training and does not remove a site from ChatGPT search. Blocking a search crawler such as OAI-SearchBot does remove it: OpenAI says sites opted out of OAI-SearchBot are not shown in ChatGPT search answers. A site that wants to be cited should allow the search crawlers and the user-triggered fetchers.
How does robots.txt apply to AI crawlers?
Each crawler follows the most specific group that names it. A group for GPTBot replaces the rules under User-agent: * for that crawler; it does not add to them. So a path blocked for every crawler must be repeated in each named group. robots.txt is also a request, not a lock: a user agent can be faked, and OpenAI and Perplexity publish the IP addresses of their real crawlers so that a visit can be checked.
How SEOAnything helps
The robots.txt generator writes rules for each kind of AI crawler, and the AI crawler log reader shows which crawlers visited, from your own access log, in the browser. Go to robots.txt generator.
Sources
- OpenAI: overview of OpenAI crawlers
- Google Search Central: Google's common crawlers
- Perplexity: Perplexity crawlers
- RFC 9309: Robots Exclusion Protocol
Related terms
- robots.txt: robots.txt is a text file at a site's root that tells crawlers which paths they may fetch. Its syntax, its limits, and why it cannot keep a page out of search.
- llms.txt: llms.txt is a proposed Markdown file at a site's root that gives language models a summary and a list of key pages. What goes in it and who reads it today.
- Generative engine optimization (GEO): Generative engine optimization (GEO) makes pages easy for AI answer engines such as ChatGPT, Gemini and Perplexity to find, quote and cite. What it involves.
- noindex: noindex is a rule that tells search engines not to show a page in results. How to set it in a meta tag or a header, and why robots.txt must not block the page.
Updated 20 November 2026 · Search engines and AI platforms change their rules: check the sources above for what is true today.