To be cited by AI assistants, allow their search crawlers in robots.txt. Blocking a training crawler such as GPTBot does not remove a site from ChatGPT search; blocking OAI-SearchBot does. This guide sorts the crawlers by job, gives two files you can copy, and names the one mistake that silently opens or closes a site.
What are the three kinds of AI crawler?
AI crawlers do three different jobs, and each job has its own user agent:
- Training crawlers collect pages that may be used to train models.
- Search crawlers build the index an assistant searches when it answers.
- User-triggered fetchers open one page because a person asked about it or pasted its link.
A rule for one kind does nothing to the others. OpenAI's documentation says each setting is independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.
Which user agent belongs to which company?
| Company | Training | Search | A person's request |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | none | PerplexityBot | Perplexity-User |
| Google-Extended | Googlebot | none | |
| Apple | Applebot-Extended | Applebot | none |
| Meta | Meta-ExternalAgent | none | Meta-ExternalFetcher |
| Common Crawl | CCBot | none | none |
| Microsoft | none | Bingbot | none |
Two rows need a note. Google-Extended is not a crawler: Google says it has no user agent string of its own and is a token that controls whether pages Googlebot already crawled are used for Gemini models. PerplexityBot is a search crawler: Perplexity says it is not used to crawl content for AI foundation models.
The names come from each company's own page: OpenAI, Anthropic, Perplexity and Google.
What happens when you block each kind?
Blocking a training crawler keeps your pages out of future training. It does not remove you from search. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.
Blocking a search crawler removes you from that assistant's answers. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot may reduce a site's visibility in search responses.
Blocking a user-triggered fetcher has uneven results. Anthropic says disabling Claude-User stops Claude retrieving the page for a user's question. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.
A change is not instant. OpenAI says its systems can take about 24 hours to follow a robots.txt update.
A robots.txt that allows AI search and blocks training
This file keeps one private folder closed to everyone, lets the search crawlers and fetchers read the rest, and opts out of training:
User-agent: *
Disallow: /admin/
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /admin/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
A robots.txt that allows everything public
A site that wants the widest reach lets all three kinds in and closes only what is private. SEOAnything's own robots.txt does this: it lists the search crawlers, the fetchers and the training crawlers in three groups, and every group repeats the same rules, which leave everything open except /api. The pattern, shortened:
User-agent: *
Disallow: /api
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /api
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /api
Which mistake silently changes your rules?
Named groups do not inherit. Under the Robots Exclusion Protocol, a crawler obeys only the most specific group that names it, and Google's documentation says the other groups are ignored.
So this file opens /admin/ to GPTBot, although the first group closes it:
User-agent: *
Disallow: /admin/
User-agent: GPTBot
Allow: /
Repeat every shared rule inside each named group, as the two files above do. The robots.txt glossary page has the other limits, such as the 500 KiB that Google reads.
What can robots.txt not do?
- It is not a lock. A user agent can be faked. OpenAI and Perplexity publish the IP addresses of their real crawlers so that a visit can be checked.
- It does not remove a page from an index. For that, leave the page crawlable and add noindex.
- It does not open your firewall. Bot protection often challenges requests from data-centre addresses. Perplexity's documentation says a site behind a web application firewall may need to allow its bots explicitly. A crawler that robots.txt welcomes can still receive a 403.
- It is not llms.txt. llms.txt is a reading guide for language models and grants or blocks nothing.
How do you check what the crawlers really get?
- Build the file with the free robots.txt generator, which writes separate groups for search, assistant and training crawlers.
- Open your own
/robots.txtand read it once as each crawler: find the group that names it, and ignore the rest. - Check the status code a crawler receives, not only the rules. AI Visibility Studio reports crawler access for each AI crawler, with search, user and training crawlers kept apart.
- Read your access log. The AI crawler log reader in the SEO tools shows which crawlers came, in your browser.
Letting the crawlers in is the first step of generative engine optimization, not the whole of it: a page still has to answer the question well enough to be quoted. GEO vs SEO covers what comes after access.
FAQ
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot collects pages for model training. ChatGPT search uses OAI-SearchBot, and OpenAI says the two settings are independent. Blocking OAI-SearchBot is what removes a site from ChatGPT search answers.
Is Google-Extended a crawler?
No. Google-Extended is a robots.txt token with no user agent string of its own. It controls whether pages Googlebot crawled are used for Gemini models, and Google says it does not affect inclusion or ranking in Google Search.
Do AI crawlers obey robots.txt?
The training and search crawlers of OpenAI, Anthropic and Perplexity are documented as following it. User-triggered fetchers differ: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
Comments 0
Join the conversation. Log in or create an account to like and comment.
Confirm your email to comment. The link is in your inbox.
No comments yet. Start the conversation.