robots.txt for AI Crawlers: Which Bots to Allow or Block

Which AI crawlers to allow and which to block in robots.txt: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, with files you can copy.

To be cited by AI assistants, allow their search crawlers in robots.txt. Blocking a training crawler such as GPTBot does not remove a site from ChatGPT search; blocking OAI-SearchBot does. This guide sorts the crawlers by job, gives two files you can copy, and names the one mistake that silently opens or closes a site.

What are the three kinds of AI crawler?

AI crawlers do three different jobs, and each job has its own user agent:

  • Training crawlers collect pages that may be used to train models.
  • Search crawlers build the index an assistant searches when it answers.
  • User-triggered fetchers open one page because a person asked about it or pasted its link.

A rule for one kind does nothing to the others. OpenAI's documentation says each setting is independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot.

Which user agent belongs to which company?

CompanyTrainingSearchA person's request
OpenAIGPTBotOAI-SearchBotChatGPT-User
AnthropicClaudeBotClaude-SearchBotClaude-User
PerplexitynonePerplexityBotPerplexity-User
GoogleGoogle-ExtendedGooglebotnone
AppleApplebot-ExtendedApplebotnone
MetaMeta-ExternalAgentnoneMeta-ExternalFetcher
Common CrawlCCBotnonenone
MicrosoftnoneBingbotnone

Two rows need a note. Google-Extended is not a crawler: Google says it has no user agent string of its own and is a token that controls whether pages Googlebot already crawled are used for Gemini models. PerplexityBot is a search crawler: Perplexity says it is not used to crawl content for AI foundation models.

The names come from each company's own page: OpenAI, Anthropic, Perplexity and Google.

What happens when you block each kind?

Blocking a training crawler keeps your pages out of future training. It does not remove you from search. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal.

Blocking a search crawler removes you from that assistant's answers. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic says disabling Claude-SearchBot may reduce a site's visibility in search responses.

Blocking a user-triggered fetcher has uneven results. Anthropic says disabling Claude-User stops Claude retrieving the page for a user's question. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person started the action, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason.

A change is not instant. OpenAI says its systems can take about 24 hours to follow a robots.txt update.

A robots.txt that allows AI search and blocks training

This file keeps one private folder closed to everyone, lets the search crawlers and fetchers read the rest, and opts out of training:

User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

A robots.txt that allows everything public

A site that wants the widest reach lets all three kinds in and closes only what is private. SEOAnything's own robots.txt does this: it lists the search crawlers, the fetchers and the training crawlers in three groups, and every group repeats the same rules, which leave everything open except /api. The pattern, shortened:

User-agent: *
Disallow: /api

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /api

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /api

Which mistake silently changes your rules?

Named groups do not inherit. Under the Robots Exclusion Protocol, a crawler obeys only the most specific group that names it, and Google's documentation says the other groups are ignored.

So this file opens /admin/ to GPTBot, although the first group closes it:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Allow: /

Repeat every shared rule inside each named group, as the two files above do. The robots.txt glossary page has the other limits, such as the 500 KiB that Google reads.

What can robots.txt not do?

  • It is not a lock. A user agent can be faked. OpenAI and Perplexity publish the IP addresses of their real crawlers so that a visit can be checked.
  • It does not remove a page from an index. For that, leave the page crawlable and add noindex.
  • It does not open your firewall. Bot protection often challenges requests from data-centre addresses. Perplexity's documentation says a site behind a web application firewall may need to allow its bots explicitly. A crawler that robots.txt welcomes can still receive a 403.
  • It is not llms.txt. llms.txt is a reading guide for language models and grants or blocks nothing.

How do you check what the crawlers really get?

  1. Build the file with the free robots.txt generator, which writes separate groups for search, assistant and training crawlers.
  2. Open your own /robots.txt and read it once as each crawler: find the group that names it, and ignore the rest.
  3. Check the status code a crawler receives, not only the rules. AI Visibility Studio reports crawler access for each AI crawler, with search, user and training crawlers kept apart.
  4. Read your access log. The AI crawler log reader in the SEO tools shows which crawlers came, in your browser.

Letting the crawlers in is the first step of generative engine optimization, not the whole of it: a page still has to answer the question well enough to be quoted. GEO vs SEO covers what comes after access.

FAQ

No. GPTBot collects pages for model training. ChatGPT search uses OAI-SearchBot, and OpenAI says the two settings are independent. Blocking OAI-SearchBot is what removes a site from ChatGPT search answers.

Is Google-Extended a crawler?

No. Google-Extended is a robots.txt token with no user agent string of its own. It controls whether pages Googlebot crawled are used for Gemini models, and Google says it does not affect inclusion or ranking in Google Search.

Do AI crawlers obey robots.txt?

The training and search crawlers of OpenAI, Anthropic and Perplexity are documented as following it. User-triggered fetchers differ: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

Share this post

Comments 0

    Publish yours next.

    Write it yourself, or turn a rising search into a scored draft. Your readers need no account.

    Write a post

    Research your own topic: see what each market searches today, find the words people type with keyword research, check a page with the free SEO tools, or look up a term in the SEO glossary.

    More from the blog