Technical · free tool
Build a robots.txt, with rules for the AI crawlers.
Choose what the AI crawlers may read, list the paths no crawler should fetch, add your sitemap address, and copy a valid robots.txt. Nothing you type leaves your browser.
How it works
Search and training are separate choices
AI companies run different crawlers for different jobs. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the index their assistants search; GPTBot and ClaudeBot collect text that may be used to train models, and Google-Extended is the token that controls that use at Google. Blocking training does not remove a page from AI search answers; blocking the search crawlers does.
What robots.txt does not do
It asks crawlers not to fetch a path; it does not hide a page. A blocked page can still appear in search if other sites link to it, and a firewall or bot challenge can turn a crawler away whatever robots.txt says. To keep a page out of search, let it be fetched and give it a noindex tag.
Where it goes
Save the text as robots.txt at the root of your site, so it answers at https://your-site.com/robots.txt. Each subdomain needs its own file.
Questions
Should I block AI crawlers?
It depends on what you want. If you want to be found and cited in AI search answers, allow the search crawlers. Whether your text may be used to train models is a separate decision, and this tool lets you make it on its own.
Does blocking GPTBot remove my site from ChatGPT search?
No. ChatGPT search uses OAI-SearchBot. GPTBot is the crawler for model training.
How do I check what my robots.txt allows today?
The free page check on the SEOAnything home page reads your robots.txt and shows which AI search crawlers it lets in.
Do more with an account
These tools check one text or one address at a time. A free SEOAnything account audits whole pages, crawls a site, reads your Search Console numbers and drafts posts; every feature and its limits are written down.
Updated 2026-11-16 · more free tools: serp preview · url slug generator · word counter and readability · keyword density checker · redirect checker · http header checker · meta tag checker · structured data checker