Skip to content
Get a free mockup
Menu

How to block AI crawlers from training on your site, without leaving AI search

Haider Ali· · 5 min read

AI companies use separate crawlers for training their models and for answering questions, and you can block the first while welcoming the second. Which names are which, the robots.txt to copy, Google-Extended explained, and the Cloudflare setting that can take you out of Google.

"Block AI bots" can mean two very different things. One is keeping your content out of the data AI companies train their models on. The other is stopping AI assistants from reading your pages when they answer someone's question. The first is a reasonable choice for many sites. The second takes you out of ChatGPT, Claude, Perplexity and Copilot answers, which is where more and more customers now look.

The two are controlled separately, and they are easy to mix up. In our check of 348 small-business websites, 20 had robots.txt rules against AI training crawlers, and 6 of those 20 also blocked an AI search crawler, most likely by copying a "block all AI bots" list. Here is how to block the first without the second.

Training crawlers and search crawlers

CompanyTraining: fine to blockSearch and answers: block these and you disappear
OpenAIGPTBotOAI-SearchBot, ChatGPT-User
AnthropicClaudeBotClaude-SearchBot, Claude-User
GoogleGoogle-Extended, a robots.txt name rather than a crawlerGooglebot, which feeds Search, AI Overviews and AI Mode
AppleApplebot-ExtendedApplebot
Common CrawlCCBot, whose open archive many models are trained onNone
Metameta-externalagentNone in this list
PerplexityNone: Perplexity says PerplexityBot is not used to train AI modelsPerplexityBot, Perplexity-User

Sources: each company's crawler documentation (OpenAI, Anthropic, Google, Perplexity). Every name in the training column is also in the opt-out list of Cloudflare's managed robots.txt.

The robots.txt to copy

# Opt out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# Everyone else, AI search crawlers included, is welcome
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

A crawler follows only the group that names it most specifically, so the training crawlers read their own group and stay out, while OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot fall under * and keep reading your site. The file lives at yourdomain/robots.txt; on WordPress, Yoast and Rank Math both have an editor for it. More robots.txt templates.

What to leave alone

  • OAI-SearchBot. Block it and OpenAI leaves you out of ChatGPT's search answers, apart from plain navigational links.
  • Claude-SearchBot and Claude-User. Anthropic says disabling either may reduce your visibility in Claude's answers.
  • PerplexityBot. Perplexity says this is the crawler that surfaces and links websites in its results.
  • Googlebot and Bingbot. Google's AI Overviews and AI Mode are part of Search, and Microsoft says Copilot is built on Bing's crawl and index.

Sites in our check blocked these far less often than the training crawlers, but not never. In robots.txt, GPTBot and ClaudeBot were each disallowed by 19 sites, against 7 for ChatGPT-User, 5 for Claude-User, 5 for PerplexityBot and 4 for OAI-SearchBot.

Google-Extended, explained

Google-Extended is the name most people get wrong. It is not a crawler: Googlebot does the crawling, and Google-Extended is a robots.txt name that tells Google whether what it crawls may be used to train future Gemini models and to ground answers in the Gemini apps and Vertex AI (Google's common crawlers). Google says it has no effect on inclusion or ranking in Google Search.

It follows that blocking Google-Extended does not take you out of AI Overviews or AI Mode, because those are part of Search. To leave them, Search Console has a Search generative AI control, open to all sites since August 31, 2026, which removes a site from AI Overviews, AI Mode and Discover's AI features, along with all the traffic they send. The AI Overviews controls compared.

Microsoft and Bing

Bingbot collects pages for both search and AI. On October 30, 2026, Bing announced separate preferences for AI training and for grounding, which means using content to inform AI-generated answers in Bing and Copilot (Bing's announcement). They replace NOARCHIVE and NOCACHE as AI controls: after March 1, 2027, Bing will stop reading those two tags as training or grounding preferences. The new preferences do not change whether you appear in normal Bing results. Check Bing's post for the exact syntax before you add them.

Cloudflare: the setting that can block Google

If your site is on Cloudflare, its AI crawler controls sort bots into Search, Agent and Training (Cloudflare's documentation). Since September 15, 2026, Cloudflare counts Googlebot, Bingbot and Applebot as mixed-purpose crawlers, because they feed both search and AI. Under Training, Disallow AI Training lets them keep crawling for search, while Block stops them altogether and takes you out of Google and Bing (Cloudflare's announcement). Cloudflare also notes that Disallow AI Training does not automatically pass a no-training preference to Bing through robots.txt. The other Cloudflare settings to check.

What robots.txt cannot do

  • It is a request, not a lock. Cloudflare's documentation is plain that following robots.txt is voluntary. The large AI companies say their crawlers respect it; not every crawler does.
  • Visits a person asks for follow different rules. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person requested the page. Anthropic is the exception: it says Claude-User follows robots.txt.
  • It only works going forward. Anthropic, for example, describes blocking ClaudeBot as a signal that your future content should be left out of training. Pages already collected are not removed.
  • It covers one hostname. shop.example.com needs its own robots.txt.

To refuse requests outright you need a firewall rule, such as Cloudflare's AI Crawl Control or your host's bot settings. Use the same split there: refuse training crawlers if you like, never the search ones. Firewalls are where we saw the most blocking: of the 315 sites that let a browser in, 51 turned away ClaudeBot's request and 35 GPTBot's, but 15 also turned away OAI-SearchBot and 17 PerplexityBot. Our checker sends each crawler's published user-agent name from our own server, so a firewall that verifies crawlers by IP address will turn our test away even when the real crawler gets in. Read a block as "your firewall reacts to this name", then confirm in your firewall's event log.

Should a small business block AI training at all?

It is your call, and either answer is reasonable. If your content is the product, such as original articles, research or photography, opting out of training protects it. If your site mainly exists to bring in customers, there is less at stake either way. Opting out of training does not change your eligibility for AI search answers, which come from the search crawlers above. The one mistake to avoid is blocking those by accident.

Questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI treats the two settings as independent.

Will blocking AI crawlers hurt my Google rankings?

Blocking GPTBot, ClaudeBot, CCBot or Google-Extended does not affect Google Search; for Google-Extended, Google says so itself. Blocking Googlebot does, and so does choosing Block for Training on Cloudflare.

How do I block all AI bots?

With a robots.txt group for every name above, but you would also leave ChatGPT, Claude, Perplexity and Copilot answers. Block the training crawlers only, unless leaving AI search is what you want.

Day 6 of 20. See which crawlers your robots.txt and firewall let in, training and search separately, with the free AI visibility checker.

Want to know what AI crawlers see on your site? Our free AI visibility checker runs these checks against any page in about 10 seconds, no signup. If it finds something you'd rather hand over, our AI SEO service fixes it.

More from the blog

Get a free homepage mockup

Tell us about your business and we'll design a custom homepage concept within 48 hours — free, no strings attached.