Skip to content
Get a free mockup
Menu

ClaudeBot: what it is, whether to block it, and how Claude finds websites

Haider Ali· · 4 min read

ClaudeBot is the crawler Anthropic uses to collect training data, and the one small-business sites refused most often in our check. That is fine on its own. Its two siblings, Claude-SearchBot and Claude-User, are the ones that put you in Claude's answers. What each does, and how to block only the first.

If you have looked at your server logs or your Cloudflare dashboard lately, ClaudeBot may well be near the top. It is Anthropic's crawler, and it was the crawler small-business websites refused most often in our check. Whether that matters depends on which of Anthropic's three bots you are refusing.

Anthropic's three bots

From Anthropic's article on how it crawls the web (Anthropic's help center, dated April 7, 2026):

BotWhat it doesIf you block it
ClaudeBotCollects public web content that could be used to train Anthropic's modelsYour future content is left out of training data
Claude-SearchBotCrawls the web to improve Claude's search resultsAnthropic stops indexing your content for search, which may reduce your visibility in Claude's answers
Claude-UserVisits a page when a Claude user's question calls for itClaude cannot fetch your page for that user, which may reduce your visibility when users search

So ClaudeBot is the training crawler, and blocking it does not take you out of Claude's answers. Blocking Claude-SearchBot or Claude-User does.

Does ClaudeBot respect robots.txt?

Anthropic says all three bots follow standard robots.txt rules, support the non-standard Crawl-delay rule, and do not try to get past CAPTCHAs. That includes Claude-User, which sets Anthropic apart: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. A robots.txt that blocks everything will keep Claude from reading your pages even when a user asks it to.

Anthropic also warns that blocking its IP addresses may not work reliably, because it stops the bots from reading your robots.txt in the first place. For Anthropic's bots, robots.txt is the control to use.

How to block ClaudeBot, and only ClaudeBot

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

That opts you out of training while Claude-SearchBot and Claude-User keep reading your site under *. robots.txt covers one hostname, so repeat it on every subdomain you want covered, as Anthropic's own instructions say. The same approach for every AI company.

If ClaudeBot is simply crawling faster than your server likes, slow it down instead of blocking it:

User-agent: ClaudeBot
Crawl-delay: 10

Crawl-delay asks for a pause, in seconds, between requests. Anthropic honors it; Google does not support it.

Why ClaudeBot visits so often

AI crawlers are busy. When Vercel and MERJ measured traffic across Vercel's network in December 2024, Anthropic's crawler made 370 million requests in a month, against 569 million for OpenAI's GPTBot and about 4.5 billion for Googlebot (their study). They also found that none of the major AI crawlers ran JavaScript: they read the HTML your server sends, so a page built by scripts can look empty to them. JavaScript SEO.

What our check found

Of the 315 small-business websites in our study that let a browser in, 51 turned away a request from ClaudeBot, more than any other crawler we tested (GPTBot was next, at 35). In robots.txt, 19 sites disallowed ClaudeBot.

Blocking ClaudeBot is a legitimate choice. The problem is collateral damage: 14 sites also turned away Claude-User, and 5 disallowed it in robots.txt, which keeps Claude from reading those sites for its users. Claude-SearchBot was not one of the nine crawlers in our study. Our checker sends each crawler's published user-agent name from our own server, so a firewall that verifies crawlers by IP address will turn our test away even when the real crawler gets in. Read a block as "your firewall reacts to this name", then confirm in your firewall's event log.

How Claude finds websites for its answers

When Claude searches the web, Anthropic's own crawl is only part of the picture. Anthropic's list of subprocessors names three web search providers, Brave Search, TurboPuffer and Parallel Web Systems (Anthropic's Trust Center), and its help center says Claude's image results come from Bing (web search in Claude). Anthropic does not say how these are combined. What you control is the same as for every assistant: let Claude-SearchBot and Claude-User in, keep your text in the HTML, and say clearly who you are and what you offer.

Questions

What is ClaudeBot?

Anthropic's web crawler for collecting training data for its Claude models. It names itself ClaudeBot in the user-agent of its requests.

Should I block ClaudeBot?

Block it if you do not want your content used to train Anthropic's models; it will not affect your visibility in Claude's answers. Leave Claude-SearchBot and Claude-User alone unless you want out of Claude altogether.

Is that request really from Anthropic?

Anyone can call their bot ClaudeBot. The real one follows robots.txt, so a "ClaudeBot" that ignores yours, or hammers your login page, is an impostor. Services such as Cloudflare check crawlers against their real owners and can block the fakes for you.

Day 14 of 20. See whether ClaudeBot, Claude-User and the other AI crawlers can reach your site with the free AI visibility checker.

Want to know what AI crawlers see on your site? Our free AI visibility checker runs these checks against any page in about 10 seconds, no signup. If it finds something you'd rather hand over, our AI SEO service fixes it.

More from the blog

Get a free homepage mockup

Tell us about your business and we'll design a custom homepage concept within 48 hours — free, no strings attached.