ClaudeBot: what it is, whether to block it, and how Claude finds websites
ClaudeBot is the crawler Anthropic uses to collect training data, and the one small-business sites refused most often in our check. That is fine on its own. Its two siblings, Claude-SearchBot and Claude-User, are the ones that put you in Claude's answers. What each does, and how to block only the first.
If you have looked at your server logs or your Cloudflare dashboard lately, ClaudeBot may well be near the top. It is Anthropic's crawler, and it was the crawler small-business websites refused most often in our check. Whether that matters depends on which of Anthropic's three bots you are refusing.
Anthropic's three bots
From Anthropic's article on how it crawls the web (Anthropic's help center, dated April 7, 2026):
| Bot | What it does | If you block it |
|---|---|---|
ClaudeBot | Collects public web content that could be used to train Anthropic's models | Your future content is left out of training data |
Claude-SearchBot | Crawls the web to improve Claude's search results | Anthropic stops indexing your content for search, which may reduce your visibility in Claude's answers |
Claude-User | Visits a page when a Claude user's question calls for it | Claude cannot fetch your page for that user, which may reduce your visibility when users search |
So ClaudeBot is the training crawler, and blocking it does not take you out of Claude's answers. Blocking Claude-SearchBot or Claude-User does.
Does ClaudeBot respect robots.txt?
Anthropic says all three bots follow standard robots.txt rules, support the non-standard Crawl-delay rule, and do not try to get past CAPTCHAs. That includes Claude-User, which sets Anthropic apart: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page. A robots.txt that blocks everything will keep Claude from reading your pages even when a user asks it to.
Anthropic also warns that blocking its IP addresses may not work reliably, because it stops the bots from reading your robots.txt in the first place. For Anthropic's bots, robots.txt is the control to use.
How to block ClaudeBot, and only ClaudeBot
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
That opts you out of training while Claude-SearchBot and Claude-User keep reading your site under *. robots.txt covers one hostname, so repeat it on every subdomain you want covered, as Anthropic's own instructions say. The same approach for every AI company.
If ClaudeBot is simply crawling faster than your server likes, slow it down instead of blocking it:
User-agent: ClaudeBot
Crawl-delay: 10
Crawl-delay asks for a pause, in seconds, between requests. Anthropic honors it; Google does not support it.
Why ClaudeBot visits so often
AI crawlers are busy. When Vercel and MERJ measured traffic across Vercel's network in December 2024, Anthropic's crawler made 370 million requests in a month, against 569 million for OpenAI's GPTBot and about 4.5 billion for Googlebot (their study). They also found that none of the major AI crawlers ran JavaScript: they read the HTML your server sends, so a page built by scripts can look empty to them. JavaScript SEO.
What our check found
Of the 315 small-business websites in our study that let a browser in, 51 turned away a request from ClaudeBot, more than any other crawler we tested (GPTBot was next, at 35). In robots.txt, 19 sites disallowed ClaudeBot.
Blocking ClaudeBot is a legitimate choice. The problem is collateral damage: 14 sites also turned away Claude-User, and 5 disallowed it in robots.txt, which keeps Claude from reading those sites for its users. Claude-SearchBot was not one of the nine crawlers in our study. Our checker sends each crawler's published user-agent name from our own server, so a firewall that verifies crawlers by IP address will turn our test away even when the real crawler gets in. Read a block as "your firewall reacts to this name", then confirm in your firewall's event log.
How Claude finds websites for its answers
When Claude searches the web, Anthropic's own crawl is only part of the picture. Anthropic's list of subprocessors names three web search providers, Brave Search, TurboPuffer and Parallel Web Systems (Anthropic's Trust Center), and its help center says Claude's image results come from Bing (web search in Claude). Anthropic does not say how these are combined. What you control is the same as for every assistant: let Claude-SearchBot and Claude-User in, keep your text in the HTML, and say clearly who you are and what you offer.
Questions
What is ClaudeBot?
Anthropic's web crawler for collecting training data for its Claude models. It names itself ClaudeBot in the user-agent of its requests.
Should I block ClaudeBot?
Block it if you do not want your content used to train Anthropic's models; it will not affect your visibility in Claude's answers. Leave Claude-SearchBot and Claude-User alone unless you want out of Claude altogether.
Is that request really from Anthropic?
Anyone can call their bot ClaudeBot. The real one follows robots.txt, so a "ClaudeBot" that ignores yours, or hammers your login page, is an impostor. Services such as Cloudflare check crawlers against their real owners and can block the fakes for you.
Day 14 of 20. See whether ClaudeBot, Claude-User and the other AI crawlers can reach your site with the free AI visibility checker.