Cloudflare SEO: the settings that can hide your site from Google and AI search
A few Cloudflare settings decide whether search engines and AI assistants can read your site, and several changed in 2026: blocking AI training can now stop Googlebot too. The five settings that matter, what each does today, and a ten-minute check.
Many small-business websites sit behind Cloudflare, often switched on by a web designer years ago and never looked at since. A handful of its settings decide whether Google, Bing and AI assistants can read your pages at all. In 2026 several of them changed.
In our check of 348 small-business sites, 19 let a browser in but turned AI search crawlers away, and 33 refused even our browser-style request. Some of those refusals came back as Cloudflare challenge pages. Here are the five settings to check, what each one does now, and how to tell if it is costing you.
1. AI crawler and agent access
This is the setting Cloudflare used to call "Block AI bots". Its current version, documented as AI Crawler and Agent access policies and managed from AI Crawl Control, sorts AI crawlers into three behaviors: Search, Agent and Training. For each you can choose Allow, Block, Block on pages with ads, or, for Training only, Disallow, which is published through robots.txt.
Two facts matter most:
- The default changed back. In July 2025 Cloudflare made blocking AI crawlers the starting point for new domains. Its documentation, updated on November 2, 2026, now says that for new domains Search, Agents and Training all start at "Allow". If your site was set up in between, check what it is set to today.
- "Block" on Training also blocks Google. Since September 15, 2026, Cloudflare treats Googlebot, Bingbot and Applebot as crawlers that serve both search and AI. Under Disallow AI Training they keep crawling your site for search; in Cloudflare's words, "selecting Block stops them entirely." Sites that had the old toggle on were migrated to Disallow AI Training.
What we would set for a business that wants to be found: Search on Allow. Agent on Allow, or Block on pages with ads if you run ads. Training is your choice, but if you want to opt out, use Disallow AI Training, never Block.
2. Bot Fight Mode
Bot Fight Mode is the free plan's one-switch bot defence. It picks out traffic that looks like known bots and answers it with computationally expensive challenges. Two details make it risky for visibility:
- Cloudflare says it cannot be skipped or bypassed with WAF custom rules or Page Rules, so there is no way to make an exception for a crawler you want.
- Cloudflare's current page does not say whether verified search crawlers such as Googlebot are exempt, so do not assume they are.
If it is on, check Security Events for challenged requests from Googlebot, Bingbot, OAI-SearchBot, Claude-User or PerplexityBot. If you see them and need finer control, Super Bot Fight Mode on paid plans supports skip rules.
3. Firewall rules and rate limits
Custom rules that block by user agent, country or network, and rate limits set low to stop scrapers, catch crawlers too. The most self-defeating version is a limit that covers /robots.txt. In our study, 17 sites would not serve their robots.txt, and 15 of them answered "429 Too Many Requests". Google treats a robots.txt that answers 429 or a 5xx error as a server error: it pauses crawling the site, then falls back on the last copy it fetched (how Google reads robots.txt).
Look through your custom rules for anything matching bot, crawler or a list of user agents, and make sure /robots.txt and /sitemap.xml are never rate limited.
4. Email Address Obfuscation
This is part of Scrape Shield, and Cloudflare switches it on automatically when you sign up. It replaces every email address in your HTML with a link that reads "[email protected]" and adds a small script that turns it back into your address in the visitor's browser (Cloudflare's documentation).
Cloudflare does not say this affects search. Our reading is simpler: a crawler that does not run JavaScript reads "[email protected]" and nothing else, so an AI assistant asked how to contact you cannot answer from your site. 14 of the homepages we checked showed their email only this way. You can turn the feature off under Scrape Shield, or wrap your contact details in <!--email_off--> and <!--/email_off--> comments so Cloudflare leaves them alone. More on what "[email protected]" means.
5. The robots.txt Cloudflare adds for you
Cloudflare can put its own rules in front of your robots.txt. Its managed robots.txt disallows training crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider, and adds a content-signals line. It is being replaced by Bot Preference Sync, which writes the same kind of rules when Training is set to Disallow.
None of that blocks search crawlers, and Cloudflare notes that robots.txt is voluntary for crawlers anyway. But it means the robots.txt the world sees may not be the one you wrote. Open yourdomain/robots.txt in a browser and read what is actually served.
One more: "I'm Under Attack" mode
This mode shows every visitor a check page before the site loads. Crawlers cannot pass it. It is meant for the hours of an attack, and it is easy to forget it is on. If your site still shows that check page, switch it off.
How to check all of this in ten minutes
- In the Cloudflare dashboard, open AI Crawl Control and note the Search, Agent and Training settings.
- Open Security, Events and filter by user agent: Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot. Any blocks or challenges you did not intend show up here.
- Check Bot Fight Mode, Scrape Shield and your custom rules and rate limits against the sections above.
- Read your live
/robots.txt. - Run the free AI visibility checker on your homepage. It fetches the page as each crawler and as a browser, and tells you which ones are turned away.
Then confirm with Google: in Search Console, URL Inspection, "Test live URL" shows whether Googlebot can fetch the page right now. The study behind the numbers in this post: what we found checking 400 small-business websites.