AI search optimization for small businesses: what we found checking 400 websites

"AI search optimization" is often sold as a new discipline with new tricks. On the small-business websites we checked, the problems were older and plainer than that: crawlers that were never let in, pages that arrive empty, and businesses that never say who they are in a form a machine can read. Most of the fixes take minutes.

To find out how common those problems are, we ran our AI visibility checker against 400 small-business homepages picked at random on November 3, 2026. 348 answered. This is what it found, how we measured it, and the order we would fix things in.

What AI search optimization means in practice

ChatGPT search, Claude, Perplexity, Google's AI Overviews and Microsoft Copilot differ in a lot of ways, but at the moment that matters to a business they behave alike: when a question needs current information, the assistant looks it up in a search index, reads a handful of pages and writes its answer from them. Your site can only be one of those pages if three things are true.

• Reachable. The crawler that fetches pages for that assistant is allowed in by robots.txt and is not turned away by a firewall.
• Readable. Your words are in the HTML the crawler downloads. Most AI crawlers do not run JavaScript, so text that a script adds after the page loads never reaches them.
• Identifiable. The page says, in plain text and in structured data, which business it belongs to: the name, what you do, where you are and how to reach you.

Reviews, mentions and the usual ranking signals still decide which reachable, readable pages get picked. But a page that fails one of those three is out before that contest starts, and those failures are the ones a site owner can fix this week.

How we checked

• The sample. 400 homepages drawn at random (seed 20261103) from two of our own directories, thrift, vintage and antique stores on thriftstores.io and ice rinks on rinknearme.com, in 48 US states. 348 answered. The other 52 timed out, had lapsed domains or refused every request.
• The checks. The same ones the free checker runs on any page: the robots.txt rules for nine crawlers (OAI-SearchBot, ChatGPT-User, GPTBot, Claude-User, ClaudeBot, PerplexityBot, Google-Extended, Googlebot and Bingbot); the homepage fetched once as each crawler and once as a browser; the words, title and H1 in the raw HTML; JSON-LD structured data; noindex and snippet rules; the canonical tag; an email address in the HTML; HTTPS; and llms.txt.
• A hard problem is one that can keep a page out of AI answers on its own: a robots.txt rule against an AI search crawler, a robots.txt that will not load, a firewall that lets a browser in but turns AI search crawlers away, a homepage with almost no text in its HTML, or a noindex tag.
• Limits. Homepages only, one day, and local retail and recreation businesses rather than every kind of company. Our checker sends each crawler's published user-agent name from our own server, so a firewall that verifies crawlers by IP address will turn our test away even when the real crawler gets in. Read a block as "your firewall reacts to this name", then confirm in your firewall's event log.

The results

No email address on the homepage66%

No structured data (JSON-LD)50%

Under 300 words in the HTML41%

At least one hard problem24%

Nearly blank without JavaScript16%

robots.txt blocks AI search, or will not load7%

Firewall stops AI search crawlers5.5%

Passed every check19%

Finding	Sites	Share
At least one hard problem	85 of 348	24%
Passed every check	59 of 315	19%
Nearly blank without JavaScript (under 80 words in the HTML)	49 of 315	16%
Under 300 words in the HTML	130 of 315	41%
No structured data (JSON-LD)	158 of 315	50%
No email address on the homepage	208 of 315	66%
Email shown only through Cloudflare's obfuscation	14 of 315	4%
robots.txt blocks an AI search crawler, or will not load	24 of 348	7%
Firewall lets a browser in but stops AI search crawlers	19 of 348	5.5%
Firewall stopped even our browser-style request	33 of 348	9%
Homepage says noindex	5 of 315	1.6%
Canonical tag points to a different URL	15 of 315	5%
Still served over plain HTTP	32 of 348	9%
Has an llms.txt file	78 of 348	22%

Five things the numbers say

1. Firewalls turn AI crawlers away more often than robots.txt does

Only 7 sites had a robots.txt rule against an AI search crawler. Many more refused crawlers at the door. ClaudeBot's request was turned away by 51 of the 315 sites that let our browser-style request in, and GPTBot's by 35, mostly with HTTP 403. Those two collect training data, and blocking them is a fair choice that does not take you out of AI search. The crawlers that decide whether you can be found, OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot, were each refused by 14 to 17 sites.

Requests named Googlebot and Bingbot were refused by 17 and 18. Good firewalls refuse anything that only claims to be Googlebot, which is exactly what our test is, so treat those numbers as "this firewall reacts to the name", and check your own firewall log before you change anything. The Cloudflare settings behind most of these blocks.

2. One homepage in six is nearly blank to most AI crawlers

49 homepages (16%) had fewer than 80 words in their HTML, and some had none. In a browser they look normal, because JavaScript fills them in after the page loads. Google runs that JavaScript before it indexes a page. When Vercel and MERJ studied AI crawler traffic in December 2024, none of the major AI crawlers rendered JavaScript (Gemini, which uses Googlebot, and Applebot were the exceptions), so they receive the empty shell. The median homepage carried 307 words in its HTML. How to tell whether your words are in the HTML.

3. Most businesses never say who they are in a form machines read

Half the homepages had no JSON-LD structured data. On the ones that did, the most common type was WebSite (129 homepages), which most SEO plugins add on their own. Organization appeared on 77, and LocalBusiness, the type that carries a shop's address, phone number and opening hours, on only 37. Here is the block we would add.

4. Two in three put no email address on the homepage

208 homepages (66%) had no email address in their HTML at all. On 14 more, Cloudflare's email obfuscation had replaced it with "[email protected]", which is all a crawler that skips JavaScript reads. An assistant asked how to contact a business can only answer from what it can read. What "[email protected]" means and how to turn it off.

5. llms.txt made no difference

78 sites (22%) had an llms.txt file, the proposed guide for AI models. They had a hard problem exactly as often as the sites without one: 19 of 78 (24.4%) against 66 of 270 (24.4%). A file that describes your site does not help if the crawlers cannot get in, or find nothing to read when they do. What llms.txt does and does not do.

What to fix first

1. robots.txt. Make sure it loads (17 sites answered our request for it with an error, 15 of them "429 Too Many Requests") and that it does not block OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot or Bingbot. Templates you can copy.
2. Your firewall. Check that your host, CDN or security plugin is not turning those crawlers away. Its event log shows every refused request.
3. Text in the HTML. Open your homepage, view the page source, and search for a sentence you can see on the page. If it is not there, ask your developer about server rendering or prerendering.
4. noindex. One WordPress checkbox switches it on for a whole site. How to check yours.
5. The canonical tag. It should point to the page itself. Of the 15 that pointed elsewhere, 3 named the site's old http:// address and 4 named a staging or preview address left over from the build, on hosts like netlify.app and github.io.
6. Structured data. An Organization or LocalBusiness block with your name, address, phone number, logo and the profiles you own elsewhere.
7. Contact details as plain text, not inside an image and not behind obfuscation.

After that comes the slower work that has always decided who gets picked: being mentioned and reviewed elsewhere, answering the questions your customers actually ask, and being in Bing's index as well as Google's. Why Bing matters more than it used to.

The series

Over the next 20 days we are publishing one post on each fix, with a check you can run in a minute. Each title below turns into a link on the day it goes live:

• ChatGPT SEO: how ChatGPT search finds and picks pages
• robots.txt examples you can copy, AI crawlers included
• JavaScript SEO: why most AI crawlers miss script-built pages
• Cloudflare settings that hide sites from Google and AI
• How to block AI training without leaving AI search
• The WordPress checkbox that hides a whole site
• What "[email protected]" means on a website
• Schema markup for small businesses, with a 13-line example
• Bing SEO: why Bing decides what Copilot can find
• Do you need llms.txt?
• React SEO: how our own homepage showed Google an empty page
• Why is my website not showing up on Google?
• What is ClaudeBot?
• GPTBot, OAI-SearchBot and ChatGPT-User explained
• How to optimize for Google AI Overviews
• Where does ChatGPT get its information?
• Perplexity SEO: how Perplexity finds and cites websites
• "Blocked by robots.txt" in Search Console, fixed
• How to check if ChatGPT recommends your business

Questions

Is AI search optimization different from SEO?

Less than the name suggests. Google says there are no extra requirements for appearing in AI Overviews or AI Mode: a page has to be indexed and eligible to be shown with a snippet, and the usual guidance applies (AI features and your website). The differences are in the details. There are more crawlers to let in, because each AI company runs its own; there is less tolerance for pages that need JavaScript; and there is more reward for stating plain facts that an answer can quote.

Can I submit my website to ChatGPT?

There is no form. ChatGPT search finds pages with its own crawler, OAI-SearchBot, and sometimes through outside search providers (OpenAI's business documentation names Bing). So the practical steps are to let OAI-SearchBot in, make sure Bing has indexed you, and make sure crawlers find text when they arrive. What else helps.

How do I check my own site?

Paste your homepage into the free AI visibility checker. It runs every check in this study, shows what each crawler received, and explains each problem in plain English. It is free, and there is nothing to sign up for.