robots.txt examples: 6 safe templates for small-business websites, AI crawlers included

A robots.txt file is a short list of rules at the root of your site that tells crawlers where they may go. Most small-business sites need three or four lines. Most of the problems we see come from copying the wrong ones from a forum, a plugin or an old guide.

In our check of 348 small-business sites, 7 had a rule blocking an AI search crawler, 4 blocked Googlebot from their own homepage, and 17 had a robots.txt that would not load at all. Here are six templates that avoid all of that, what every line does, and how to check yours.

How to find any site's robots.txt

Type the domain followed by /robots.txt into your browser, for example www.example.com/robots.txt. It is a public file, so you can read anyone's. If you get a 404 page, the site has none, which is fine: Google treats a missing file "as if a valid robots.txt file didn't exist" and crawls everything (28 of the sites we checked had none).

How the rules work, in five sentences

From Google's robots.txt specification:

• Rules come in groups, each starting with a User-agent: line that names a crawler, or * for every crawler.
• Each crawler obeys only one group, the most specific one that matches its name, and ignores the rest. The name match is case-insensitive.
• Within the group, the rule with the longest matching path wins, and on a tie the least restrictive one does.
• Google reads the first 500 KiB of the file and caches it for up to 24 hours.
• robots.txt controls crawling, not indexing. To keep a page out of Google, use a noindex tag instead.

The crawlers worth knowing in 2026

Name in robots.txt	Who	What it does
Googlebot	Google	Google Search, including AI Overviews and AI Mode
Bingbot	Microsoft	Bing search, which Copilot answers from
OAI-SearchBot	OpenAI	Finds pages for ChatGPT search
ChatGPT-User	OpenAI	Fetches a page when a ChatGPT user asks for it
GPTBot	OpenAI	Collects pages for training OpenAI's models
Claude-SearchBot	Anthropic	Finds pages for Claude's search
Claude-User	Anthropic	Fetches a page when a Claude user asks for it
ClaudeBot	Anthropic	Collects pages for training Anthropic's models
PerplexityBot	Perplexity	Finds pages for Perplexity's answers
Google-Extended	Google	Not a crawler but a robots.txt name: controls use for Gemini training and grounding

More detail on each company's crawlers: OpenAI's and Anthropic's.

1. Allow everything (the right default for most sites)

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Every crawler may fetch every page, and the Sitemap line tells them where your list of pages is. If you have nothing to hide from crawlers, this is the whole file.

2. WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

This is what WordPress itself generates when there is no robots.txt file on disk: keep crawlers out of the admin area, but let them reach admin-ajax.php, which some themes use to load content. If Yoast or Rank Math makes your sitemap, use its address instead (usually /sitemap_index.xml). Do not block /wp-content/ or /wp-includes/: those hold the CSS and JavaScript Google needs to render your pages.

3. An online store

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Cart, checkout, account and internal search pages are private or endless, so crawling them wastes a crawler's time. Shopify writes rules like these for you; change them only through its robots.txt.liquid template.

4. Opt out of AI training, stay in AI search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Each named group applies only to that crawler, so OpenAI's, Anthropic's and Google's training are opted out, along with Common Crawl's CCBot, whose archive many models are trained on. Every other crawler, including OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, falls under * and is welcome. Google says disallowing Google-Extended does not affect Google Search. The full guide to blocking AI training.

Be careful with copied "block all AI bots" lists. Of the 20 sites in our check with rules against AI training crawlers, 6 also blocked an AI search crawler, which takes them out of the answers they presumably still wanted to appear in.

5. A staging or test site

User-agent: *
Disallow: /

This asks every crawler to stay out, which is what you want while a copy of the site is being built. It is not enough on its own: a blocked URL can still be listed if someone links to it, so put staging sites behind a password as well. Above all, never let this file reach the live site. 3 of the sites we checked were blocking every crawler.

6. What not to write

• Disallow: / under User-agent: * on a live site. It blocks Google, Bing and every AI crawler.
• A group for Googlebot that you forget to finish. Googlebot then ignores the * group entirely, including its Allow lines.
• noindex lines. Google does not support noindex in robots.txt. Use the meta tag.
• Crawl-delay to slow Google down. Google ignores it; Bing reads it.
• Blocking CSS, JavaScript or image folders. Google renders pages like a browser and needs them.
• Rules for pages you want removed. If a page is blocked, Google never sees its noindex, so it can stay listed.

Test it before you publish

1. Open your live file at /robots.txt and read it top to bottom. If you use Cloudflare, it may add rules of its own in front of yours.
2. In Google Search Console, Settings, robots.txt report shows the copy Google fetched and any errors.
3. URL Inspection on your key pages should never say "Blocked by robots.txt". What to do if it does.
4. The free AI visibility checker reads your robots.txt and reports, for Googlebot, Bingbot and seven AI crawlers, whether each may fetch the page.

Day 3 of 20, from our check of 400 small-business websites.