Skip to content
Get a free mockup
Menu

GPTBot, OAI-SearchBot and ChatGPT-User: OpenAI's crawlers explained

Haider Ali· · 3 min read

OpenAI sends four bots to websites. GPTBot collects training data, OAI-SearchBot finds pages for ChatGPT search, ChatGPT-User visits on a person's behalf and OAI-AdsBot checks ads. What each does, the exact user agents, the robots.txt for each choice, and how to spot a fake.

OpenAI sends four different bots to websites, and they do very different jobs. Block the wrong one and you leave ChatGPT's search answers; block the right one and nothing changes except that your pages stay out of OpenAI's training data. Here is each bot with its exact user-agent, the robots.txt for each choice, and how to tell a real OpenAI request from a fake.

OpenAI's bots at a glance

BotJobFollows robots.txt?
OAI-SearchBotFinds pages to show in ChatGPT search. Block it and you are left out of ChatGPT's search answersYes
GPTBotCollects content that may be used to train OpenAI's foundation modelsYes
ChatGPT-UserVisits pages on a person's behalf, when a question in ChatGPT or a custom GPT's action needs it. Does not crawl on its ownNot necessarily: OpenAI says robots.txt rules may not apply
OAI-AdsBotChecks the safety of pages submitted as ads on ChatGPTNot stated; it only visits pages submitted as ads

Source: OpenAI's crawler documentation, which stresses that the settings are independent: a site can allow OAI-SearchBot to appear in search and disallow GPTBot to stay out of training. If you allow both, OpenAI may use one crawl for both purposes rather than visiting twice.

The user-agent strings

OpenAI gives these as examples and says the version numbers may change:

OAI-SearchBot: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBot:        Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User:  Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot:    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Match on the bot's name, not the whole string, so your rule keeps working when the version changes.

robots.txt for each choice

In ChatGPT search, out of training (the usual choice)

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Everything allowed

User-agent: *
Allow: /

You do not need to mention OpenAI at all. If your robots.txt never names its bots, they follow your * rules.

Out of ChatGPT search as well

User-agent: OAI-SearchBot
Disallow: /

User-agent: GPTBot
Disallow: /

OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT's search answers, though they may still appear as navigational links, and ChatGPT-User may still visit when a person asks about a page. Changes reach ChatGPT search in about 24 hours.

Is it really OpenAI?

Anyone can put "GPTBot" in a user-agent. OpenAI publishes the IP address ranges each of its bots uses, linked from its crawler documentation, and asks sites to accept OAI-SearchBot requests from them. To check a request, compare its IP address with the published list for that bot; Cloudflare and other services that verify crawlers do this for you. Treat a "GPTBot" from any other address as a stranger.

What our check found

BotDisallowed in robots.txt (of 348 sites)Turned away by the firewall (of 315)
GPTBot1935
ChatGPT-User714
OAI-SearchBot415

Most of the blocking was aimed at GPTBot, as you would expect. But 15 sites also turned away OAI-SearchBot, which keeps them out of ChatGPT search whether the owner meant it or not. Our checker sends each crawler's published user-agent name from our own server, so a firewall that verifies crawlers by IP address will turn our test away even when the real crawler gets in. Read a block as "your firewall reacts to this name", then confirm in your firewall's event log.

Why GPTBot visits so often

In Vercel and MERJ's December 2024 measurement across Vercel's network, GPTBot made 569 million requests in a month, about an eighth of Googlebot's 4.5 billion, and none of the major AI crawlers they tested, OpenAI's included, ran JavaScript (their study). If GPTBot's load is the problem, blocking it costs you nothing in search. If OAI-SearchBot finds an empty page, that does cost you. JavaScript SEO.

Questions

What is GPTBot?

OpenAI's web crawler for collecting content that may be used to train its models. It is separate from OAI-SearchBot, the crawler behind ChatGPT search.

Should I block GPTBot?

If you do not want your content used to train OpenAI's models, yes. It does not affect ChatGPT search, and it does not affect Google. Blocking every AI training crawler at once.

What is ChatGPT's user agent?

When ChatGPT visits a page while answering someone, it identifies itself as ChatGPT-User. Its search crawler is OAI-SearchBot and its training crawler GPTBot. The full strings are above.

Day 15 of 20. Check what OAI-SearchBot, ChatGPT-User and GPTBot each get from your site with the free AI visibility checker.

Want to know what AI crawlers see on your site? Our free AI visibility checker runs these checks against any page in about 10 seconds, no signup. If it finds something you'd rather hand over, our AI SEO service fixes it.

More from the blog

Get a free homepage mockup

Tell us about your business and we'll design a custom homepage concept within 48 hours — free, no strings attached.