GPTBot, OAI-SearchBot and ChatGPT-User: OpenAI's crawlers explained
OpenAI sends four bots to websites. GPTBot collects training data, OAI-SearchBot finds pages for ChatGPT search, ChatGPT-User visits on a person's behalf and OAI-AdsBot checks ads. What each does, the exact user agents, the robots.txt for each choice, and how to spot a fake.
OpenAI sends four different bots to websites, and they do very different jobs. Block the wrong one and you leave ChatGPT's search answers; block the right one and nothing changes except that your pages stay out of OpenAI's training data. Here is each bot with its exact user-agent, the robots.txt for each choice, and how to tell a real OpenAI request from a fake.
OpenAI's bots at a glance
| Bot | Job | Follows robots.txt? |
|---|---|---|
OAI-SearchBot | Finds pages to show in ChatGPT search. Block it and you are left out of ChatGPT's search answers | Yes |
GPTBot | Collects content that may be used to train OpenAI's foundation models | Yes |
ChatGPT-User | Visits pages on a person's behalf, when a question in ChatGPT or a custom GPT's action needs it. Does not crawl on its own | Not necessarily: OpenAI says robots.txt rules may not apply |
OAI-AdsBot | Checks the safety of pages submitted as ads on ChatGPT | Not stated; it only visits pages submitted as ads |
Source: OpenAI's crawler documentation, which stresses that the settings are independent: a site can allow OAI-SearchBot to appear in search and disallow GPTBot to stay out of training. If you allow both, OpenAI may use one crawl for both purposes rather than visiting twice.
The user-agent strings
OpenAI gives these as examples and says the version numbers may change:
OAI-SearchBot: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-User: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
Match on the bot's name, not the whole string, so your rule keeps working when the version changes.
robots.txt for each choice
In ChatGPT search, out of training (the usual choice)
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Everything allowed
User-agent: *
Allow: /
You do not need to mention OpenAI at all. If your robots.txt never names its bots, they follow your * rules.
Out of ChatGPT search as well
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT's search answers, though they may still appear as navigational links, and ChatGPT-User may still visit when a person asks about a page. Changes reach ChatGPT search in about 24 hours.
Is it really OpenAI?
Anyone can put "GPTBot" in a user-agent. OpenAI publishes the IP address ranges each of its bots uses, linked from its crawler documentation, and asks sites to accept OAI-SearchBot requests from them. To check a request, compare its IP address with the published list for that bot; Cloudflare and other services that verify crawlers do this for you. Treat a "GPTBot" from any other address as a stranger.
What our check found
| Bot | Disallowed in robots.txt (of 348 sites) | Turned away by the firewall (of 315) |
|---|---|---|
| GPTBot | 19 | 35 |
| ChatGPT-User | 7 | 14 |
| OAI-SearchBot | 4 | 15 |
Most of the blocking was aimed at GPTBot, as you would expect. But 15 sites also turned away OAI-SearchBot, which keeps them out of ChatGPT search whether the owner meant it or not. Our checker sends each crawler's published user-agent name from our own server, so a firewall that verifies crawlers by IP address will turn our test away even when the real crawler gets in. Read a block as "your firewall reacts to this name", then confirm in your firewall's event log.
Why GPTBot visits so often
In Vercel and MERJ's December 2024 measurement across Vercel's network, GPTBot made 569 million requests in a month, about an eighth of Googlebot's 4.5 billion, and none of the major AI crawlers they tested, OpenAI's included, ran JavaScript (their study). If GPTBot's load is the problem, blocking it costs you nothing in search. If OAI-SearchBot finds an empty page, that does cost you. JavaScript SEO.
Questions
What is GPTBot?
OpenAI's web crawler for collecting content that may be used to train its models. It is separate from OAI-SearchBot, the crawler behind ChatGPT search.
Should I block GPTBot?
If you do not want your content used to train OpenAI's models, yes. It does not affect ChatGPT search, and it does not affect Google. Blocking every AI training crawler at once.
What is ChatGPT's user agent?
When ChatGPT visits a page while answering someone, it identifies itself as ChatGPT-User. Its search crawler is OAI-SearchBot and its training crawler GPTBot. The full strings are above.
Day 15 of 20. Check what OAI-SearchBot, ChatGPT-User and GPTBot each get from your site with the free AI visibility checker.