GPTBot is the web crawler OpenAI uses for its foundation models. OpenAI's own wording: GPTBot "is used to make our generative AI foundation models more useful and safe", and "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models" (OpenAI). It is one of four OpenAI user agents, and each one needs its own decision. Check whether GPTBot can crawl your site before you change any rule.
What GPTBot does, and what it doesn't
OpenAI's GPTBot reads public web pages so OpenAI can use them in training. It does not decide what ChatGPT shows in its search answers, and it does not fetch a page because a user asked for it. Those jobs belong to other crawlers.
That split matters because the three jobs lead to different results:
- Training shapes what the model knows without searching, and changes only when a new model ships.
- Search decides which pages ChatGPT can read and cite at answer time.
- User actions fetch a page a person pointed ChatGPT to.
Why the difference matters for your site
Most site owners care about two outcomes at once: keeping their writing out of model training, and staying visible when buyers ask ChatGPT for a recommendation. With OpenAI the two don't collide: the training decision sits on GPTBot and the search decision sits on OAI-SearchBot, and you can answer each one differently. The mistake to avoid is treating "AI bots" as one thing, in robots.txt or in a CDN setting, and blocking search while you meant to block training.
OpenAI's crawlers: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot
| User agent | What OpenAI uses it for | robots.txt | IP ranges |
|---|---|---|---|
GPTBot |
Training: "to make our generative AI foundation models more useful and safe" | Honoured | gptbot.json |
OAI-SearchBot |
Surfaces websites in ChatGPT's search features | Honoured | searchbot.json |
ChatGPT-User |
Actions a user starts in ChatGPT; "not used for crawling the web in an automatic fashion" | "robots.txt rules may not apply" | chatgpt-user.json |
OAI-AdsBot |
Checks the safety of web pages submitted as ads on ChatGPT; not used for training | Not stated | adsbot.json |
Source: OpenAI, Overview of OpenAI crawlers, checked on November 18, 2026. OpenAI says "each setting is independent of the others". For every AI bot, not only OpenAI's, see the AI crawlers list.
Should you block GPTBot?
Decide what you want from OpenAI, then pick the row:
| You want | GPTBot | OAI-SearchBot | Result |
|---|---|---|---|
| No training, but appear in ChatGPT search | Disallow | Allow | Your pages can still be read and cited in ChatGPT search |
| Everything allowed | Allow | Allow | Your content may be used in training and in search |
| Nothing from OpenAI | Disallow | Disallow | Your pages are kept out of ChatGPT search answers too |
To block training and allow search, the first row:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Rules apply per user agent, so a Disallow for GPTBot doesn't affect OAI-SearchBot. For the other rules (one folder, one file type, the full string) see GPTBot user agent and robots.txt rules. Remember that a CDN or firewall "block AI bots" switch can override what your robots.txt says.
Does blocking GPTBot affect your visibility in ChatGPT?
Not in ChatGPT search. Visibility in ChatGPT's search answers depends on OAI-SearchBot, which OpenAI describes as the agent "used to surface websites in search results in ChatGPT's search features" (OpenAI). Answers that come from the model alone reflect what it learned in training, which is where GPTBot sits, and they change only when OpenAI ships a new model.
OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. If AI visibility matters to you, track your brand in ChatGPT on a fixed set of prompts before and after the change, so you see what moved instead of guessing. The guide on how to rank in ChatGPT covers the rest of the checklist.
How to spot GPTBot in your logs
To monitor AI crawler activity, filter your access logs by user agent first, then confirm each hit by IP. A GPTBot request carries GPTBot in its user agent string; the complete string is on the GPTBot page of our AI crawlers list. Count requests per day and per status code: a sudden drop to zero after a deploy usually means a new rule is blocking it.
- Match the IP, not only the name. A user agent string is easy to fake. OpenAI publishes the IP ranges of each crawler (linked in the table above); check a request's IP against the right list before you trust it.
- Look for 403 and 429 responses. If your firewall or CDN answers OpenAI's crawlers with errors, your
robots.txtisn't the rule that counts. - Check again after every change. After you edit
robots.txtor a bot rule, review your logs over the next days and confirm the crawlers you allowed still get a 200.
Sources
- Overview of OpenAI crawlers · OpenAI, checked 2026-11-18
- GPTBot IP ranges · OpenAI
- OAI-SearchBot IP ranges · OpenAI
- ChatGPT-User IP ranges · OpenAI
- OAI-AdsBot IP ranges · OpenAI