AI training
Collect pages that may train models. Blocking them keeps future content out of training sets.
9 crawlers28 AI crawlers from OpenAI, Anthropic, Google, Perplexity and others: what each one does, its user agent and the robots.txt rule to allow or block it.
Checked against each operator’s documentation on Nov 17, 2026
| Crawler | Used for |
|---|---|
| OAI-AdsBotOpenAI | Other |
| Google-CloudVertexBotGoogle | Other |
Most AI companies run separate crawlers. Block the kind you don’t want and keep the rest.
Collect pages that may train models. Blocking them keeps future content out of training sets.
9 crawlersBuild the index AI answers search and cite. Blocking them can keep you out of those answers.
9 crawlersFetch one page when a person asks. Operators say robots.txt may not always apply.
8 crawlersEach engine reads the web with its own crawlers. Allowing the search and user-triggered ones lets it read and cite your pages. It does not decide whether the answer names your brand: that you can only measure.
Only crawlers with a primary source are listed. Is it working? See whether ChatGPT, Gemini and six more engines mention your brand.
The rule set most sites ask for. It disallows every training crawler and control token in this directory and leaves search and user-triggered fetchers alone.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot User-agent: AI2Bot Disallow: /
What the bots are and how to control them.
Bots run by AI companies that fetch web pages. Some collect pages that may train models (GPTBot, ClaudeBot), some build the index AI search answers cite (OAI-SearchBot, PerplexityBot), and some fetch a page when a person asks (ChatGPT-User).
With robots.txt: a “User-agent:” line with the bot’s token and “Disallow: /”. Each bot page has the block and allow rules to copy, and the AI crawler checker shows what your current file does.
The operators listed here say their crawlers follow robots.txt. Several say their user-triggered fetchers may not, because a person asked for the page. Each bot page quotes what its operator documents.
Control tokens, not crawlers. They never visit your site; they decide whether pages Googlebot or Applebot already fetched may be used for AI training (and, for Google-Extended, grounding in Gemini). They don’t affect search results.
User agents can be faked. Most operators publish IP ranges (linked on each bot page) or support reverse DNS checks. Match the request IP against them before trusting the user agent.
Every bot is checked against its operator’s own documentation, with the date shown, and the list is reviewed every quarter. Bots without a primary source are not listed.
If you want AI search answers to read and cite your pages, allow the search and user-triggered crawlers, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. According to their operators, blocking training crawlers such as GPTBot or ClaudeBot does not remove you from AI search. Allowing a crawler lets an engine read your pages; it does not make the engine name your brand.
In your server or CDN logs: filter requests by the user agent tokens on this list and confirm them against each operator’s published IP ranges. Brisano doesn’t read your logs; the AI crawler checker shows what your robots.txt allows.
See whether GPTBot, ClaudeBot and others can reach your site.
Free toolValidate any site’s llms.txt against the format.
GuideWhat OpenAI’s crawlers do and how to control them.
TrackerSee whether Perplexity names and cites you.
Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.