AI crawlers list and directory

28 AI crawlers from OpenAI, Anthropic, Google, Perplexity and others: what each one does, its user agent and the robots.txt rule to allow or block it.

Checked against each operator’s documentation on Nov 17, 2026

Crawler Used for robots.txt robots.txt token
GPTBotOpenAI AI training Follows robots.txt GPTBot
ClaudeBotAnthropic AI training Follows robots.txt ClaudeBot
Google-ExtendedGoogle AI training Control token only Google-Extended
Applebot-ExtendedApple AI training Control token only Applebot-Extended
Meta-ExternalAgentMeta AI training Follows robots.txt meta-externalagent
AmazonbotAmazon AI training Follows robots.txt Amazonbot
MistralAI-TrainingMistral AI AI training Follows robots.txt MistralAI-Training
CCBotCommon Crawl AI training Follows robots.txt CCBot
AI2BotAi2 AI training Not documented AI2Bot

Training, search and user-triggered crawlers

Most AI companies run separate crawlers. Block the kind you don’t want and keep the rest.

GPTBot ClaudeBot Google-Extended

AI training

Collect pages that may train models. Blocking them keeps future content out of training sets.

9 crawlers
OAI-SearchBot Claude-SearchBot Googlebot

AI search

Build the index AI answers search and cite. Blocking them can keep you out of those answers.

9 crawlers
ChatGPT-User Claude-User Google-GeminiNotebook

User-triggered

Fetch one page when a person asks. Operators say robots.txt may not always apply.

8 crawlers

Which crawler each AI engine uses to read your site

Each engine reads the web with its own crawlers. Allowing the search and user-triggered ones lets it read and cite your pages. It does not decide whether the answer names your brand: that you can only measure.

Only crawlers with a primary source are listed. Is it working? See whether ChatGPT, Gemini and six more engines mention your brand.

Block AI training, stay in AI search

The rule set most sites ask for. It disallows every training crawler and control token in this directory and leaves search and user-triggered fetchers alone.

AI crawler and robots.txt checker

robots.txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: AI2Bot
Disallow: /

AI crawlers FAQ

What the bots are and how to control them.

What are AI crawlers?

Bots run by AI companies that fetch web pages. Some collect pages that may train models (GPTBot, ClaudeBot), some build the index AI search answers cite (OAI-SearchBot, PerplexityBot), and some fetch a page when a person asks (ChatGPT-User).

How do I block AI crawlers?

With robots.txt: a “User-agent:” line with the bot’s token and “Disallow: /”. Each bot page has the block and allow rules to copy, and the AI crawler checker shows what your current file does.

Do AI crawlers respect robots.txt?

The operators listed here say their crawlers follow robots.txt. Several say their user-triggered fetchers may not, because a person asked for the page. Each bot page quotes what its operator documents.

What are Google-Extended and Applebot-Extended?

Control tokens, not crawlers. They never visit your site; they decide whether pages Googlebot or Applebot already fetched may be used for AI training (and, for Google-Extended, grounding in Gemini). They don’t affect search results.

How do I verify a request really comes from an AI crawler?

User agents can be faked. Most operators publish IP ranges (linked on each bot page) or support reverse DNS checks. Match the request IP against them before trusting the user agent.

How often is this list updated?

Every bot is checked against its operator’s own documentation, with the date shown, and the list is reviewed every quarter. Bots without a primary source are not listed.

Which AI crawlers should I allow?

If you want AI search answers to read and cite your pages, allow the search and user-triggered crawlers, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. According to their operators, blocking training crawlers such as GPTBot or ClaudeBot does not remove you from AI search. Allowing a crawler lets an engine read your pages; it does not make the engine name your brand.

How do I see which AI crawlers visit my site?

In your server or CDN logs: filter requests by the user agent tokens on this list and confirm them against each operator’s published IP ranges. Brisano doesn’t read your logs; the AI crawler checker shows what your robots.txt allows.

Files are the start. Answers are the result.

Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.