AI crawler and robots.txt checker

Enter your domain and this robots.txt checker shows which AI crawlers, like GPTBot, ClaudeBot and PerplexityBot, are allowed or blocked. Free, no sign-up.

Free · 60 checks a day · reads robots.txt only, no AI

What the robots.txt checker shows for AI crawlers

An old Disallow line can keep you out of AI search without anyone noticing. Here is an example for brightlane.com, a sample invoicing brand that blocks AI training but not AI search.

brightlane.comrobots.txt · 412 bytes · Nov 16 robots.txt read

22 of 28 AI crawlers can reach /

CrawlerStatusRule that decides
GPTBotAI training Blocked Disallow: /
OAI-SearchBotAI search Allowed No matching rule
ClaudeBotAI training Blocked Disallow: /
PerplexityBotAI search Allowed No matching rule
  1. 1

    One line per crawler

    Allowed, partly blocked or blocked, for the exact path you entered, with what each bot is used for.

  2. 2

    The rule that decides

    The robots.txt line that wins for that bot, with its line number, so you know what to edit.

  3. 3

    Grouped by purpose

    Training crawlers, AI search crawlers and user-triggered fetchers, so you can block one kind and keep the others.

How to check your robots.txt file for AI bots

We read one public file and apply the Robots Exclusion Protocol the way the crawlers do.

  1. 1

    Enter a domain or URL

    A bare domain checks the home page; a full URL checks that path, like /blog/post.

    brightlane.com/blog
  2. 2

    We fetch robots.txt

    As BrisanoBot, from https://domain/robots.txt, following up to five redirects like RFC 9309 says.

    GET /robots.txt · 200
  3. 3

    We match each crawler

    Its own User-agent group if there is one, else “*”. The longest matching rule wins.

    User-agent: GPTBot
  4. 4

    You see who gets in

    Every crawler from our directory, with its status, the deciding line and what it is used for.

    Allowed Blocked

What to do with the result

Blocking is a business choice. These are the three usual situations.

All allowed AI training AI search User-triggered

Every AI crawler can get in

Search crawlers can index you and training crawlers can collect your pages.

What to do: If you’re fine with training, nothing to do. If not, block training bots and keep the search ones.

See every AI crawler
Mixed AI training AI search User-triggered

Training blocked, search allowed

The common middle ground: GPTBot or ClaudeBot out, OAI-SearchBot and PerplexityBot in.

What to do: Check that the search crawlers you care about are allowed, then track whether AI answers cite you.

Track ChatGPT answers
All blocked AI training AI search User-triggered

AI search crawlers are blocked

OAI-SearchBot, PerplexityBot or Claude-SearchBot can’t fetch your pages, so their answers can’t cite them.

What to do: If that wasn’t on purpose, remove the rule or a “User-agent: *” Disallow, then check again.

GPTBot user agent and robots.txt rules

robots.txt rules for AI crawlers

Three common choices. Paste the block into your robots.txt, then check again.

Block AI training, stay in AI search

Training crawlers and control tokens are disallowed. Search and user-triggered fetchers keep access, so AI answers can still cite you.

robots.txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: AI2Bot
Disallow: /

Allow all AI crawlers

Every AI crawler in our directory may fetch everything. Same as having no rule for them.

robots.txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Google-GeminiNotebook
User-agent: Google-Agent
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: AI2Bot
Allow: /

Block all AI crawlers

Disallow: / for every AI bot in our directory, except Googlebot (it also runs Google Search).

robots.txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Google-GeminiNotebook
User-agent: Google-Agent
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: AI2Bot
Disallow: /

Training, search and user crawlers: which to allow

GPTBot is OpenAI’s crawler for pages that may be used in training. OpenAI runs two more: OAI-SearchBot, which shows sites in ChatGPT search, and ChatGPT-User, which fetches a page when a person asks. Each one follows its own robots.txt rule (OpenAI, “Overview of OpenAI crawlers”, read on 18 November 2026).

Most AI companies now split crawlers the same way: one for training, one for search, one for user requests. Blocking the training crawler does not block the search one, and the search crawler is the one that lets answers cite you.

robots.txt is a request, not a wall: the operators below say their crawlers follow it, and some user-triggered fetchers say rules may not apply. Firewalls and CDNs can block bots too, which robots.txt can’t show. More in What is GPTBot? and the AI crawlers list.

Three kinds of AI crawler
AI training
Collect pages that may train models. Blocking them keeps future content out of training sets. GPTBot, ClaudeBot, Google-Extended
AI search
Build the index AI answers search and cite. Blocking them can keep you out of those answers. OAI-SearchBot, PerplexityBot, Claude-SearchBot
User-triggered
Fetch one page when a person asks. Operators say robots.txt may not always apply. ChatGPT-User, Claude-User, Perplexity-User

AI crawler checker FAQ

robots.txt, GPTBot and the other AI bots.

How do I know if GPTBot is blocked on my site?

Enter your domain above. The checker reads your robots.txt and shows whether GPTBot, and every other AI crawler in our directory, is allowed or blocked for that path, with the line that decides it.

How do I block AI crawlers in robots.txt?

Add a group per crawler: “User-agent: GPTBot” and “Disallow: /” on the next line. Or copy one of the rule sets on this page: block training only, or Disallow: / for every AI crawler.

If I block GPTBot, will I disappear from ChatGPT?

Not from ChatGPT search, according to OpenAI: GPTBot is for training, OAI-SearchBot shows sites in ChatGPT search, and each setting is independent (OpenAI, “Overview of OpenAI crawlers”, read on 18 November 2026). Block GPTBot and keep OAI-SearchBot to stay out of training but in search.

Does blocking Google-Extended remove me from Google Search?

No. Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google Search Central, “Google’s common crawlers”, read on 18 November 2026). It only controls whether Google may use your pages for some of its AI products, such as Gemini.

Why does the checker show “allowed” but bots still can’t reach my site?

robots.txt is only one layer. A firewall, CDN bot protection or a login wall can block crawlers whatever robots.txt says. The checker reads robots.txt only.

Can it show when AI bots visit my site?

No. It reads robots.txt, the rules, not the visits. Visits from AI crawlers show up in your server logs or your CDN analytics.

Does this checker call AI?

No. It reads one public file and applies the Robots Exclusion Protocol (RFC 9309). It is free, 60 checks a day, and results are saved for 24 hours at a shareable link.

Files are the start. Answers are the result.

Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.