Every AI crawler can get in
Search crawlers can index you and training crawlers can collect your pages.
What to do: If you’re fine with training, nothing to do. If not, block training bots and keep the search ones.
See every AI crawlerEnter your domain and this robots.txt checker shows which AI crawlers, like GPTBot, ClaudeBot and PerplexityBot, are allowed or blocked. Free, no sign-up.
Free · 60 checks a day · reads robots.txt only, no AI
An old Disallow line can keep you out of AI search without anyone noticing. Here is an example for brightlane.com, a sample invoicing brand that blocks AI training but not AI search.
22 of 28 AI crawlers can reach /
Disallow: /
No matching rule
Disallow: /
No matching rule
Allowed, partly blocked or blocked, for the exact path you entered, with what each bot is used for.
The robots.txt line that wins for that bot, with its line number, so you know what to edit.
Training crawlers, AI search crawlers and user-triggered fetchers, so you can block one kind and keep the others.
We read one public file and apply the Robots Exclusion Protocol the way the crawlers do.
A bare domain checks the home page; a full URL checks that path, like /blog/post.
brightlane.com/blogAs BrisanoBot, from https://domain/robots.txt, following up to five redirects like RFC 9309 says.
GET /robots.txt · 200Its own User-agent group if there is one, else “*”. The longest matching rule wins.
User-agent: GPTBotEvery crawler from our directory, with its status, the deciding line and what it is used for.
Allowed BlockedBlocking is a business choice. These are the three usual situations.
Search crawlers can index you and training crawlers can collect your pages.
What to do: If you’re fine with training, nothing to do. If not, block training bots and keep the search ones.
See every AI crawlerThe common middle ground: GPTBot or ClaudeBot out, OAI-SearchBot and PerplexityBot in.
What to do: Check that the search crawlers you care about are allowed, then track whether AI answers cite you.
Track ChatGPT answersOAI-SearchBot, PerplexityBot or Claude-SearchBot can’t fetch your pages, so their answers can’t cite them.
What to do: If that wasn’t on purpose, remove the rule or a “User-agent: *” Disallow, then check again.
GPTBot user agent and robots.txt rulesThree common choices. Paste the block into your robots.txt, then check again.
Training crawlers and control tokens are disallowed. Search and user-triggered fetchers keep access, so AI answers can still cite you.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot User-agent: AI2Bot Disallow: /
Every AI crawler in our directory may fetch everything. Same as having no rule for them.
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Google-GeminiNotebook User-agent: Google-Agent User-agent: Applebot User-agent: Applebot-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-externalagent User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: MistralAI-Training User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: CCBot User-agent: DuckAssistBot User-agent: AI2Bot Allow: /
Disallow: / for every AI bot in our directory, except Googlebot (it also runs Google Search).
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Google-GeminiNotebook User-agent: Google-Agent User-agent: Applebot User-agent: Applebot-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-externalagent User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: MistralAI-Training User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: CCBot User-agent: DuckAssistBot User-agent: AI2Bot Disallow: /
GPTBot is OpenAI’s crawler for pages that may be used in training. OpenAI runs two more: OAI-SearchBot, which shows sites in ChatGPT search, and ChatGPT-User, which fetches a page when a person asks. Each one follows its own robots.txt rule (OpenAI, “Overview of OpenAI crawlers”, read on 18 November 2026).
Most AI companies now split crawlers the same way: one for training, one for search, one for user requests. Blocking the training crawler does not block the search one, and the search crawler is the one that lets answers cite you.
robots.txt is a request, not a wall: the operators below say their crawlers follow it, and some user-triggered fetchers say rules may not apply. Firewalls and CDNs can block bots too, which robots.txt can’t show. More in What is GPTBot? and the AI crawlers list.
robots.txt, GPTBot and the other AI bots.
Enter your domain above. The checker reads your robots.txt and shows whether GPTBot, and every other AI crawler in our directory, is allowed or blocked for that path, with the line that decides it.
Add a group per crawler: “User-agent: GPTBot” and “Disallow: /” on the next line. Or copy one of the rule sets on this page: block training only, or Disallow: / for every AI crawler.
Not from ChatGPT search, according to OpenAI: GPTBot is for training, OAI-SearchBot shows sites in ChatGPT search, and each setting is independent (OpenAI, “Overview of OpenAI crawlers”, read on 18 November 2026). Block GPTBot and keep OAI-SearchBot to stay out of training but in search.
No. Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google Search Central, “Google’s common crawlers”, read on 18 November 2026). It only controls whether Google may use your pages for some of its AI products, such as Gemini.
robots.txt is only one layer. A firewall, CDN bot protection or a login wall can block crawlers whatever robots.txt says. The checker reads robots.txt only.
No. It reads robots.txt, the rules, not the visits. Visits from AI crawlers show up in your server logs or your CDN analytics.
No. It reads one public file and applies the Robots Exclusion Protocol (RFC 9309). It is free, 60 checks a day, and results are saved for 24 hours at a shareable link.
Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.