What is ClaudeBot? User agent and robots.txt rules
ClaudeBot is Anthropic’s crawler for web content that may be used to train its models. Blocking it keeps your future content out of Anthropic’s training data.
ClaudeBot user agent
Anthropic documents the name only: requests contain “ClaudeBot”.
robots.txt rules for ClaudeBot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read. Not sure what your file does today? Run the AI crawler checker.
Block ClaudeBot
Disallows every path for ClaudeBot. That keeps your future pages out of what Anthropic may use for training. It does not change search results.
User-agent: ClaudeBot Disallow: /
Allow ClaudeBot
Lets ClaudeBot fetch every path, whatever “User-agent: *” says.
User-agent: ClaudeBot Allow: /
Good to know
ClaudeBot also honours the non-standard Crawl-delay directive. Anthropic advises against IP blocking to opt out, since it can stop the bot from reading your robots.txt.
ClaudeBotWhat ClaudeBot means for Claude
Blocking ClaudeBot changes what Anthropic may use from your pages for training. It doesn’t tell you whether Claude names your brand today: only the answers show that. Brisano asks Claude your buyers’ questions as often as you choose and shows whether you are mentioned, in what position and which sources are cited.
ClaudeBot FAQ
What is ClaudeBot?
ClaudeBot is Anthropic’s crawler for web content that may be used to train its models. Blocking it keeps your future content out of Anthropic’s training data.
How do I block ClaudeBot?
Add “User-agent: ClaudeBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what Anthropic may use for training. It does not change search results.
Does ClaudeBot respect robots.txt?
Yes. Anthropic says ClaudeBot follows robots.txt rules written for it.
What is the ClaudeBot user agent?
Anthropic documents the name “ClaudeBot”, which appears in its requests.
How can I verify ClaudeBot requests?
Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.
Is ClaudeBot the same as Claude?
No. ClaudeBot is Anthropic’s crawler for content that may train its models. Claude-SearchBot builds the index for Claude’s search answers, and Claude-User fetches a page when a Claude user asks.
Source: support.claude.com, checked Nov 17, 2026. AI crawlers list and directory
Files are the start. Answers are the result.
Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.