What is Claude-SearchBot? User agent and robots.txt rules

Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.

AI search Follows robots.txt Anthropic

Claude-SearchBot user agent

Anthropic documents the name only: requests contain “Claude-SearchBot”.

robots.txt rules for Claude-SearchBot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read. Not sure what your file does today? Run the AI crawler checker.

Block Claude-SearchBot

Disallows every path for Claude-SearchBot. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.

robots.txt
User-agent: Claude-SearchBot
Disallow: /

Allow Claude-SearchBot

Lets Claude-SearchBot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: Claude-SearchBot
Allow: /

Good to know

Anthropic notes that blocking it may reduce your visibility in user search results.

Operator Anthropic
Used forAI search
robots.txtFollows robots.txt
robots.txt tokenClaude-SearchBot
Documentationsupport.claude.com
Can Claude-SearchBot reach your site? Enter your domain: we read your robots.txt and show Claude-SearchBot’s status.

What Claude-SearchBot means for Claude

Allowing Claude-SearchBot lets Claude read your pages to build answers. It doesn’t decide whether your brand is named in them. Brisano measures that: it asks Claude your buyers’ questions as often as you choose and shows whether you are mentioned, in what position and which sources are cited.

Claude-SearchBot FAQ

What is Claude-SearchBot?

Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.

How do I block Claude-SearchBot?

Add “User-agent: Claude-SearchBot” and “Disallow: /” to your robots.txt. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.

Does Claude-SearchBot respect robots.txt?

Yes. Anthropic says Claude-SearchBot follows robots.txt rules written for it.

What is the Claude-SearchBot user agent?

Anthropic documents the name “Claude-SearchBot”, which appears in its requests.

How can I verify Claude-SearchBot requests?

Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.

Source: support.claude.com, checked Nov 17, 2026. AI crawlers list and directory

Files are the start. Answers are the result.

Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.