What is Claude-SearchBot? User agent and robots.txt rules
Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.
Claude-SearchBot user agent
Anthropic documents the name only: requests contain “Claude-SearchBot”.
robots.txt rules for Claude-SearchBot
Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read. Not sure what your file does today? Run the AI crawler checker.
Block Claude-SearchBot
Disallows every path for Claude-SearchBot. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.
User-agent: Claude-SearchBot Disallow: /
Allow Claude-SearchBot
Lets Claude-SearchBot fetch every path, whatever “User-agent: *” says.
User-agent: Claude-SearchBot Allow: /
Good to know
Anthropic notes that blocking it may reduce your visibility in user search results.
Claude-SearchBotWhat Claude-SearchBot means for Claude
Allowing Claude-SearchBot lets Claude read your pages to build answers. It doesn’t decide whether your brand is named in them. Brisano measures that: it asks Claude your buyers’ questions as often as you choose and shows whether you are mentioned, in what position and which sources are cited.
Claude-SearchBot FAQ
What is Claude-SearchBot?
Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.
How do I block Claude-SearchBot?
Add “User-agent: Claude-SearchBot” and “Disallow: /” to your robots.txt. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.
Does Claude-SearchBot respect robots.txt?
Yes. Anthropic says Claude-SearchBot follows robots.txt rules written for it.
What is the Claude-SearchBot user agent?
Anthropic documents the name “Claude-SearchBot”, which appears in its requests.
How can I verify Claude-SearchBot requests?
Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.
Source: support.claude.com, checked Nov 17, 2026. AI crawlers list and directory
Files are the start. Answers are the result.
Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.