What is Googlebot? User agent and robots.txt rules

Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features, so Googlebot’s robots.txt rules are the control for them too.

AI search Follows robots.txt Google

Googlebot user agent

As documented by Google. Version numbers change; match on the token.

User agent
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

robots.txt rules for Googlebot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read. Not sure what your file does today? Run the AI crawler checker.

Block Googlebot

Disallows every path for Googlebot. That keeps your pages out of Google’s AI search index, so its answers can’t cite them.

robots.txt
User-agent: Googlebot
Disallow: /

Allow Googlebot

Lets Googlebot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: Googlebot
Allow: /

Good to know

Blocking Googlebot also removes you from regular Google Search. To limit what AI features show, Google points to nosnippet, data-nosnippet, max-snippet or noindex instead.

Operator Google
Used forAI search
robots.txtFollows robots.txt
robots.txt tokenGooglebot
Can Googlebot reach your site? Enter your domain: we read your robots.txt and show Googlebot’s status.

What Googlebot means for Google AI Overviews and Google AI Mode

Allowing Googlebot lets Google AI Overviews and Google AI Mode read your pages to build answers. It doesn’t decide whether your brand is named in them. Brisano measures that: it asks Google AI Overviews and Google AI Mode your buyers’ questions as often as you choose and shows whether you are mentioned, in what position and which sources are cited.

Googlebot FAQ

What is Googlebot?

Googlebot is Google Search’s crawler. Google’s AI Overviews and AI Mode are Search features, so Googlebot’s robots.txt rules are the control for them too.

How do I block Googlebot?

Add “User-agent: Googlebot” and “Disallow: /” to your robots.txt. That keeps your pages out of Google’s AI search index, so its answers can’t cite them.

Does Googlebot respect robots.txt?

Yes. Google says Googlebot follows robots.txt rules written for it.

What is the Googlebot user agent?

Google documents it as “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”. Version numbers change, so match on “Googlebot”.

How can I verify Googlebot requests?

Match the request IP against Google’s published list (https://developers.google.com/static/search/apis/ipranges/googlebot.json). The user agent alone can be faked.

Does blocking Googlebot remove me from AI Overviews?

Yes, and from regular Google Search too: AI Overviews and AI Mode are Search features. To limit what AI features show, Google points to nosnippet, data-nosnippet, max-snippet or noindex.

Source: developers.google.com, checked Nov 17, 2026. AI crawlers list and directory

Files are the start. Answers are the result.

Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.