What is AI2Bot? User agent and robots.txt rules

AI2Bot is the crawler of the Allen Institute for AI (Ai2). It finds web content that Ai2 uses to train its open language models.

AI training Not documented Ai2

AI2Bot user agent

As documented by Ai2. Version numbers change; match on the token.

User agent
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)

robots.txt rules for AI2Bot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read. Not sure what your file does today? Run the AI crawler checker.

Block AI2Bot

Disallows every path for AI2Bot. That keeps your future pages out of what Ai2 may use for training. It does not change search results.

robots.txt
User-agent: AI2Bot
Disallow: /

Allow AI2Bot

Lets AI2Bot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: AI2Bot
Allow: /

Good to know

Ai2’s page doesn’t say how it reads robots.txt; it suggests filtering on the user agent.

Operator Ai2
Used forAI training
robots.txtNot documented
robots.txt tokenAI2Bot
IP rangesNot published
Documentationallenai.org
Can AI2Bot reach your site? Enter your domain: we read your robots.txt and show AI2Bot’s status.

AI2Bot FAQ

What is AI2Bot?

AI2Bot is the crawler of the Allen Institute for AI (Ai2). It finds web content that Ai2 uses to train its open language models.

How do I block AI2Bot?

Add “User-agent: AI2Bot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what Ai2 may use for training. It does not change search results.

Does AI2Bot respect robots.txt?

Ai2’s documentation doesn’t say. Block it in robots.txt and, if needed, at your firewall by its published IPs.

What is the AI2Bot user agent?

Ai2 documents it as “Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)”. Version numbers change, so match on “AI2Bot”.

How can I verify AI2Bot requests?

Ai2 publishes no IP list for AI2Bot. Treat the user agent as a hint, not proof.

Source: allenai.org, checked Nov 17, 2026. AI crawlers list and directory

Files are the start. Answers are the result.

Your buyers’ questions on up to eight AI engines, run as often as you choose, with full history. See whether ChatGPT mentions your brand.