There is no single official list of AI crawlers, but about a dozen names cover most of what you will want to control. Each company usually runs separate bots for training, search and user requests, so you set one rule per name. Below are the ones to know, followed by a ready-to-edit robots.txt.
Complete AI Crawler List for robots.txt
The bots to know, what each one does, and a starter file you can edit.
The short answer
There is no official list, but about a dozen AI crawler names cover most needs. Companies run separate bots for training, search and user requests, so you set one rule per name. Use the starter robots.txt below and test it before relying on it.
Bottom lineSet a deliberate rule for each bot by job, then test your robots.txt instead of assuming it works.
On this page
An automated bot run by an AI company that fetches web pages, either to train models, to build an AI search index or to load a page when a user asks about it.
OpenAI
- GPTBot: training data.
- OAI-SearchBot: ChatGPT search index.
- ChatGPT-User: pages fetched on a person's request.
- OAI-AdsBot: checks landing pages submitted as ads.
Anthropic
- ClaudeBot: training data.
- Claude-SearchBot: search results.
- Claude-User: pages fetched on a person's request.
Older names you may still see in robots.txt files are anthropic-ai and Claude-Web. Third-party guides report that these were Anthropic's earlier names, but Anthropic's own page lists only the three bots above. Keeping a rule for the old names does no harm.
Google-Extended is a control token, not a separate crawler. It tells Google whether your content may be used to train future Gemini models and to ground answers in Gemini Apps. Google says it does not affect inclusion or ranking in Google Search.
Worried about search impact? See does blocking GPTBot hurt your Google ranking.
Perplexity
- PerplexityBot: search index.
- Perplexity-User: pages fetched on a person's request.
Others
- CCBot: Common Crawl's crawler. Common Crawl publishes an open web archive that many AI projects are reported to draw on.
- Applebot-Extended: a control token for whether Apple may use your content to train its models.
To allow or block any of these, follow our guide on how to allow or block GPTBot in robots.txt.
If you suspect your robots.txt is already catching bots you meant to allow, read about blocking AI bots by accident.
A starting robots.txt
This example opts out of training while leaving search and user-request bots allowed. Change it to match your own decision.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Bots not listed stay allowed unless a wildcard rule blocks them.
Things that trip people up
Blocking a training bot does not block the same company's search bot, and blocking a search bot can reduce how often you are cited. Bots that act on a user's request do not all behave the same way. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, while Anthropic says its bots, Claude-User included, honor robots.txt. And robots.txt is a request, not access control.
Names change, and new ones appear, so treat any list, including this one, as a snapshot. We checked how often large sites block these crawlers in our scan of 97 big sites.
Test your file
Paste your domain into the AI crawler checker. It checks your robots.txt against the known AI crawlers and tells you which are blocked, whether by name or by a wildcard rule. Once you know the names, the next step is to check if ChatGPT can see your site.
Related pages
- Glossary: AI crawler
- Free tool: AI crawler checker
- Allow or block a crawler: How to allow or block GPTBot in robots.txt
- Search impact: Does blocking GPTBot hurt your Google ranking
- If something is blocked by mistake: robots.txt blocking AI bots by accident
- Confirm the result: How to check if ChatGPT can see your site
- The data: AI-readiness scan of 97 large sites
- AEO
Training bots
These bots collect content that may be used to train models.
What we did: Disallowed GPTBot and ClaudeBot in the starter robots.txt to opt out of training.
- !AEO
Search bots
Blocking search bots can reduce how often AI search tools cite you.
What we did: Left OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed in the starter robots.txt to keep visibility.
- !AEO
User-request bots
These bots fetch a page when a person asks, and OpenAI and Perplexity say theirs may ignore robots.txt, while Anthropic says Claude-User honors it.
What we did: Left them allowed in the starter file, so check each vendor and use firewall rules if you must stop one.
What this cannot tell you
This is a snapshot of commonly documented crawler names and cannot cover new bots or undocumented ones.
Rankings and recommendations reflect our own testing and criteria. Pricing and features change often — always confirm current details directly with the provider.
FAQ
No. Each company documents its own bots, and the names change over time.
It is a control token. It governs use of your content for Gemini and does not affect normal Search indexing.
Not necessarily. Search bots are separate, so they need their own rules.
Not always. robots.txt is a request. OpenAI says it may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores it, while Anthropic says its bots honor it.
See how your own site scores
Run the same AEO, SEO, and GEO checks this explainer is about — free, rule-based, no signup.
Comments
No comments yet — be the first.