Free tool
Free AI Crawler Checker (GPTBot, ClaudeBot & more)
Reads a site's robots.txt and reports which of 13 known AI crawlers it blocks, including GPTBot, ClaudeBot and PerplexityBot, and whether each is blocked by name or by a wildcard rule. One of twelve checks in the AEO score, worth 14 of 100 points. No API calls, no cost.
AI crawler
- — robots.txt is read (no file means crawlers are allowed)
- — 13 known AI crawlers checked, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended
- — Blanket User-agent: * blocks are applied unless a crawler has its own rule
Every scan scores all three areas — SEO, AEO and GEO — so you get the full picture in one go.
Looking for something more specific? Free AEO Checker
Enter a URL above — free, rule-based, no API cost.
Last updated
About the AI crawler checker
Many AI tools fetch pages with their own crawlers, and a site's robots.txt can allow or block each one. Some of these crawlers are used to train models, some feed AI search, and some fetch a page when a person asks about it. For example, GPTBot is OpenAI's training crawler, while OAI-SearchBot is used for ChatGPT search. This checker reads the robots.txt of the site you submit and tests 13 known AI crawler names against it. It reports a block when a rule shuts the whole site (Disallow: /), either for a crawler by name or through a blanket User-agent: * rule that the crawler has no rule of its own to override. Blocking is a legitimate choice, so the checker reports what it finds without judging it. It does not detect blocks set elsewhere, such as in a firewall or CDN bot protection.
How the AI crawler checker works
- You submit a page URL, and Half Dev fetches robots.txt from that site's root.
- It checks 13 AI crawler names. A crawler follows the rule group that names it; if none does, the User-agent: * group applies. A group that also says Allow: / counts as allowing.
- No block earns the full 14 points of this check, blocking some crawlers earns 7, and blocking all 13 earns 0. A site with no robots.txt scores 14, because crawlers are allowed by default. This is one of twelve AEO checks, so you also see the rest of the AEO score and the other two scores.
How to read your AI crawler score
AI crawler access is one of twelve checks in the AEO score, worth 14 of its 100 points. No block scores 14. Blocking some of the 13 known AI crawlers scores 7. Blocking all 13 scores 0. A site with no robots.txt scores 14, because crawlers are allowed by default.
The note under the check names the crawlers found blocked, and says whether each was blocked by name or by a wildcard rule.
No crawler blocked
A site with no robots.txt also scores 14, because crawlers are allowed by default.
Some of the 13 blocked
The note under the check names each one and says whether it was blocked by name or by a wildcard rule.
All 13 blocked
No known AI crawler can read the site.
How to fix each AI crawler block
A wildcard block
Lowers the scoreA group with User-agent: * and Disallow: / blocks every crawler that has no group of its own, including all the AI crawlers. Googlebot and other search crawlers are blocked too.
Fix: Remove the wildcard block, or narrow it to the paths you want hidden. Edits made by plugins, hosts and CDNs often create this rule without you noticing.
A block by name
Lowers the scoreA group such as User-agent: GPTBot with Disallow: / blocks that crawler only.
Fix: If it is deliberate, nothing to fix. The score is lower because those AI systems cannot read the site. If it is not deliberate, remove the Disallow line from that group.
Several crawlers sharing one rule
Lowers the scoreTwo or more User-agent lines above a single Disallow: / form one group, and every crawler named in it is blocked.
Fix: Split the group if you want different rules for different crawlers.
Allow and Disallow in the same group
No effectA group that has both Disallow: / and Allow: / is counted as allowing.
Fix: Keep only the line you mean. Two conflicting lines are confusing for people even if crawlers resolve them.
Rules for single folders
No effectA rule such as Disallow: /admin is not counted as a block, because the crawler can still read the rest of the site.
Fix: Nothing to fix. Only whole-site blocks lower the score.
Example: a robots.txt that blocks GPTBot and CCBot
A robots.txt that blocks two AI crawlers by name and keeps one folder private for everyone else. The checker reports a partial block.
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: / Disallow: /admin
AI crawler access (robots.txt)
robots.txt blocks 2 of 13 known AI crawlers: GPTBot, CCBot. Fine if deliberate; those AI systems can't read the site.
Who should check AI crawler access
- A publisher that decided to allow AI search checks that its robots.txt really does.
- A site owner who sees no AI referrals checks whether a plugin or hosting default blocked the crawlers.
- An agency checks every client's robots.txt before a redesign or migration.
What the AI crawler checker does not check
- It reads robots.txt only. A firewall, CDN or bot filter that refuses AI crawlers is invisible to it.
- It counts whole-site blocks only. Rules for single folders are ignored.
- robots.txt is a request that well-behaved crawlers follow. It is not a lock, and some crawlers that act on a person's request may not follow it.
- The list of 13 crawlers is a snapshot. New bots appear and names change.
FAQ
Thirteen: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, Claude-Web, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and CCBot.
It depends on what you want. Allowing them makes it possible for AI tools to read and quote your pages. Blocking them keeps your content away from those systems. This checker reports what your robots.txt does and doesn't tell you which to choose.
Crawlers are allowed by default, so the check passes with full points.
It is a request that well-behaved crawlers honour, not a lock. Anything stricter, such as a firewall or CDN bot rule, is not visible to this checker.
Only whole-site blocks (Disallow: /), by crawler name or through User-agent: *. A rule that blocks a single folder is not counted, because the crawler can still read the rest of the site.
In your robots.txt, remove the whole-site Disallow from that crawler's group, or add a group for that crawler that says Allow: /. Your CMS or hosting may expose robots.txt as a setting.
No. GPTBot is OpenAI's crawler, and Googlebot follows its own rules. A wildcard block that also covers Googlebot is what can hurt.
No. Each host has its own robots.txt, so scan the exact host you care about, such as blog.yourdomain.com.
Not necessarily. Blocking some AI crawlers is a valid choice. The score shows how much access they have, not whether your choice is right.
Terms explained: AI Crawler · robots.txt · Answer Engine Optimization
Want the whole picture? See all three checkers.