The exact rules, plus the OpenAI crawlers a GPTBot rule does not cover.
The short answer
Block GPTBot with a robots.txt group that sets User-agent to GPTBot and Disallow to a slash. Allow it by removing the group or adding Allow. OpenAI runs separate crawlers for search and user requests, so a GPTBot rule alone does not control them.
To block GPTBot, add a group to your robots.txt with User-agent: GPTBot and Disallow: /. To allow it, remove that group or use Allow: /. If you have no rule for it and no blanket block, GPTBot is allowed by default. The part people miss is that OpenAI runs more than one crawler, and each needs its own rule.
Add this to the robots.txt at the root of your site:
User-agent: GPTBot
Disallow: /
An automated bot run by an AI company that fetches web pages, either to train models, to build an AI search index or to load a page when a user asks about it.
GPTBot is the crawler OpenAI uses to collect content that may be used to train its models. This rule stops that crawler from fetching any page on your site. Wondering about SEO impact? See does blocking GPTBot hurt your Google ranking.
If you have no rule for GPTBot, it is allowed. To say so explicitly, or to allow it while keeping other crawlers out of one folder, use:
User-agent: GPTBot
Allow: /
User-agent: *
Disallow: /private/
GPTBot follows its own group only. The wildcard group below it does not apply to GPTBot, because a crawler follows the single most specific group that names it.
This is where accidental blocks happen. OpenAI uses separate names for separate jobs:
A rule for GPTBot does not cover the others. If you want to opt out of training but still appear in ChatGPT search, block GPTBot and leave OAI-SearchBot allowed:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
OpenAI isn't the only company with crawlers. Here is the AI crawler list for robots.txt.
A blanket User-agent: * with Disallow: / blocks every AI crawler that has no group of its own, and so can some security plugins and CDN settings that edit robots.txt for you. After any change, run your domain through the AI crawler checker. It reads your robots.txt and lists which known AI crawlers are blocked, either by name or by a wildcard rule.
If your robots.txt is already blocking bots, see robots.txt blocking AI bots by accident.
Remember that robots.txt is a request, not a lock. Well-behaved crawlers follow it. If you need to stop a bot for certain, use firewall rules.
To confirm what ChatGPT can reach, see how to check if ChatGPT can see your site.
It collects content that may be used for model training.
What we did: Disallow GPTBot in robots.txt to opt out of training.
It feeds ChatGPT search, so blocking it can reduce how often you appear there.
What we did: Leave OAI-SearchBot allowed if you want visibility.
It fetches pages when a person asks ChatGPT to open them, and OpenAI says robots.txt may not apply.
What we did: Set a rule on purpose, and use firewall rules if you must stop it.
What this cannot tell you
This covers OpenAI's documented crawler names only and does not cover how other AI companies handle robots.txt.
Rankings and recommendations reflect our own testing and criteria. Pricing and features change often — always confirm current details directly with the provider.
Add a robots.txt group with User-agent set to GPTBot and Disallow set to a single slash.
Not by itself. Search uses a separate crawler called OAI-SearchBot, which needs its own rule.
Yes, unless a rule or a wildcard block says otherwise.
Yes. A User-agent star group with Disallow slash blocks it along with every other crawler that has no group of its own.
See how your own site scores
Run the same AEO, SEO, and GEO checks this explainer is about — free, rule-based, no signup.
Comments
No comments yet — be the first.