robots.txt for AI Crawlers: Allow and Deny Recipes
Three recipes and a bot-by-bot table for controlling GPTBot, ClaudeBot, PerplexityBot and the rest from one small text file.
To control AI crawlers, add User-agent and Disallow rules to the robots.txt file at your domain root. You can block every AI bot, block training but keep AI search, or allow all. The file is advisory: well-behaved bots obey it, but it is not access control.
How robots.txt controls AI crawlers
Each AI company ships one or more named agents. You target them by their user-agent token in robots.txt, then decide what they may fetch. There are three moves worth knowing: block a bot entirely, block only the training crawlers while letting the AI search bots through, or leave everything open. The recipes below cover all three. Every one belongs in a single file at https://example.com/robots.txt.
Recipe 1: block all AI crawlers
This denies every documented AI agent and adds the two opt-out tokens that govern training use. Paste it as-is, or trim the bots you do not care about:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /One caveat worth stating plainly: blocking GPTBot stops OpenAI's training crawler, not ChatGPT search (that is OAI-SearchBot) and not user-triggered browsing (that is ChatGPT-User). The recipe above covers all three, which is why each gets its own block.
Recipe 2: allow AI search but block AI training
A common goal is to stay visible in AI answer engines while keeping your content out of model training. Allow the search bots, disallow the training crawlers and the training tokens:
# Allow AI search engines
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Block AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Google-Extended and Applebot-Extended are the key to this split. They are robots.txt tokens, not separate crawlers. Disallowing them opts your content out of Gemini and Apple AI training while Googlebot and Applebot keep indexing you for normal search.
Recipe 3: allow everything (the default)
If you took nothing, this is the effective state. An empty Disallow lets every crawler fetch every path:
User-agent: *
Disallow:A missing robots.txt file means the same thing. You only need explicit rules when you want to restrict something.
The 11 AI crawlers, compared
| Bot | Company | Purpose | Block to stop… |
|---|---|---|---|
| GPTBot | OpenAI | Crawls to train models | Training on your content |
| OAI-SearchBot | OpenAI | Powers ChatGPT search results | Appearing in ChatGPT search |
| ChatGPT-User | OpenAI | User-triggered browsing and actions | User-initiated fetches from ChatGPT |
| ClaudeBot | Anthropic | Crawls to train models | Training on your content |
| Claude-User | Anthropic | User-triggered fetches | User-initiated fetches from Claude |
| PerplexityBot | Perplexity | Indexes for its answer engine | Indexing in Perplexity |
| Perplexity-User | Perplexity | User-triggered fetches | User-initiated fetches from Perplexity |
| Google-Extended | Token controlling Gemini/Vertex AI training use | Content use for Gemini/Vertex AI training | |
| Applebot-Extended | Apple | Token controlling Apple AI training use | Content use for Apple AI training |
| CCBot | Common Crawl | Builds a public dataset used for AI training | Inclusion in the Common Crawl dataset |
| Bytespider | ByteDance (TikTok) | AI crawler | ByteDance AI crawling |
What robots.txt can and cannot do
A few rules decide whether your file works at all:
- It must sit at the domain root. Crawlers read
https://example.com/robots.txtand nowhere else. A file in a subfolder is ignored, and every subdomain needs its own. - The token is case-insensitive.
GPTBotandgptbotmatch the same crawler. Keep the documented casing for readability, but do not worry about an exact match. - Some entries are tokens, not crawlers.
Google-ExtendedandApplebot-Extendednever fetch anything. They only signal whether Google and Apple may use already-crawled content for AI training. - It is advisory, not a wall. robots.txt is voluntary. Reputable bots honor it; it is not access control and will not stop a crawler that chooses to ignore it. For hard blocking, use firewall rules, authentication, or edge blocking by user agent and IP.
Pair it with an llms.txt file
robots.txt decides who may crawl. An llms.txt file does the opposite job: it points AI systems at the content you want them to read and understand. The two work together, and you can build one with the llms.txt generator. To see how your whole site looks to AI crawlers, run the AI readiness check.