robots.txt and AI Crawlers: GPTBot, OAI-SearchBot and ClaudeBot

To control AI crawlers in robots.txt, separate three purposes: training (GPTBot, ClaudeBot, Google-Extended), search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and actions requested by a user (ChatGPT-User, Claude-User, Perplexity-User). Blocking training does not remove you from search, and blocking search removes you from the answers. Also, robots.txt is a polite request, not a security barrier.

Quick summary

• In AI search (see the difference between SEO, AEO and GEO), each company runs more than one bot, and each has a job: train models, build a search index or fetch a page when someone asks.
• Blocking the training bot (such as GPTBot) is a content policy decision. Blocking the search bot (such as OAI-SearchBot) removes the site from that product's search answers.
• ChatGPT-User and Perplexity-User exist for user actions and, according to the companies themselves, may ignore robots.txt. Claude-User, according to Anthropic, obeys it.
• Google-Extended does not affect inclusion or ranking in Google Search. It is a usage control for Gemini models.
• Confirm what is happening in your server logs, validating the IP against the official lists, because anyone can fake a user agent.
• No setting guarantees an AI citation, and nobody can say whether allowing or blocking training changes how often you are mentioned.

How to read this: if you own a site and just want to know "what do I allow?", read the summary, the table and the three sample files. If you are on the technical team, the sections on logs, protocol limits and edge cases matter most. This post complements basic technical SEO.

What are AI crawlers, and why are there so many?

AI crawlers are bots that visit web pages for three distinct purposes: collecting content that may train models, building a search index for answers with sources, and opening a specific page when a user asks. The companies split the bots precisely so that site owners can allow one use and deny another.

In plain English: imagine three kinds of visitors to your store. A photographer who records everything for a catalog (training), a tour guide who writes down the address to recommend you to customers (search) and a customer who asks an assistant to "go look at that shop window and tell me about it" (user action). robots.txt is the sign on the door saying who may come in. But the sign depends on the goodwill of whoever reads it.

OpenAI says its settings are independent. According to OpenAI's official crawler documentation, a webmaster can allow OAI-SearchBot so the site appears in results and disallow GPTBot to signal that the content should not train models. Anthropic and Perplexity also separate the roles, as the table below shows.

Which user agents are official, and what does each one do?

The table lists the tokens documented by the companies as of October 4, 2026. The tokens are the names you use in User-agent: in robots.txt, and they can change. Always check the documentation before you publish.

Company	Token	Purpose	Obeys robots.txt?	What blocking does
OpenAI	OAI-SearchBot	ChatGPT search	Yes	Site does not appear in ChatGPT search answers, although it may still appear as a navigational link
OpenAI	GPTBot	Model training	Yes	Signals that the content should not train OpenAI models
OpenAI	ChatGPT-User	User actions in ChatGPT and GPTs	"May not apply," according to OpenAI	Does not control search. For that, use OAI-SearchBot
Anthropic	ClaudeBot	Model training	Yes	Future materials are excluded from training sets
Anthropic	Claude-SearchBot	Search result quality	Yes	Prevents the content from being indexed for search
Anthropic	Claude-User	Lookup requested by a user in Claude	Yes, according to Anthropic	Prevents the content from being retrieved in answers to users
Perplexity	PerplexityBot	Listing and linking sites in Perplexity search	Yes	Site stops being listed. According to Perplexity, it is not used to train foundation models
Perplexity	Perplexity-User	User action in Perplexity	Generally not	Little effect, since it ignores robots.txt rules
Google	Google-Extended	Gemini model training and grounding	Yes (control token)	Does not affect Google Search

Sources: OpenAI, Anthropic, Perplexity and Google. OpenAI also lists OAI-AdsBot, which only visits pages submitted as ads in ChatGPT and, according to the company, does not collect data for training.

Does Google-Extended affect AI Overviews?

There is no indication of that in the documentation. Google says Google-Extended is a standalone token that controls the use of content for training future generations of Gemini models and for grounding in Gemini apps and Vertex AI, and that it does not affect inclusion in Google Search and is not used as a ranking signal. It also has no HTTP user agent string of its own: crawling uses Google's existing strings. The documentation on Google's AI features treats AI Overviews and AI Mode separately, and the controls there are nosnippet, data-nosnippet, max-snippet and noindex. Reducing snippets also reduces visibility in those features.

For SEO professionals: Googlebot remains the only bot that matters for indexing and, as a result, for Search's AI features, which require an indexed page that is eligible for a snippet. If you block Googlebot by mistake while trying to block "AI bots," you leave Search entirely. That is why you should never use User-agent: * with Disallow: / as a shortcut.

Block or allow: what are the trade-offs?

The decision depends on what you want to protect and what you want to win. There is no universal answer, and in the documentation reviewed, no company claims that allowing training increases how often you are cited. What is known is the direct effect of each block.

Decision	Documented direct effect	Risk or cost
Block only the training bot	Signals that the content should not train models	The effect on mentions in answers is unknown
Block the search bot	Disappears from that product's search answers	You lose the chance to be cited and to receive visits from it
Allow everything	Maximum reach and use of the content	Your text may be part of model training
Block everything	Maximum control	Invisibility in AI and, for user actions, the block may not even be respected

For a site that depends on organic acquisition (blog, e-commerce, SaaS), the most common path is to allow the search bots and decide on training separately. If the content is the product (publishers, paid courses), the math changes, and blocking training may make sense for commercial and legal reasons. To understand what each search bot does with the page afterward, read how LLMs search, with RAG and grounding.

How do you write the robots.txt? Three examples

Each User-agent group applies only to the bots named in it. robots.txt lives at the root of each host (https://yoursite.com/robots.txt), and a subdomain needs its own file. Anthropic advises applying the rule to each subdomain you want to exclude.

Example A: appear in AI search, without training. This is the most common choice for sites that want traffic and do not want to feed training.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Example B: allow everything. The default behavior if the file does not mention these bots.

User-agent: *
Allow: /

Example C: block only one area. Useful for keeping AI bots out of account areas, carts or drafts without taking the blog offline.

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /area-do-cliente/
Disallow: /carrinho/

In plain English: think of robots.txt as a list of rules at a building's front desk. Each guard (bot) reads the part with its name on it. If its name does not appear, it follows the general rule, and if it does appear, it follows only its own group's rule.

For SEO professionals: RFC 9309 defines that the crawler picks the group that matches its product token, case-insensitively, and that the minimum size a crawler must parse is 500 KiB. Large files may have rules ignored at the end. Put the AI rules at the top or keep the file lean. Anthropic supports the non-standard Crawl-delay extension for ClaudeBot, but the extension is not part of the RFC.

Does robots.txt protect my content?

No. RFC 9309 itself says these rules are not a form of access authorization. They work with bots that choose to obey. Anyone who wants to collect content without asking can ignore the file and use any user agent. For content that should not be public, use authentication, a paywall or WAF rules.

Two practical consequences. First, according to Google's robots.txt documentation, it is for controlling crawl traffic, not for keeping a page out of Google; for that, use noindex. Second, Anthropic discourages blocking by IP, because it can prevent the bot from reading robots.txt itself and so from learning that you asked for exclusion.

How do you check the logs to see whether bots are visiting and are who they say they are?

Filter the access logs by user agent, validate the IPs against the official lists and check the response codes. User agents are free text, so a hit carrying GPTBot's name does not prove it came from OpenAI. Each company publishes IP ranges in JSON for this check.

1. Gather the logs. Export the access log from the server, CDN or WAF for the last 7 to 30 days.

2. Filter by token. In Nginx or Apache common log format, where the IP is the first field, this command counts hits per IP:

   grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
   

3. Validate the IPs. Download the official lists and compare: openai.com/searchbot.json, openai.com/gptbot.json, openai.com/chatgpt-user.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. Hits with the right user agent and an IP outside the list are suspicious.

4. Look for robots.txt. OpenAI says OAI-SearchBot and GPTBot may include a robots.txt marker in the user agent when fetching the file, which helps when the log does not show the path.

5. Look at the status codes. Many 403 or 429 responses to bots you want to allow indicate that the CDN or WAF is blocking before robots.txt comes into play. OpenAI recommends allowing the published IP ranges, and Perplexity recommends combining user agent and IP in the WAF rule.

6. Wait for the effect. For search, OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt change.

What if the bot ignores my robots.txt?

If the bot is ChatGPT-User or Perplexity-User, that is expected. OpenAI states that, because these are user-initiated actions, robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores the file. According to OpenAI, ChatGPT-User is also not used to decide whether content appears in search.

If the bot should obey (GPTBot, ClaudeBot, PerplexityBot), validate the IP first. If it is fake, block it in the WAF. If it is genuine, check the file: a syntax error, a file on the wrong host or a more specific rule overriding the general one solves most cases. After that, contact the company through the channels listed in its documentation.

Hypothetical example: an online store wants to appear in ChatGPT answers but does not want its product descriptions in training. The team adopts Example A and checks in the log that OAI-SearchBot gets 200 on category and product pages but 403 on /collections/. The culprit is a CDN rule that blocks "unknown bots." After allowing the official IP ranges, the team waits about 24 hours, repeats the check and records the date of the change. If mentions in ChatGPT shift later, it will know what changed on the site. To track mentions, see how to appear in ChatGPT.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Not directly. According to OpenAI, GPTBot is for training, and OAI-SearchBot is the ChatGPT search bot. The settings are independent. A site that blocks OAI-SearchBot does not appear in search answers, although it may still show up as a navigational link.

Do I need to allow OAI-SearchBot to appear in ChatGPT?

OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing the published IP ranges to make sure the site appears in search results. This does not guarantee a citation: appearing depends on the page being relevant to the question.

Perplexity-User ignores robots.txt. What can I do?

If you want to control access, use server-side controls such as authentication or WAF rules that combine user agent and IP, since Perplexity itself says this bot usually ignores robots.txt because a user triggers it. If the content is public, weigh whether the block is worth the lost visibility.

Does Google-Extended affect my ranking?

No. Google's documentation states that Google-Extended does not affect inclusion in Google Search and is not used as a ranking signal. It only controls the use of content for training and grounding Gemini models, in the apps and in Vertex AI.

Does robots.txt remove content that has already been collected?

None of the documentation reviewed promises that. Anthropic describes the block as a signal to exclude future materials, and OpenAI says disallowing GPTBot signals that the content should not be used in training. For requests about data already collected, use each company's contact channel.

Next step

With the bots configured, the next step is to decide whether to create a file for agents as well: read llms.txt: is it worth creating?. To track whether the changes alter your citations, Contentor offers AI Visibility, which monitors, for the questions you choose, whether ChatGPT cites your brand. Think of it as a complement to the log check, not a substitute.

Read also:

• Bing Webmaster Tools and IndexNow for AI search
• How LLMs search: RAG and grounding
• How to appear in ChatGPT