Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the crawler used to surface sites in ChatGPT search. You can block GPTBot and still allow OAI-SearchBot.

What is Google-Extended?

Google-Extended is a robots.txt control token that lets sites opt out of their content being used for Google's Gemini models and related AI training. It is not a separate crawler and does not affect whether a site appears in Google Search, including AI Overviews.

Do AI user agents respect robots.txt?

Training and search crawlers from the major companies say they respect robots.txt. Agents that fetch a page because a user asked can behave differently; OpenAI notes that robots.txt rules may not apply to user-initiated ChatGPT-User requests.

How long does a robots.txt change take to apply?

OpenAI says it can take around 24 hours for its systems to adjust after a robots.txt update. Other crawlers vary, so allow a few days before checking logs.

Should I block Common Crawl's CCBot?

CCBot collects an open web dataset that many organisations use to train models. Blocking it reduces the chance of a site being included in future training datasets built from Common Crawl. It does not affect Google, Bing or the AI search crawlers of OpenAI, Anthropic or Perplexity, so blocking it has little effect on AI search visibility.

AI Crawler User Agents: GPTBot, ClaudeBot, PerplexityBot and How to Control Them

Many clients have asked their agency to "block AI" in robots.txt. Some agencies did it with a single list of bot names. A good number of those sites then quietly disappeared from ChatGPT search, Perplexity answers and Claude's web results, because the list blocked the crawlers that power AI search along with the ones that collect training data.

AI companies now run several crawlers with different jobs, and the difference matters. This reference lists the main AI user agents, explains what each one does, and gives robots.txt patterns for the decisions clients actually want to make.

The three jobs AI crawlers do

Nearly every AI company separates its crawling into three roles.

Training crawlers collect content that may be used to train future models.
Search crawlers build the index that the company's AI search uses to find and cite pages.
User-triggered agents fetch a specific page in real time because a user asked about it or shared a link.

Blocking a training crawler stops future training use. Blocking a search crawler or user agent stops the site from being found or cited in that company's AI answers. Most clients want the first and not the second.

The main AI user agents

Always confirm current names against each company's documentation, because they change. As of late 2026, the main agents are:

Company	User agent	Role	Notes
OpenAI	GPTBot	Training	Blocking it does not affect ChatGPT search
OpenAI	OAI-SearchBot	Search index	Used to surface sites in ChatGPT search
OpenAI	ChatGPT-User	User-triggered fetch	robots.txt may not apply to user-initiated requests
Anthropic	ClaudeBot	Training	Blocking signals exclusion from future training
Anthropic	Claude-SearchBot	Search index	Improves Claude's search results
Anthropic	Claude-User	User-triggered fetch	Blocking it can reduce visibility in user-directed search
Perplexity	PerplexityBot	Search index	Surfaces and links sites in Perplexity
Perplexity	Perplexity-User	User-triggered fetch	Fetches pages for user actions; compliance claims vary
Google	Googlebot	Search, including AI Overviews and AI Mode	Blocking it removes the site from Google Search
Google	Google-Extended	Control token, not a crawler	Opts out of Gemini training and grounding use
Apple	Applebot	Search for Siri and Spotlight	Renders JavaScript
Apple	Applebot-Extended	Control token	Opts out of Apple's AI training use
Microsoft	Bingbot	Bing search, which Copilot draws on	Blocking it affects Bing and Copilot
Common Crawl	CCBot	Open dataset used by many trainers	Blocking reduces inclusion in future datasets
Meta	meta-externalagent	Training	Used for Meta's AI models
ByteDance	Bytespider	Training	Used for ByteDance's models

Two entries need special care.

Google-Extended is not a crawler. Googlebot does the crawling, and Google-Extended is a token in robots.txt that tells Google not to use content for its Gemini models. It has no effect on whether the site appears in Google Search, AI Overviews or AI Mode. Blocking Googlebot to avoid AI Overviews would remove the site from Google entirely.

User-triggered agents such as ChatGPT-User behave like a person's browser acting on request. OpenAI notes that robots.txt rules may not apply to them. They also tend to arrive when someone has asked about the site specifically, which is usually a visit you want.

robots.txt patterns for common decisions

Allow everything (the default)

If a client has no objection to AI use, the simplest robots.txt allows all crawlers and points to the sitemap. Most sites should start here.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Block AI training, stay visible in AI search

This is what most clients mean when they ask to "block AI". It blocks the main training crawlers and control tokens while leaving search and user agents alone.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Applebot and Bingbot are all still allowed by the final rule.

Block a private section from every crawler

For areas such as account pages, staging copies or internal tools, a path rule under the wildcard agent applies to everyone. Remember that robots.txt is a request, not security. Truly private content needs authentication.

User-agent: *
Disallow: /account/
Disallow: /internal/

Block all AI, including search

Some publishers decide to block AI entirely. That is a legitimate business decision, but make sure the client understands the trade-off: they will not be cited in ChatGPT search, Claude or Perplexity answers. Add the search and user agents to the blocked list as well, and accept that user-triggered agents may not honour the rule.

Which agents should a client allow?

The right setup depends on what the client wants from AI. This decision table covers the common positions.

Client position	Training crawlers	Search crawlers	User-triggered agents	Typical client
Maximum visibility	Allow	Allow	Allow	Most businesses that sell products or services
Visible but not used for training	Block	Allow	Allow	Brands cautious about training use
Licensing content to AI companies	Block, or allow per agreement	Allow or per agreement	Allow	Large publishers
No AI involvement	Block	Block	Block, accepting limits	Sites with strict legal or contractual constraints

For most businesses, the first two rows are the only realistic choices. Being visible in AI search is now part of being visible at all, and blocking search agents trades that away for little benefit.

Whatever the choice, document it. Write the decision and the date in the client file so that the next person who edits robots.txt knows why each rule exists.

Mistakes we see on client sites

Blocking everything with "AI" in a list from a blog post. Old lists often include search agents that did not exist when the list was written.
Blocking Googlebot to avoid AI Overviews. This removes the site from Google Search.
Forgetting that CDNs and firewalls also block bots. A security rule that challenges unknown bots can block AI search crawlers even when robots.txt allows them. Check firewall logs as well as robots.txt.
Wildcard rules that override specific ones. Crawlers apply the most specific matching group. Test the file instead of assuming.
No sitemap reference. Search crawlers use it to discover pages.

How to verify a crawler is genuine

Anyone can send a request claiming to be GPTBot. To confirm a request is genuine, check its IP address against the ranges each company publishes. OpenAI and Perplexity publish JSON lists of IP ranges for their bots, and Google and Bing support reverse DNS verification. Our guide to AI bot log file analysis covers verification and reporting in detail.

Testing your rules

Validate the syntax. Use Search Console's robots.txt report for Googlebot and a robots.txt testing tool for other agents.
Test specific agents and paths. Check that GPTBot is blocked and OAI-SearchBot is allowed for a sample of important URLs.
Wait, then check logs. OpenAI says changes can take about 24 hours to apply. Review server logs after a few days to confirm the expected agents are visiting or staying away.
Re-run your AI citation checks. If search agents are allowed and pages are readable, citations should not drop. See AI share of voice.

Access is only the first step

Allowing a search crawler means it can fetch the page. It does not mean it can read it. Most AI crawlers do not execute JavaScript, so content that only appears after scripts run is invisible to them even with perfect robots.txt rules. See can AI crawlers read JavaScript?.

And llms.txt does not change any of this. It is a content summary with no access-control role, and AI crawlers rarely request it. See does llms.txt actually work?.

How Rankwyn helps

Rankwyn's audits check every connected site's robots.txt against the current list of AI training, search and user agents and flag rules that block AI search by mistake. Proposed robots.txt changes go through the approval queue, so nothing changes on a client site without sign-off. See the AEO Citation Radar.

Key takeaways

AI companies separate training, search and user-triggered crawlers, and each needs its own decision.
Blocking GPTBot, ClaudeBot or Google-Extended stops training use without removing a site from AI search.
Blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot or the user agents can remove a site from those companies' AI answers.
Google-Extended is a control token, not a crawler, and never affects Google Search visibility.
Check firewalls and CDNs as well as robots.txt, verify bots by IP, and confirm results in logs.

The bottom line

"Block AI" is not one decision. Most clients want to keep their content out of training while staying visible in AI search, and that is easy to do with per-agent rules. Audit every client's robots.txt and firewall rules against the current list of agents, fix accidental search blocks first, and confirm the result in server logs.