FishingSEO
AI in SEO

Should You Block AI Crawlers? An SEO Decision Guide

By FishingSEO••9 min read

Short answer: Most sites that care about search traffic should not block every AI crawler. Block the bots used for model training if you don't want your content used that way. Keep the bots that power search and citations if you want to show up in AI answers. Never block Googlebot unless you mean to leave Google Search, because it also feeds Google's AI Overviews and AI Mode.

"AI crawler" covers several kinds of bots that do different jobs. Blocking them all at once is the most common mistake. This guide covers what each major bot does, what blocking it actually changes, and how to decide.

AI crawlers do three different jobs

The major AI companies now publish separate user agents for separate purposes. They fall into three groups:

  1. Training crawlers collect content that may be used to train future AI models.
  2. Search or indexing crawlers build an index so an AI product can find and cite your pages in answers.
  3. User-initiated fetchers visit a page in real time because a person asked the assistant to look something up.

Cloudflare uses nearly the same grouping. In its July 2026 update it labels the groups Search, Agent, and Training.

The rest of the decision depends on this split. Blocking a training crawler mostly affects how your content is used. Blocking a search crawler affects whether you appear in that product at all.

What the major bots do (documented facts)

CompanyTokenPurposeWhat blocking it does
GoogleGooglebotGoogle Search crawlingAlso controls AI Overviews and AI Mode. Google says AI is built into Search, so there is no separate AI opt-out
GoogleGoogle-ExtendedGemini training and groundingGoogle says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal"
OpenAIGPTBotTraining foundation modelsSignals your content should not be used for training
OpenAIOAI-SearchBotChatGPT search resultsYour site won't appear in ChatGPT search answers, though it can still appear as navigational links
OpenAIChatGPT-UserFetches pages for user requestsOpenAI says robots.txt rules "may not apply" because a user started the action
AnthropicClaudeBotTrainingExcludes your future content from training datasets
AnthropicClaude-SearchBotSearch qualityYour site may appear less often or less accurately in Claude's search results
AnthropicClaude-UserFetches pages for user requestsClaude can't retrieve your pages when users ask, which may reduce visibility
PerplexityPerplexityBotSearch results and linksYour site may not appear in Perplexity results. Perplexity says the bot is not used for foundation-model training
PerplexityPerplexity-UserFetches pages for user requestsPerplexity says this fetcher "generally ignores robots.txt rules"
AppleApplebot-ExtendedApple foundation-model trainingPages can still appear in Apple search results, and the rule isn't used in ranking

Two details stand out:

  • Google-Extended and Applebot-Extended are not separate crawlers. They are robots.txt tokens that control how content collected by Googlebot or Applebot may be used.
  • Blocking Google-Extended does not remove you from AI Overviews. Google's documentation points to nosnippet, data-nosnippet, max-snippet, and noindex for limiting what appears in its AI features. Those controls limit your regular search snippets too. If you're thinking about the noindex route, see our guide on When to Noindex AI-Generated Pages: An SEO Decision Guide for the trade-offs.

What robots.txt can and cannot do

Robots.txt is a request, not a lock. The standard behind it, RFC 9309, says its rules "are not a form of access authorization." It also says the protocol "is not a substitute for valid content security measures."

In practice, this means:

  • Compliant bots follow it. The bots listed above are documented by their operators as following robots.txt, except for the user-initiated fetchers noted in the table.
  • Some bots may not follow it. In August 2025, Cloudflare reported that Perplexity used undeclared crawlers with generic browser user agents after its declared bots were blocked. This is Cloudflare's account of its own tests. In the same tests, Cloudflare reported that ChatGPT-User respected robots.txt.
  • Blocking isn't retroactive. Anthropic, for example, describes a ClaudeBot block as excluding future content. It doesn't address material collected before the block.
  • If content must stay private, put it behind authentication. Robots.txt can't protect it.

Check your CDN before you edit robots.txt

Your hosting or CDN settings may already be blocking AI bots without your knowledge. Cloudflare began blocking AI crawlers by default for new domains in July 2025.

In July 2026, Cloudflare announced a bigger change that took effect on September 15, 2026. According to its announcement:

  • New domains block Training and Agent bots by default on pages that show ads. Search bots stay allowed.
  • Crawlers that combine search and training are judged by their most restrictive purpose. Cloudflare named Googlebot, Applebot, and Bingbot as examples that could be blocked for customers who chose to block training.

Later, Cloudflare's September 15, 2026 press release said it now designates some crawler operators as "Accountable," including Google, Apple, and Microsoft. Accountable operators must offer a separate way to opt out of training, among other criteria. Under the new "Disallow AI Training" setting, only mixed-use crawlers that are not Accountable are blocked when a site blocks training.

These settings are new and still changing. If you use Cloudflare, open your security settings and confirm that search crawlers, Googlebot especially, are allowed. A setting meant to block AI training should never quietly remove you from search.

A decision framework

Work through these questions in order.

1. Do you depend on search traffic?

If yes, keep Googlebot and Bingbot allowed. That isn't really optional. AI-search bots such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot are a separate choice, but blocking them removes you from products where people increasingly find sources.

2. Do you object to your content being used for training?

If yes, block the training tokens: GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended. The companies above document that these blocks are separate from their search features. Google and Apple state that the training tokens don't affect search ranking.

3. Is your content your product?

Paywalled publishers, course creators, and proprietary data providers have more to lose when an AI answer replaces a visit. They may reasonably block more, including AI search bots, and pursue licensing or paid access instead. Cloudflare's 2026 report says more than 50 publisher–AI licensing agreements have been signed since 2023.

4. Does your business benefit from being cited?

SaaS companies, local businesses, consultants, and most brand blogs usually gain when an AI assistant names them or links to them. For these sites, blocking AI search and user-initiated fetchers is usually a cost with little benefit.

5. Is crawl load the actual problem?

If bots are straining your server, you may not need a full block. Anthropic documents support for Crawl-delay, and CDN-level rate limiting is another option.

Hypothetical scenarios

These examples are illustrative, not case studies.

  • A B2B software blog wants to be cited when buyers ask ChatGPT or Claude for tool comparisons. Reasonable choice: allow all search and user bots. Block training bots only if the company has a policy reason to.
  • A subscription news site sees AI answers summarizing its reporting. Reasonable choice: block training tokens, consider limiting AI search bots, use nosnippet or paywall markup on premium pages, and review CDN-level controls.
  • A hobby recipe site runs on ad revenue. Reasonable choice: this is a real trade-off. AI answers may reduce visits, but AI search may also send some traffic. Blocking training while keeping search bots is a middle path, and the site can review referral data over time.

Example robots.txt: block training, keep search

# Block AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Allow AI search and citations
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

A few practical notes:

  • Robots.txt applies per host, so each subdomain needs its own file. Anthropic says this explicitly. If you're deciding where content should live, see Subdomains vs. Subfolders: An SEO Decision Guide.
  • OpenAI and Perplexity say changes can take about 24 hours to take effect in their systems.
  • Bot names change. Check each operator's documentation before you edit, and again from time to time.
  • Anthropic warns that blocking by IP address can stop bots from reading your robots.txt, which makes opt-outs less reliable.

Revisit the decision regularly

This isn't a one-time setting. New bots appear, operators split or rename user agents, and CDN defaults change, as Cloudflare's September 2026 update showed. Check your server logs or CDN analytics for which bots actually visit and how often. Add robots.txt to the reviews you already run, in the same way you would handle AI Content Update Triggers: An SEO Decision Guide. Also write your AI crawler policy into your How to Build an AI SEO Style Guide in 1 Hour so your team applies it consistently.

Conclusion

Whether to block AI crawlers is really a set of separate decisions. Training bots, AI search bots, and user-initiated fetchers each have different consequences, and the major operators let you control them separately. For most sites that rely on search, the reasonable default is to keep Googlebot and the AI search bots allowed, and to block training tokens only if you object to that use. Robots.txt is voluntary and not retroactive. Your CDN may also be making choices for you, so check its settings as well as your robots.txt file.

References

Back to all posts