Blog · Answer Checkup team · Last checked

robots.txt for AI crawlers: what to allow and block

If an AI engine's search crawler can't fetch your pages, it can't cite them. robots.txt is where you decide which crawlers get in. The main thing to know is that AI companies run separate bots for search and for model training, so you can let AI search read your site and still say no to training.

Below are the bots from OpenAI, Perplexity, Google, Anthropic and Common Crawl, copy-paste rules, and how crawlers actually read the file.

Search bots, training bots and user fetchers

There are three kinds of bot to tell apart.

  • Search crawlers index pages so an engine can show and cite them in answers. Blocking these is what keeps you out of AI search.
  • Training crawlers collect content that may be used to train future models. Blocking these doesn't remove you from search answers.
  • User-triggered fetchers load a page because a person asked the assistant something. These may not follow robots.txt at all.

How each company describes its bots, in its own documentation:

Companyrobots.txt tokenWhat it's for
OpenAIOAI-SearchBotSearch. Surfaces sites in ChatGPT's search features
OpenAIGPTBotTraining of OpenAI's foundation models
OpenAIChatGPT-UserFetches a page for a user action. Robots.txt rules may not apply
PerplexityPerplexityBotSearch. Surfaces and links sites in Perplexity results. Not used to train foundation models
PerplexityPerplexity-UserFetches a page for a user's question. Generally ignores robots.txt
GoogleGooglebotGoogle Search, including AI Overviews and AI Mode
GoogleGoogle-ExtendedA control token, not a separate crawler. Covers Gemini training and grounding in Gemini Apps and Vertex AI
AnthropicClaude-SearchBotSearch. Indexes content to improve Claude's search results
AnthropicClaudeBotCollects content that could contribute to model training
AnthropicClaude-UserFetches a page when a Claude user asks a question
Common CrawlCCBotBuilds Common Crawl's open, public archive of web crawl data

Sources: OpenAI crawlers, Perplexity crawlers, Google common crawlers, Google AI features and your website, Anthropic's crawler page, Common Crawl's CCBot page.

A few details from those pages matter in practice:

  • OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers," though they can still appear as navigational links.
  • Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." For AI Overviews and AI Mode, the control is your robots.txt rules for Googlebot, plus snippet controls like nosnippet.
  • OpenAI says robots.txt changes can take about 24 hours to affect search results. Perplexity says changes may take up to 24 hours.
  • Anthropic says its bots honor robots.txt, including Claude-User. It warns that blocking Claude-User or Claude-SearchBot may reduce your visibility in Claude's search results.
  • Common Crawl warns that some crawlers falsely identify themselves as CCBot, and publishes its IP ranges so you can verify real visits.

Other AI companies run their own bots and publish their own pages. Check theirs before you add rules for them.

Copy-paste rules

Allow AI search, block training

This is a common choice: you stay in AI search answers and opt out of training.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Google-Extended covers both Gemini training and grounding in Gemini Apps, so blocking it opts you out of both. Leave it out of the file if you'd rather Gemini could use your pages. It doesn't affect Google Search either way. Drop the CCBot line if you don't mind being in Common Crawl's archive.

Allow everything

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

A missing robots.txt has much the same effect.

Block AI search too

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

Only do this if you really don't want to appear in those answers.

How crawlers read robots.txt

The rules are standardized in RFC 9309, the Robots Exclusion Protocol. The parts people trip on:

  • Bot names are matched case-insensitively. A crawler uses the group whose User-agent line matches its token, and falls back to User-agent: * if none does.
  • The most specific rule wins. When an Allow and a Disallow both match a URL, the one with the longer path is used. If they are equivalent, Allow should win.
  • Status codes matter. If /robots.txt returns a 4xx error, crawlers may access anything. If it returns a 5xx server error, crawlers must assume everything is disallowed. A broken server config can block every bot without you writing a single rule.
  • Changes aren't instant. Crawlers may cache the file and should not use a cached copy for more than 24 hours.
  • It isn't security. The RFC says these rules "are not a form of access authorization." Don't list private paths in robots.txt to hide them, because the file is public.

robots.txt is necessary, not sufficient

Our previous site, showsupin.com, was a one-page "launching soon" placeholder whose robots.txt allowed every crawler. On October 15, 2026 at 12:47 AM ET we ran our own report on it: 25 buyer questions, 3 times on each engine, through the OpenAI API with web search and the Perplexity API. None of the 150 answers mentioned or cited it, for a score of 0 out of 100. The answers made 510 citations to 294 different pages elsewhere, mostly articles, review pages and comparison posts. robots.txt opens the door. Being worth citing, and being written about elsewhere, gets you into the answer. Our AI SEO guide covers those steps, and our ChatGPT check has the full result.

Check what's actually blocking you

Before you decide you're open, check these:

  1. Load https://example.com/robots.txt in a browser. It should return plain text, not an error page or your homepage.
  2. Read every group, including old ones from a plugin or a previous developer. A stray User-agent: * with Disallow: / blocks everyone that has no group of its own.
  3. Check your firewall or CDN bot settings. Some have bot settings that block AI crawlers even when robots.txt allows them, and Perplexity's crawler page has guidance on allowlisting. OpenAI and Perplexity both publish IP ranges so you can allow their search crawlers: OpenAI's and Perplexity's.
  4. Look in your server logs for OAI-SearchBot, PerplexityBot and Claude-SearchBot in the user agent. If they never show up, something upstream is stopping them, or they haven't found you yet.
  5. Add an llms.txt file while you're there. It doesn't replace robots.txt, but it gives agents a clean summary. See our llms.txt guide.

FAQ

Should I block GPTBot?
That's a choice about training, not search. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one for ChatGPT search, and each setting is independent. You can block GPTBot and still allow OAI-SearchBot.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended doesn't affect inclusion in Google Search. AI Overviews and AI Mode follow your Googlebot rules and snippet controls.
Why do I still see ChatGPT-User or Perplexity-User in my logs after blocking?
Both companies describe these as fetchers triggered by a user's request. OpenAI says robots.txt rules may not apply, and Perplexity says its user fetcher generally ignores them.
How long until a robots.txt change takes effect?
OpenAI says about 24 hours for search results and Perplexity says up to 24 hours.
Is allowing AI crawlers enough to get recommended?
No. Our old placeholder site allowed every crawler and scored 0 out of 100. Access is the first step. See how to check if ChatGPT recommends your business to measure where you stand.

Check your own site

Run the free scan on the Answer Checkup homepage. It checks whether the main AI crawlers can reach your site, and whether you have llms.txt and structured data in place. For the research side, see our guide to generative engine optimization.

Answer Checkup is not affiliated with OpenAI, Perplexity, Google, Anthropic or Common Crawl.
Contact: hello@answercheckup.com

Sources

All pages fetched and checked on October 15, 2026.