How to Allow AI Crawlers in robots.txt (2026 Guide)

TL;DR

To allow AI crawlers in robots.txt, remove any blocking rule, then allow the search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot.
AI bots do three different jobs: training, search indexing and user-triggered fetching. Allowing one kind does not allow the others.
Gemini has no crawler of its own. Google-Extended is only a control token, and Google says it does not affect Search inclusion or rankings.
A CDN, firewall, security plugin or a robots.txt file that returns a server error can block bots even when your file says Allow.
Robots.txt only removes a barrier. It does not earn a mention in ChatGPT, Claude, Gemini, Grok or Perplexity, and nobody can promise you one.

How do I allow AI crawlers in robots.txt?

First, delete any rule that blocks them. Then give each AI search crawler its own group with Allow: /. Three names matter most: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude) and PerplexityBot (Perplexity). Training bots like GPTBot and ClaudeBot are a separate decision. Gemini has no crawler of its own. Keep in mind that robots.txt only removes a barrier. It doesn't earn you mentions.

Here is a starter block you can paste into a plain text file at yoursite.com/robots.txt:

# AI search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# User-triggered fetchers (optional, see notes below)
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Put your real sitemap address on the last line. If your current file blocks private folders such as an admin area, copy those Disallow lines into each group above. I explain why in the section on how rules are read.

This block allows search and user fetches. It says nothing about training. If you also want to allow training bots, add groups for GPTBot, ClaudeBot and Google-Extended. If you want to block training, see the middle option further down.

First, check whether you are blocking AI bots right now

Open yoursite.com/robots.txt in a private browser window. You may be blocking AI bots without ever having decided to. The mo.agency audit points out that some CMS platforms and security plugins add these rules automatically, and you might not even know they're there.

Look for these patterns:

A blanket block: User-agent: * followed by Disallow: /. This blocks every crawler that follows robots.txt, including Google and Bing.
A Google-only allowlist: User-agent: * with Disallow: /, plus a Googlebot group with Allow: /. That only allows Google. Every AI crawler is blocked.
Named AI blocks: groups for GPTBot, ClaudeBot, PerplexityBot or Google-Extended with Disallow: /, often added by a plugin or copied from a "block AI" guide.
Outdated names: groups for anthropic-ai, Claude-Web or anything mentioning Bard. More on those below.

Before you delete a blanket block, ask whoever built the site why it's there. A developer may have added it while the site was under construction and never removed it after launch.

The three kinds of AI bots, and why the difference matters

Short answer: AI bots do three different jobs. Training crawlers collect content for future models. Search crawlers build the index an assistant uses when it answers with sources. User-triggered fetchers load a page because a person asked about it. Each kind follows different rules.

Bot (vendor)	Job	Follows robots.txt?
GPTBot (OpenAI)	Training	Yes
OAI-SearchBot (OpenAI)	ChatGPT search	Yes
ChatGPT-User (OpenAI)	User-triggered fetch	May not apply
ClaudeBot (Anthropic)	Training	Yes
Claude-SearchBot (Anthropic)	Claude search index	Yes
Claude-User (Anthropic)	User-triggered fetch	Yes
PerplexityBot (Perplexity)	Perplexity search	Yes
Perplexity-User (Perplexity)	User-triggered fetch	Generally no
Google-Extended (Google)	Control token only	Token, not a crawler

Here's what this means in practice. OpenAI says each setting is independent of the others. For example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. For both Anthropic and OpenAI, blocking the training bot does not block the search bot or the user-requested fetcher.

It also works the other way. If you allow GPTBot but block OAI-SearchBot, you've given away training access and still stepped out of ChatGPT search.

Platform by platform: ChatGPT, Claude, Gemini, Perplexity and Grok

ChatGPT (OpenAI)

Allow OAI-SearchBot. OpenAI's documentation is direct about it: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI recommends allowing OAI-SearchBot in your robots.txt file and allowing requests from its published IP ranges.

ChatGPT-User is different. OpenAI warns that robots.txt rules may not apply to ChatGPT-User , because a person starts the fetch. You can still add an allow group for it. Just don't count on robots.txt to control it either way.

On timing, a secondary guide to OpenAI's docs says that for search results, it can take about 24 hours after a robots.txt update for systems to adjust. Treat that as a rough guide, not a promise. More detail is in how to get found in ChatGPT.

Claude (Anthropic)

Allow Claude-SearchBot, and Claude-User too if you want Claude to read your pages when someone asks about them. Anthropic updated its crawler documentation with a formal breakdown of its three web crawlers and their individual purposes , as reported by Search Engine Journal. ClaudeBot handles training, Claude-SearchBot handles search indexing, and Claude-User handles fetches a user triggers.

Anthropic handles user fetches differently from OpenAI and Perplexity. Anthropic says all three of its bots honor robots.txt, including Claude-User. So if you block Claude-User, you really do stop Claude from fetching your pages for users. Anthropic says blocking it "prevents our system from retrieving your content in response to a user query" and may reduce your visibility.

Clean up old tokens. Before ClaudeBot, Anthropic operated under the user-agent strings Claude-Web and Anthropic-AI. Both are now deprecated. If your robots.txt still references those strings, you may have outdated and ineffective directives in place. See how to get found in Claude for the rest of the picture.

Gemini, Google AI Overviews and AI Mode (Google)

There's no separate Gemini crawler to allow. Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings, and the robots.txt token is used in a control capacity.

According to Google, the token manages whether crawled content may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google also says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" .

Google AI Overviews and AI Mode are a different matter. They run on the regular Search index, so Googlebot is the bot that matters there. One guide sums it up: disallowing Google-Extended only opts your content out of Gemini training and grounding, while AI Overviews on the results page are governed separately through Googlebot and snippet controls. Google's AI features page describes the same setup. Never block Googlebot or Bingbot unless you mean to leave those search engines entirely. For the next steps, read how to get found in Gemini and how to get found in Google AI Overviews.

Perplexity

Allow PerplexityBot. Perplexity recommends allowing PerplexityBot in your site's robots.txt file and permitting requests from its published IP ranges so your site can appear in its search results. Perplexity also says PerplexityBot is not used to crawl content for AI foundation models.

Perplexity-User works differently. Since a user requested the fetch, this fetcher generally ignores robots.txt rules. One honest gap: Perplexity does not document exactly what happens to citations if you block PerplexityBot . Be skeptical of anyone who claims to know. More in how to get found in Perplexity.

Grok (xAI)

I can't give you a Grok line, because xAI hasn't published one. One reference that checks claims against vendor docs reports that xAI publishes no crawler documentation page. Another notes there is no published user agent and no IP range, which is why guides telling you to allow "xAI-Bot" can't show where that string came from.

Third-party registries list different names, including GrokBot, xAI-Grok and xAI-SearchBot. None of them trace back to xAI. There's also a separate browsing agent. Grok Bot is an AI agent product that xAI launched in August 2026 , and robots.txt can ask GrokBot to stay away, but it does nothing to Grok Bot. If you don't block every bot with User-agent: *, you aren't blocking Grok through robots.txt. See how to get found in Grok for what you can influence.

Apple and others

Apple's Applebot-Extended works much like Google-Extended. It's a token that tells Apple whether your content may be used to train its models. The regular Applebot handles search features. Details are in Apple's Applebot page. If Apple matters to your customers, the listing side is covered in Apple Business Connect setup.

How robots.txt rules are actually read

Short answer: a crawler picks the one group that names it and ignores everything else. If no group names it, it uses the * group. Inside that group, the longest matching path wins. The standard behind this is RFC 9309, the Robots Exclusion Protocol.

A named group beats *. That's why a named OAI-SearchBot group works even when User-agent: * says Disallow: /. Group order in the file doesn't matter. The match is by name, so you don't need to put AI groups "above" anything.
Side effect: a named bot skips the * rules completely. If your * group blocks /wp-admin/ or /cart/, copy those lines into each named group you add.
The longest match wins. If Allow: /blog/ and Disallow: / both match a URL, the longer, more specific rule applies.
One group can name several bots. You can stack User-agent: lines over a single set of rules, which keeps the file short.
Paths are case sensitive. /Blog/ and /blog/ are different paths.
Each host needs its own file. Rules at www.yoursite.com/robots.txt don't cover shop.yoursite.com.
A missing file means no limits. If robots.txt returns "not found," crawlers may fetch anything.
A broken file can mean "blocked." Under RFC 9309, if robots.txt is unreachable because of a server error, crawlers should assume the whole site is disallowed. A flaky host can quietly lock out every compliant bot.

Also keep in mind what robots.txt is not. Google's Gary Illyes said robots.txt, as RFC 9309 defines it, governs which URLs a crawler may request, not what happens to the content afterward.

Allow search, block training: the middle option

Yes, you can ask to be found in AI answers without offering your content for training, as long as the vendor splits its bots. Here's the setup:

# Allow AI search
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Google-Extended needs its own decision. Disallowing it opts out of Gemini training, and it also opts out of grounding in Gemini Apps. For Gemini, Google's documented control is Google-Extended, and it covers both training and grounding. If you want Gemini to be able to ground answers on your pages, leave Google-Extended allowed.

Know the limits here. This is a signal you send one vendor at a time. It doesn't remove anything a model has already learned, and it isn't a legal wall.

When robots.txt says Allow but bots still get blocked

Short answer: something in front of your site refuses the bot before it ever reads the file. Robots.txt is a polite request, and a firewall doesn't read it.

This is common enough that vendors plan for it. Perplexity's docs include step-by-step Cloudflare WAF settings to allow its bots , plus instructions for AWS WAF. One SEO who built a bot access checker described sites where bots were getting blocked at the infrastructure level by CDNs, firewalls, and rate limiters even though robots.txt welcomed them.

Check these places, in this order:

Your CDN or host dashboard. Look for bot protection, "block AI bots" or AI crawler toggles.
Your security plugin, if you run WordPress or a similar CMS.
IP and country rules. If you allow by IP, use each vendor's published list. OpenAI and Perplexity both publish theirs. Anthropic publishes IP ranges at claude.com/crawling/bots.json , according to one reference.
Your robots.txt response itself. It should load with a normal response, not a server error or a login page.
Your page content. A secondary source reports that AI crawlers do not run JavaScript, so a page whose text only appears after scripts load looks empty to them. Make sure your key text is in the HTML your server sends.

How to check it worked

You can't see inside an AI index, but you can confirm the barrier is gone.

Fetch the live file. Load yoursite.com/robots.txt in a private window. Make sure it's the file you edited, not a cached or plugin-generated copy.
Read your server logs. Search for OAI-SearchBot, Claude-SearchBot, PerplexityBot and the others. Check that they get normal responses, not blocks or challenges.
Verify the IPs. Anyone can fake a user-agent string. Compare the requesting IPs with the vendors' published lists.
Check your CDN's bot analytics for blocked requests from those names.
Give it time. RFC 9309 suggests crawlers shouldn't cache robots.txt for more than 24 hours in normal conditions, so changes aren't instant.

If the bots arrive and get a normal response, you've done what robots.txt can do. To see what the assistants actually say about you, follow how to check if AI recommends your business. For a broader review, use the AI visibility audit checklist.

Common mistakes I see in AI robots.txt files

Allowing GPTBot and assuming that covers ChatGPT search. It doesn't. OAI-SearchBot is the search control.
Using dead tokens. anthropic-ai and Claude-Web are deprecated. Bard is now Gemini, and Google-Extended is the token.
Treating Google-Extended as a crawler. It never shows up in your logs. It's a control token only.
Forgetting that named groups skip the * rules. This can open private folders you meant to keep closed.
Copying a Grok token from an unsourced list. It does no harm, but don't treat it as a control.
Adding llms.txt and expecting a lift. In a vendor study relayed on X, Ahrefs logs across 137K domains showed 97% of valid llms.txt files got zero requests in May. See llms.txt for small business.

Robots.txt only removes a barrier: what actually earns mentions

Allowing a crawler means it can read your site. It doesn't mean any assistant will cite or recommend you.

Vendor studies make the same point. In a series of Ahrefs studies summarized by Ahrefs' Tim Soulo, ChatGPT cited about half of the URLs it retrieved, and "best X" listicles were the most-cited page type. These are vendor studies, not settled fact. But the direction is clear: getting fetched and getting cited are two different steps.

The work that tends to matter happens outside robots.txt:

Pages that answer the buyer's question directly, with specific facts, prices where you can share them, and clear service descriptions.
Consistent business facts everywhere: the same name, phone, address or service area, and description on Google, Bing, Apple and the directories in your industry.
Third-party mentions: reviews, roundups, comparison pages and directories that AI tools already cite. For local searches, BrightLocal data relayed on X found that individual business websites got 42% of citations, so your own site still counts.
Structured data that matches the visible page. See local business schema for AI search.

If a competitor keeps getting named and you don't, why ChatGPT recommends my competitor walks through the usual reasons.

Where Get Found with AI fits

I run Get Found with AI for businesses across the United States. The goal is to help you get found, cited and recommended in ChatGPT, Claude, Gemini, Grok, Perplexity and Google's AI features. A robots.txt check is the first step, because there's no point doing the rest if bots can't read your site. Most of the work comes after that: clear answer pages, consistent business facts, structured data and third-party presence. You can see the process on how it works and the options on services.

I can't promise any placement, citation or timeline. AI answers change from run to run, and no one controls them.

How I put this together

On October 22, 2026, I searched "allow ai crawlers robots.txt" and read the top three US results. Then I checked every bot claim against vendor documentation where it was available.

The robots.txt Audit, mo.agency. Good audit steps and a useful warning about plugins. But its recommended block leaves out OAI-SearchBot and Claude-SearchBot. It says GPTBot and ClaudeBot "drive search citations" , while OpenAI and Anthropic describe those as training bots. It also says to place allow rules above the wildcard, but under RFC 9309 order doesn't matter.
How to block AI Crawler Bots using robots.txt file, nixCraft. Clear syntax, but it's written for blocking, still refers to Bard, and doesn't separate training from search. A reader who follows it could drop out of ChatGPT search without meaning to.
LLMs.txt and Robots.txt, Goodie. A good case for not blocking AI bots. But its bot list is vague. It names "Claude or Anthropic" as tokens and calls Google-Extended "Google's AI data crawler" , when Google documents it as a control token with no crawler of its own.

What this guide adds: the three-job split with vendor sources, the correct search crawler names, Anthropic's different handling of Claude-User, the facts on Google-Extended, the RFC 9309 server-error trap, CDN and WAF checks, a testing checklist, and an honest note that xAI hasn't documented a Grok crawler.

Frequently asked questions

Does allowing GPTBot get my business into ChatGPT search?

No. OpenAI describes GPTBot as the training control and OAI-SearchBot as the search control. To be eligible for ChatGPT search answers, allow OAI-SearchBot. Allowing either one only removes a barrier. It does not guarantee you will appear.

Will blocking Google-Extended hurt my Google rankings?

Google says no. Its crawler documentation states that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether Google may use your content for Gemini training and for grounding in Gemini Apps and Vertex AI.

Can I let AI search tools use my site but block AI training?

Yes, for vendors that split their bots. Allow OAI-SearchBot and disallow GPTBot for OpenAI. Allow Claude-SearchBot and disallow ClaudeBot for Anthropic. Disallow Google-Extended for Gemini training and grounding. This is a signal you send one vendor at a time. It is not a legal wall.

Do ChatGPT-User, Claude-User and Perplexity-User follow robots.txt?

It depends on the vendor. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says all three of its bots honor robots.txt, including Claude-User, so blocking Claude-User does stop those fetches.

What robots.txt user-agent does Grok use?

xAI has not published crawler documentation as of October 22, 2026. Third-party registries list different names, and none of them come from xAI. I won't give you a Grok user-agent line I can't trace to xAI. If xAI publishes one, use xAI's own docs.

Why are AI bots still blocked when my robots.txt says Allow?

Usually a CDN, firewall, host setting or security plugin refuses them before they ever read the file. Another cause: if your robots.txt returns a server error, RFC 9309 tells crawlers to treat the whole site as disallowed. Check your edge settings and confirm the file loads with a normal response.

Will allowing AI crawlers guarantee that AI assistants recommend my business?

No. Allowing crawlers only means they can read your site. Whether an assistant mentions or cites you depends on many other signals. I can't promise any placement, citation or timeline, and you should be wary of anyone who does.

How to Allow AI Crawlers in robots.txt: Let ChatGPT, Claude, Gemini and Perplexity Read Your Site