AI crawlers: the full list, and how to let them in
An AI answer is assembled from pages a crawler was allowed to fetch. Block the wrong one and your pages never reach the shortlist — no error, no notification, just an answer that quotes someone else.
These are the 19 crawler tokens Lumicite tests, grouped by what they are for. The distinction that matters most is between the crawler that collects training data and the one that decides whether a page can be cited: they belong to the same companies, they have different names, and blocking one does not affect the other.
Five kinds of crawler, five different consequences
Search index
Most AI features sit on top of an ordinary search index. Blocking the index crawler removes you from the candidate set for AI Overviews and Copilot.
AI retrieval
These decide whether a page can enter one product's answer. Block them and that product's answers quote someone else.
On request
Not a crawl: a person pastes a link into a chat and the product fetches it live. Block it and links your own users share are unreadable.
Training
Collects training data only. Blocking it changes training; it does not change what can be cited today.
Mixed
The vendor has not separated search from training publicly. Assume both.
The 19 crawlers
robots.txt matches on the token in the first column, not on the company name. A rule written for GPTBot has no effect on OAI-SearchBot.
| Token | Run by | Purpose | What it means for citations |
|---|---|---|---|
Googlebot | Search index | The crawler behind Google's index. AI Overviews and AI Mode draw their pages from it. | |
Bingbot | Microsoft | Search index | Bing's index crawler. Copilot and several AI search products are built on that index. |
Baiduspider | Baidu | Search index | Baidu's search index, which Baidu's own AI answers also read. |
Sogou web spider | Sogou (Tencent) | Search index | Sogou's search index. Sogou is part of Tencent. |
360Spider | 360 Search | Search index | 360 Search's index. |
YisouSpider | Shenma (Alibaba) | Search index | Shenma's mobile search index. |
Bytespider | ByteDance | Mixed | ByteDance's crawler. The split between search and training is not fully published, so treat it as both. |
OAI-SearchBot | OpenAI | AI retrieval | Decides whether a page can appear in ChatGPT's search results. |
Claude-SearchBot | Anthropic | AI retrieval | Builds the index Claude uses when it searches the web. |
PerplexityBot | Perplexity | AI retrieval | Decides whether a page can be quoted in a Perplexity answer. |
ChatGPT-User | OpenAI | On request | Fetches a link when someone pastes it into a chat and asks ChatGPT to open it. |
Claude-User | Anthropic | On request | The same, for a link a person asks Claude to open. |
Perplexity-User | Perplexity | On request | The same, for links opened from Perplexity. |
GPTBot | OpenAI | Training | Training data collection. Blocking it opts out of training and changes nothing about ChatGPT search. |
ClaudeBot | Anthropic | Training | Training data collection. |
Google-Extended | Training | Only a robots.txt token. It controls Gemini training and grounding; it does not affect Google Search. | |
Applebot-Extended | Apple | Training | Only a robots.txt token. It controls Apple's foundation-model training. |
CCBot | Common Crawl | Training | Common Crawl's open corpus, which many models are pretrained on. |
meta-externalagent | Meta | Training | Meta's training data collection. |
A robots.txt that lets retrieval in
The shortest version that works: allow everything, and be explicit about the crawlers that decide whether you can be cited. Separate User-agent lines stacked above one set of rules form a single group, so this is not 19 groups, it is two.
User-agent: * Allow: / # Crawlers that decide whether a page can be cited User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Allow: /
Being explicit is not what makes it work — User-agent: * already allows them — but it survives the next edit. The typical accident is a blanket rule added later by someone who read that AI crawlers should be blocked, without knowing that the retrieval crawlers are a different list.
If you want to keep pages out of the index, disallow the path rather than the crawler: Disallow: /private/ works the same for all of them, and the audit will show it.
Blocking training is a separate decision
Google-Extended and Applebot-Extended are not crawlers. They are tokens that only say “do not use what you already crawl for training”, and blocking them changes nothing about Google Search or Applebot.
For OpenAI, Anthropic and Common Crawl the split runs through separate crawlers: GPTBot, ClaudeBot and CCBot collect training data, while OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User are what put a page in front of a model at answer time. A site can refuse training and still be quoted; the two are independent switches, and the site owners who lose citations are usually the ones who flipped both.
robots.txt is not the only gate
A crawler can also be stopped before it ever reads the file. A CDN or WAF rule that challenges “bot-looking” traffic, a firewall allowlist of search engine ranges, or a hosting panel toggle labelled “block AI bots” all decide the outcome, and none of them appear in robots.txt.
The only reliable test is the response itself. Fetch a page with the crawler's own User-Agent and look at the status code — a 403, a 429 or a JavaScript challenge page all mean the same thing to an answer engine: nothing to read.
Check your own site
The free audit reads your robots.txt for all 19 tokens and reports which ones are blocked, then fetches a page the way a browser would and grades what an AI answer could quote from it.