AI crawler directory
Search, training and user-triggered crawlers are separate agents. Blocking the wrong one can take you out of AI answers without stopping training. Every entry links the operator's own documentation.
AI search & assistants
Indexes pages so they can be found and cited in search or AI answers.
OpenAI's search crawler.
Anthropic's search crawler.
Google's main search crawler.
Apple's search crawler for Spotlight, Siri and Safari features.
Perplexity's search crawler.
Microsoft's standard search crawler.
ByteDance's crawler, documented as the user agent for Toutiao search.
Meta's indexing crawler.
Amazon's search crawler for experiences such as Alexa.
DuckDuckGo's crawler for its AI-assisted answers.
Baidu's search crawler.
User-triggered fetchers
Fetches a page in real time because a user asked the assistant to.
Fetches made when a ChatGPT or Custom GPT user asks it to visit a page.
Fetches made when a Claude user asks a question that needs your page.
Fetches made when a Perplexity user asks a question that needs your page.
Fetches requested by Meta AI users, for example when a user shares a link.
Training crawlers
Collects content that may be used to train AI models.
OpenAI's training crawler.
Anthropic's training crawler.
Meta's crawler for use cases such as training foundation AI models or improving products by indexing content directly.
Amazon's crawler, used to improve Amazon products and services.
Training control tokens
A robots.txt token that controls AI use of content another crawler fetched. It never fetches pages itself.
Open datasets
Builds an open dataset that third parties use, including for AI training.
Which of these can reach your site?
We test robots.txt and send live requests as the AI search crawlers above.
Sources: each operator's official crawler documentation, checked 2026-10-31. Entries marked “partial” rely on documentation that doesn't state every detail.