AI bots and crawlers: user agents, IP ranges and robots.txt
Which bots read your site for AI search, for model training or on a user’s request — and how to allow or block each one. Every fact comes from the bot owner’s own documentation, with a link and the date we last checked it.
| Bot | Company | What it does | Follows robots.txt |
|---|---|---|---|
| OAI-SearchBotOpenAI’s crawler for ChatGPT search | OpenAI | AI search | Yes |
| ChatGPT-UserOpenAI’s fetcher for pages ChatGPT users ask about | OpenAI | Fetches on user request | May ignore it |
| GPTBotOpenAI’s crawler for training its AI models | OpenAI | Model training | Yes |
| OAI-AdsBotOpenAI’s checker for ad landing pages on ChatGPT | OpenAI | Ad checks | Not stated |
| Claude-SearchBotAnthropic’s crawler for Claude search results | Anthropic | AI search | Yes |
| Claude-UserAnthropic’s fetcher for pages Claude users ask about | Anthropic | Fetches on user request | Yes |
| ClaudeBotAnthropic’s crawler for training Claude models | Anthropic | Model training | Yes |
| PerplexityBotPerplexity’s crawler for its search results | Perplexity | AI search | Yes |
| Perplexity-UserPerplexity’s fetcher for pages users ask about | Perplexity | Fetches on user request | May ignore it |
| DuckAssistBotDuckDuckGo’s crawler for AI-assisted answers | DuckDuckGo | AI search | Yes |
| Google-ExtendedGoogle’s robots.txt token for Gemini training and grounding | Model training | Token only | |
| Applebot-ExtendedApple’s robots.txt token for AI training | Apple | Model training | Token only |
| CCBotCommon Crawl’s crawler for its open web archive | Common Crawl | Model training | Yes |
| meta-externalagentMeta’s crawler for AI training and product indexing | Meta | Model training | Yes |
| AmazonbotAmazon’s crawler for its products and AI models | Amazon | Model training | Yes |
| GooglebotGoogle Search’s main crawler | Classic search | Yes | |
| BingbotMicrosoft Bing’s standard crawler | Microsoft | Classic search | Yes |
| ApplebotApple’s crawler for Siri, Spotlight and Safari | Apple | Classic search | Yes |
We compare this directory with the official pages and IP lists every week. Last check: 13 Nov 2026.
Bytespider (ByteDance) is not listed: ByteDance has no official page about its crawler, and we only publish what the owner documents.
Check your site
See whether your robots.txt and firewall let GPTBot, ClaudeBot, PerplexityBot in — together with every other AI bot in this directory. Free, no sign-up.
Check my site