CCBot (Common Crawl)
Model training · Current version 2.0
What it is
CCBot is the crawler of Common Crawl, a non-profit foundation that maintains an open repository of web crawl data anyone can access and analyze. Blocking it keeps your site out of Common Crawl’s future crawls. Common Crawl also runs an opt-out registry, described on its blog.
Official documentation: commoncrawl.org · checked 13 Nov 2026
CCBot user agent
CCBot/2.0 (https://commoncrawl.org/faq/)Official documentation: commoncrawl.org/ccbot · checked 13 Nov 2026
Block or allow CCBot in robots.txt
Put these lines at the top of the robots.txt file in your site’s root (example.com/robots.txt). Each subdomain needs its own file.
User-agent: CCBot
Disallow: /
User-agent: CCBot
Allow: /
# repeat here the Disallow lines of your User-agent: * group
If CCBot is not named in robots.txt, it already follows your User-agent: * rules, so you don’t need anything. A group of its own replaces the * group for this bot entirely: if you add one, repeat your general Disallow lines in it.
CCBot IP ranges
index.commoncrawl.org/ccbot.json
Official list: 5 ranges · published by the owner 11 Aug 2026 · we checked it 13 Nov 2026
Check an IP from your logs
The owner also documents a DNS check: the reverse lookup of the IP must give a host name ending in crawl.commoncrawl.org, and the forward lookup of that name must return the same IP. Official documentation
Show all 5
- 3.41.188.32/29
- 18.97.9.168/29
- 18.97.14.80/29
- 18.97.14.88/30
- 2600:1f28:365:8000::/56
Does it respect robots.txt and Crawl-delay?
robots.txt: yes, CCBot follows it according to its owner.
“CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests.” — Common Crawl documentation
Crawl-delay: supported for CCBot.
“We obey the Crawl-delay parameter for robots.txt.” — Common Crawl documentation
Should you block CCBot?
It is a legitimate choice. Blocking CCBot opts your content out of this kind of use, and search and AI answers that rely on other bots are not affected. If you want your content used by AI models, leave it open.
Check your site
See whether your robots.txt and firewall let CCBot in — together with every other AI bot in this directory. Free, no sign-up.
Questions
What is the CCBot user agent string?+
Common Crawl documents it as: CCBot/2.0 (https://commoncrawl.org/faq/)
How do I block CCBot in robots.txt?+
Add a group for it to robots.txt in your site’s root: “User-agent: CCBot” followed by “Disallow: /”.
What IP addresses does CCBot use?+
Common Crawl publishes 5 ranges at https://index.commoncrawl.org/ccbot.json. Requests that call themselves CCBot from other addresses are most likely fake.
Does CCBot respect robots.txt?+
robots.txt: yes, CCBot follows it according to its owner.