CCBot

Common Crawl's crawler. It builds an open web dataset that many AI models have been trained on.

What happens if you block it

Future pages won't enter the Common Crawl dataset. Snapshots already published are not withdrawn.

Our take

Block it if you don't want your pages in open training datasets. It has no effect on any search engine.

robots.txt

To block it:

User-agent: CCBot
Disallow: /

To allow it explicitly (useful when a wider rule blocks other bots):

User-agent: CCBot
Allow: /

The operator's own words

To prevent Common Crawl from crawling your website, include the following in your robots.txt: User-agent: CCBot Disallow: /Common Crawl documentation, checked 2026-10-31