← AI agent directory

Common Crawl · Model training

CCBot

Common Crawl publishes an open web archive that many model builders train on.

Operator
Common Crawl
Purpose
Model training
Verification
None
Follows robots.txt
Operator says it follows robots.txt

Operator documentation: https://commoncrawl.org/ccbot

01Identify and verify

How it shows up in your logs

User-agent contains

  • CCBot

Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.

Is it really CCBot?

The operator publishes no ranges or keys, so any request using this name is unverified.

CCBot runs from AWS with no published ranges.

02Publishers

How major publishers treat it

60%of 319 publishers on the leaderboard block CCBot outright (192). 83 more close some sections. 189 name it explicitly in their robots.txt.

From the scan of 21 November 2026 · see every publisher

03robots.txt

A rule you can paste

The group names CCBot exactly. A group for a different token from the same company does not cover it.

# CCBot (Common Crawl)
User-agent: CCBot
Disallow: /
CCBot (Common Crawl): what it is and how to block it · Plumb