Common Crawl · Model training
CCBot
Common Crawl publishes an open web archive that many model builders train on.
- Operator
- Common Crawl
- Purpose
- Model training
- Verification
- None
- Follows robots.txt
- Operator says it follows robots.txt
01Identify and verify
How it shows up in your logs
User-agent contains
CCBot
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really CCBot?
The operator publishes no ranges or keys, so any request using this name is unverified.
CCBot runs from AWS with no published ranges.
02Publishers
How major publishers treat it
60%of 319 publishers on the leaderboard block
CCBot outright (192). 83 more close some sections. 189 name it explicitly in their robots.txt.03robots.txt
A rule you can paste
The group names CCBot exactly. A group for a different token from the same company does not cover it.
# CCBot (Common Crawl) User-agent: CCBot Disallow: /