Anthropic · Model training
ClaudeBot
- Operator
- Anthropic
- Purpose
- Model training
- Verification
- Web Bot Auth signature
- Follows robots.txt
- Operator says it follows robots.txt
01Identify and verify
How it shows up in your logs
User-agent contains
ClaudeBotClaude-Web
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really ClaudeBot?
Plumb checks the Web Bot Auth signature against the operator's published key directory.
Signing keys: www.anthropic.com
Anthropic publishes no IP ranges; ClaudeBot / Claude-User / Claude-SearchBot verify only by signature.
02Publishers
How major publishers treat it
57%of 319 publishers on the leaderboard block
ClaudeBot outright (181). 94 more close some sections. 176 name it explicitly in their robots.txt.03robots.txt
A rule you can paste
The group names ClaudeBot exactly. A group for a different token from the same company does not cover it.
# ClaudeBot (Anthropic) User-agent: ClaudeBot Disallow: /
04Same operator
Anthropic's other agents
- Claude-SearchBotAI search indexing
- Claude-UserUser-triggered fetch
- anthropic-aiModel training