Anthropic · AI search indexing
Claude-SearchBot
- Operator
- Anthropic
- Purpose
- AI search indexing
- Verification
- Web Bot Auth signature
- Follows robots.txt
- Operator says it follows robots.txt
01Identify and verify
How it shows up in your logs
User-agent contains
Claude-SearchBot
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really Claude-SearchBot?
Plumb checks the Web Bot Auth signature against the operator's published key directory.
Signing keys: www.anthropic.com
Anthropic publishes no IP ranges; ClaudeBot / Claude-User / Claude-SearchBot verify only by signature.
02Publishers
How major publishers treat it
29%of 319 publishers on the leaderboard block
Claude-SearchBot outright (93). 152 more close some sections. 98 name it explicitly in their robots.txt.03robots.txt
A rule you can paste
The group names Claude-SearchBot exactly. A group for a different token from the same company does not cover it.
# Claude-SearchBot (Anthropic) User-agent: Claude-SearchBot Disallow: /
04Same operator
Anthropic's other agents
- Claude-UserUser-triggered fetch
- ClaudeBotModel training
- anthropic-aiModel training