← AI agent directory

Anthropic · Model training

ClaudeBot

Operator
Anthropic
Purpose
Model training
Verification
Web Bot Auth signature
Follows robots.txt
Operator says it follows robots.txt

Operator documentation: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

01Identify and verify

How it shows up in your logs

User-agent contains

  • ClaudeBot
  • Claude-Web

Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.

Is it really ClaudeBot?

Plumb checks the Web Bot Auth signature against the operator's published key directory.

Signing keys: www.anthropic.com

Anthropic publishes no IP ranges; ClaudeBot / Claude-User / Claude-SearchBot verify only by signature.

02Publishers

How major publishers treat it

57%of 319 publishers on the leaderboard block ClaudeBot outright (181). 94 more close some sections. 176 name it explicitly in their robots.txt.

From the scan of 21 November 2026 · see every publisher

03robots.txt

A rule you can paste

The group names ClaudeBot exactly. A group for a different token from the same company does not cover it.

# ClaudeBot (Anthropic)
User-agent: ClaudeBot
Disallow: /

04Same operator

Anthropic's other agents

ClaudeBot (Anthropic): what it is and how to block it · Plumb