← AI agent directory

Anthropic · AI search indexing

Claude-SearchBot

Operator
Anthropic
Purpose
AI search indexing
Verification
Web Bot Auth signature
Follows robots.txt
Operator says it follows robots.txt

Operator documentation: https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

01Identify and verify

How it shows up in your logs

User-agent contains

  • Claude-SearchBot

Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.

Is it really Claude-SearchBot?

Plumb checks the Web Bot Auth signature against the operator's published key directory.

Signing keys: www.anthropic.com

Anthropic publishes no IP ranges; ClaudeBot / Claude-User / Claude-SearchBot verify only by signature.

02Publishers

How major publishers treat it

29%of 319 publishers on the leaderboard block Claude-SearchBot outright (93). 152 more close some sections. 98 name it explicitly in their robots.txt.

From the scan of 21 November 2026 · see every publisher

03robots.txt

A rule you can paste

The group names Claude-SearchBot exactly. A group for a different token from the same company does not cover it.

# Claude-SearchBot (Anthropic)
User-agent: Claude-SearchBot
Disallow: /

04Same operator

Anthropic's other agents

Claude-SearchBot (Anthropic): what it is and how to block it · Plumb