AI agent directoryMEASURED
Who is crawling, what for, and how to tell them no.
Every bot Plumb recognises in a publisher's logs, with the robots.txt token that controls it and how its identity can be checked. Built from the same catalogue Plumb classifies traffic with, so this page and your dashboard never disagree.
- 43
- AI agents in the catalogue
- 17 of 43
- can be checked against published IP ranges or signatures
- 4
- whose operators say user-requested fetches may skip robots.txt
- 20
- graded against 319 publisher robots.txt files
01Directory
Look up an agent
The block rate is the share of leaderboard publishers whose robots.txt blocks the agent's token outright; a stripe of ochre marks partial blocks. Purposes marked * come from Plumb's classifier because the operator does not document one.
77 of 77 agentsSort: · ·
| Agent | Purpose | How Plumb verifies it | Follows robots.txt | Publishers blocking it |
|---|---|---|---|---|
| CCBotCommon Crawl | Model training | None | Yes, per operator | 60% |
| ClaudeBotAnthropic | Model training | Web Bot Auth signature | Yes, per operator | 57% |
| BytespiderByteDance | Model training | None | Not documented | 56% |
| GPTBotOpenAI | Model training | Published IP ranges | Yes, per operator | 53% |
| anthropic-aiAnthropic | Model training | Web Bot Auth signature | Not documented | 53% |
| Applebot-ExtendedApple | Model training | Published IP ranges | Yes, per operator | 51% |
| Meta-ExternalAgentMeta | Model training | None | Not documented | 50% |
| DiffbotDiffbot | Model training | None | Not documented | 50% |
| CohereBotCohere | Model training | None | Not documented | 49% |
| PerplexityBotPerplexity | AI search indexing | Published IP ranges | Yes, per operator | 48% |
| Google-ExtendedGoogle | Model training | Published IP ranges | Yes, per operator | 47% |
| AmazonbotAmazon | Model training | Reverse DNS only | Yes, per operator | 47% |
| ChatGPT-UserOpenAI | User-triggered fetch | Published IP ranges | May ignore it, per operator | 38% |
| Meta-ExternalFetcherMeta | User-triggered fetch | None | May ignore it, per operator | 33% |
| Perplexity-UserPerplexity | User-triggered fetch | Published IP ranges | May ignore it, per operator | 30% |
| OAI-SearchBotOpenAI | AI search indexing | Published IP ranges | Yes, per operator | 30% |
| Claude-SearchBotAnthropic | AI search indexing | Web Bot Auth signature | Yes, per operator | 29% |
| Claude-UserAnthropic | User-triggered fetch | Web Bot Auth signature | Yes, per operator | 28% |
| DuckAssistBotDuckDuckGo | AI search indexing | Published IP ranges | Not documented | 28% |
| MistralAI-UserMistral AI | User-triggered fetch | None | Not documented | 24% |
| AhrefsBotAhrefs | SEO tool | None | Yes, per operator | not graded |
| Ai2BotAi2 | Model training * | None | Not documented, no token | not graded |
| Amzn-SearchBotAmazon | AI search indexing * | Reverse DNS only | Not documented, no token | not graded |
| ApplebotApple | Search engine | Published IP ranges | Yes, per operator | not graded |
| axiosNo single operator | Other * | None | No token | not graded |
| BaiduspiderBaidu | Search engine * | Reverse DNS only | Not documented, no token | not graded |
| BarkrowlerBabbar | SEO tool * | None | Not documented, no token | not graded |
| bingbotMicrosoft | Search engine | Published IP ranges | Yes, per operator | not graded |
| BLEXBotWebMeUp | SEO tool * | None | Not documented, no token | not graded |
| BravebotBrave | AI search indexing * | None | Not documented, no token | not graded |
| cohere-training-data-crawlerCohere | Model training * | None | Not documented, no token | not graded |
| CrawlspaceCrawlspace | Model training * | None | Not documented, no token | not graded |
| curlNo single operator | Other * | None | No token | not graded |
| DataForSeoBotDataForSEO | SEO tool * | None | Not documented, no token | not graded |
| DeepSeekBotDeepSeek | Model training * | None | Not documented, no token | not graded |
| DotBotMoz | SEO tool | None | Yes, per operator | not graded |
| DuckDuckBotDuckDuckGo | Search engine | Published IP ranges | Yes, per operator | not graded |
| FacebookBotMeta | Model training * | None | Not documented | not graded |
| Go-http-clientNo single operator | Other * | None | No token | not graded |
| Google-AgentGoogle | User-triggered fetch | Published IP ranges | May ignore it, per operator | not graded |
| Google-CloudVertexBotGoogle | Other | Published IP ranges | Not documented | not graded |
| GoogleAgent-MarinerGoogle | User-triggered fetch * | Published IP ranges | Not documented, no token | not graded |
| GooglebotGoogle | Search engine | Published IP ranges | Yes, per operator | not graded |
| Googlebot-ExtendedGoogle | AI search indexing * | Published IP ranges | Not documented, no token | not graded |
| GoogleOtherGoogle | Other | Published IP ranges | Yes, per operator | not graded |
| GrokBotxAI | User-triggered fetch * | None | Not documented, no token | not graded |
| HeadlessChromeNo single operator | Other * | None | No token | not graded |
| iAskBotiAsk | User-triggered fetch * | None | Not documented, no token | not graded |
| ImagesiftBotImageSift | Model training * | None | Not documented, no token | not graded |
| ips-agentNo single operator | Other * | None | No token | not graded |
| jscrawlerNo single operator | Other * | None | No token | not graded |
| libwww-perlNo single operator | Other * | None | No token | not graded |
| MJ12botMajestic | SEO tool | None | Yes, per operator | not graded |
| node-fetchNo single operator | Other * | None | No token | not graded |
| okhttpNo single operator | Other * | None | No token | not graded |
| OmgilibotWebz.io | Model training * | None | Not documented | not graded |
| PanguBotHuawei | Model training * | None | Not documented, no token | not graded |
| PetalBotHuawei | AI search indexing * | None | Not documented, no token | not graded |
| python-requestsNo single operator | Other * | None | No token | not graded |
| QwantifyQwant | Search engine * | None | Not documented, no token | not graded |
| rogerbotMoz | SEO tool | None | Yes, per operator | not graded |
| ScrapyNo single operator | Other * | None | No token | not graded |
| Screaming FrogScreaming Frog | SEO tool * | None | Not documented, no token | not graded |
| SemrushBotSemrush | SEO tool | None | Yes, per operator | not graded |
| SERankingBacklinksBotSE Ranking | SEO tool * | None | Not documented, no token | not graded |
| serpstatbotSerpstat | SEO tool * | None | Not documented, no token | not graded |
| SeznamBotSeznam | Search engine * | None | Not documented, no token | not graded |
| SiteAuditBotSemrush | SEO tool * | None | Not documented, no token | not graded |
| TikTokSpiderByteDance | Model training * | None | Not documented, no token | not graded |
| TimpibotTimpi | Model training * | None | Not documented, no token | not graded |
| UptimeRobotNo single operator | Other * | None | No token | not graded |
| Webz.ioWebz.io | Model training * | None | Not documented | not graded |
| wgetNo single operator | Other * | None | No token | not graded |
| xAI-SearchBotxAI | AI search indexing * | None | Not documented, no token | not graded |
| Yahoo SlurpYahoo | Search engine * | None | Not documented | not graded |
| YandexBotYandex | Search engine | Reverse DNS only | Yes, per operator | not graded |
| YouBotYou.com | AI search indexing * | None | Not documented, no token | not graded |
02Sources
Where each column comes from
Names and user-agent patterns
Plumb's own classifier. A request matches an agent when its user-agent contains one of the listed patterns, case-insensitive. Generic HTTP libraries are listed too, because they show up in the same logs.
Purpose and robots.txt
Taken from the operator's own documentation where it exists. Where an operator says nothing, the directory says “not documented” rather than guessing. User-triggered fetchers are the usual exception to robots.txt: several operators say a page a person asked for is fetched regardless.
Verification
A user-agent is free to forge. Plumb checks a request against the operator's published IP ranges or, for operators that sign requests, the Web Bot Auth signature. Agents with neither stay unverified in your reports.
Publishers blocking it
Counted from the latest leaderboard scan, over the publishers whose robots.txt could be read. The leaderboard grades 20tokens; agents outside that set are marked “not graded”.