AI agent directoryMEASURED

Who is crawling, what for, and how to tell them no.

Every bot Plumb recognises in a publisher's logs, with the robots.txt token that controls it and how its identity can be checked. Built from the same catalogue Plumb classifies traffic with, so this page and your dashboard never disagree.

77 agents · 43 AI crawlers and fetchers · block rates from the robots.txt leaderboard scan of 21 November 2026

43
AI agents in the catalogue
17 of 43
can be checked against published IP ranges or signatures
4
whose operators say user-requested fetches may skip robots.txt
20
graded against 319 publisher robots.txt files

01Directory

Look up an agent

The block rate is the share of leaderboard publishers whose robots.txt blocks the agent's token outright; a stripe of ochre marks partial blocks. Purposes marked * come from Plumb's classifier because the operator does not document one.

77 of 77 agentsSort: · ·

AgentPurposeHow Plumb verifies itFollows robots.txtPublishers blocking it
CCBotCommon CrawlModel trainingNoneYes, per operator60%
ClaudeBotAnthropicModel trainingWeb Bot Auth signatureYes, per operator57%
BytespiderByteDanceModel trainingNoneNot documented56%
GPTBotOpenAIModel trainingPublished IP rangesYes, per operator53%
anthropic-aiAnthropicModel trainingWeb Bot Auth signatureNot documented53%
Applebot-ExtendedAppleModel trainingPublished IP rangesYes, per operator51%
Meta-ExternalAgentMetaModel trainingNoneNot documented50%
DiffbotDiffbotModel trainingNoneNot documented50%
CohereBotCohereModel trainingNoneNot documented49%
PerplexityBotPerplexityAI search indexingPublished IP rangesYes, per operator48%
Google-ExtendedGoogleModel trainingPublished IP rangesYes, per operator47%
AmazonbotAmazonModel trainingReverse DNS onlyYes, per operator47%
ChatGPT-UserOpenAIUser-triggered fetchPublished IP rangesMay ignore it, per operator38%
Meta-ExternalFetcherMetaUser-triggered fetchNoneMay ignore it, per operator33%
Perplexity-UserPerplexityUser-triggered fetchPublished IP rangesMay ignore it, per operator30%
OAI-SearchBotOpenAIAI search indexingPublished IP rangesYes, per operator30%
Claude-SearchBotAnthropicAI search indexingWeb Bot Auth signatureYes, per operator29%
Claude-UserAnthropicUser-triggered fetchWeb Bot Auth signatureYes, per operator28%
DuckAssistBotDuckDuckGoAI search indexingPublished IP rangesNot documented28%
MistralAI-UserMistral AIUser-triggered fetchNoneNot documented24%
AhrefsBotAhrefsSEO toolNoneYes, per operatornot graded
Ai2BotAi2Model training *NoneNot documented, no tokennot graded
Amzn-SearchBotAmazonAI search indexing *Reverse DNS onlyNot documented, no tokennot graded
ApplebotAppleSearch enginePublished IP rangesYes, per operatornot graded
axiosNo single operatorOther *NoneNo tokennot graded
BaiduspiderBaiduSearch engine *Reverse DNS onlyNot documented, no tokennot graded
BarkrowlerBabbarSEO tool *NoneNot documented, no tokennot graded
bingbotMicrosoftSearch enginePublished IP rangesYes, per operatornot graded
BLEXBotWebMeUpSEO tool *NoneNot documented, no tokennot graded
BravebotBraveAI search indexing *NoneNot documented, no tokennot graded
cohere-training-data-crawlerCohereModel training *NoneNot documented, no tokennot graded
CrawlspaceCrawlspaceModel training *NoneNot documented, no tokennot graded
curlNo single operatorOther *NoneNo tokennot graded
DataForSeoBotDataForSEOSEO tool *NoneNot documented, no tokennot graded
DeepSeekBotDeepSeekModel training *NoneNot documented, no tokennot graded
DotBotMozSEO toolNoneYes, per operatornot graded
DuckDuckBotDuckDuckGoSearch enginePublished IP rangesYes, per operatornot graded
FacebookBotMetaModel training *NoneNot documentednot graded
Go-http-clientNo single operatorOther *NoneNo tokennot graded
Google-AgentGoogleUser-triggered fetchPublished IP rangesMay ignore it, per operatornot graded
Google-CloudVertexBotGoogleOtherPublished IP rangesNot documentednot graded
GoogleAgent-MarinerGoogleUser-triggered fetch *Published IP rangesNot documented, no tokennot graded
GooglebotGoogleSearch enginePublished IP rangesYes, per operatornot graded
Googlebot-ExtendedGoogleAI search indexing *Published IP rangesNot documented, no tokennot graded
GoogleOtherGoogleOtherPublished IP rangesYes, per operatornot graded
GrokBotxAIUser-triggered fetch *NoneNot documented, no tokennot graded
HeadlessChromeNo single operatorOther *NoneNo tokennot graded
iAskBotiAskUser-triggered fetch *NoneNot documented, no tokennot graded
ImagesiftBotImageSiftModel training *NoneNot documented, no tokennot graded
ips-agentNo single operatorOther *NoneNo tokennot graded
jscrawlerNo single operatorOther *NoneNo tokennot graded
libwww-perlNo single operatorOther *NoneNo tokennot graded
MJ12botMajesticSEO toolNoneYes, per operatornot graded
node-fetchNo single operatorOther *NoneNo tokennot graded
okhttpNo single operatorOther *NoneNo tokennot graded
OmgilibotWebz.ioModel training *NoneNot documentednot graded
PanguBotHuaweiModel training *NoneNot documented, no tokennot graded
PetalBotHuaweiAI search indexing *NoneNot documented, no tokennot graded
python-requestsNo single operatorOther *NoneNo tokennot graded
QwantifyQwantSearch engine *NoneNot documented, no tokennot graded
rogerbotMozSEO toolNoneYes, per operatornot graded
ScrapyNo single operatorOther *NoneNo tokennot graded
Screaming FrogScreaming FrogSEO tool *NoneNot documented, no tokennot graded
SemrushBotSemrushSEO toolNoneYes, per operatornot graded
SERankingBacklinksBotSE RankingSEO tool *NoneNot documented, no tokennot graded
serpstatbotSerpstatSEO tool *NoneNot documented, no tokennot graded
SeznamBotSeznamSearch engine *NoneNot documented, no tokennot graded
SiteAuditBotSemrushSEO tool *NoneNot documented, no tokennot graded
TikTokSpiderByteDanceModel training *NoneNot documented, no tokennot graded
TimpibotTimpiModel training *NoneNot documented, no tokennot graded
UptimeRobotNo single operatorOther *NoneNo tokennot graded
Webz.ioWebz.ioModel training *NoneNot documentednot graded
wgetNo single operatorOther *NoneNo tokennot graded
xAI-SearchBotxAIAI search indexing *NoneNot documented, no tokennot graded
Yahoo SlurpYahooSearch engine *NoneNot documentednot graded
YandexBotYandexSearch engineReverse DNS onlyYes, per operatornot graded
YouBotYou.comAI search indexing *NoneNot documented, no tokennot graded

02Sources

Where each column comes from

  • Names and user-agent patterns

    Plumb's own classifier. A request matches an agent when its user-agent contains one of the listed patterns, case-insensitive. Generic HTTP libraries are listed too, because they show up in the same logs.

  • Purpose and robots.txt

    Taken from the operator's own documentation where it exists. Where an operator says nothing, the directory says “not documented” rather than guessing. User-triggered fetchers are the usual exception to robots.txt: several operators say a page a person asked for is fetched regardless.

  • Verification

    A user-agent is free to forge. Plumb checks a request against the operator's published IP ranges or, for operators that sign requests, the Web Bot Auth signature. Agents with neither stay unverified in your reports.

  • Publishers blocking it

    Counted from the latest leaderboard scan, over the publishers whose robots.txt could be read. The leaderboard grades 20tokens; agents outside that set are marked “not graded”.