← Robots.txt leaderboard

Reuters

reuters.com · GB · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
0/100 open
AI searchclosed
12/100 open
User-requested fetchclosed
10/100 open
robots.txt
HTTP 200 · 5.1 KB
Rule groups / sitemaps
4 / 14
AI bots named explicitly
3 of 20
llms.txt
no (HTTP 401)

https://reuters.com/robots.txt → https://www.reuters.com/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
ClaudeBotAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
anthropic-aiAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
Google-ExtendedGoogleBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
Applebot-ExtendedAppleBlockednames this botDisallow: / in Applebot-Extended group
CCBotCommon CrawlBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
BytespiderByteDanceBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
AmazonbotAmazonBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
meta-externalagentMetaBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
cohere-aiCohereBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
DiffbotDiffbotBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
AI search
OAI-SearchBotOpenAIPartialnames this bot26 path(s) disallowed in OAI-SearchBot group, e.g. /finance/stocks/option
Claude-SearchBotAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
PerplexityBotPerplexityBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
DuckAssistBotDuckDuckGoBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
User-requested fetch
ChatGPT-UserOpenAIPartialnames this bot26 path(s) disallowed in ChatGPT-User group, e.g. /finance/stocks/option
Claude-UserAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
Perplexity-UserPerplexityBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
MistralAI-UserMistralBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/
Meta-ExternalFetcherMetaBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /plus/

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots0121018No
Reuters (reuters.com): Blocks all AI bots · Plumb robots.txt leaderboard