← Robots.txt leaderboard

The Times

thetimes.co.uk · GB · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
5/100 open
AI searchclosed
12/100 open
User-requested fetchclosed
10/100 open
robots.txt
HTTP 200 · 1.7 KB
Rule groups / sitemaps
2 / 0
AI bots named explicitly
3 of 20
llms.txt
no (HTTP 301)

https://thetimes.co.uk/robots.txt → https://www.thetimes.co.uk/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIPartialnames this bot8 path(s) disallowed in GPTBot group, e.g. /api/
ClaudeBotAnthropicBlocked* groupDisallow: / in * group
anthropic-aiAnthropicBlocked* groupDisallow: / in * group
Google-ExtendedGoogleBlocked* groupDisallow: / in * group
Applebot-ExtendedAppleBlocked* groupDisallow: / in * group
CCBotCommon CrawlBlocked* groupDisallow: / in * group
BytespiderByteDanceBlocked* groupDisallow: / in * group
AmazonbotAmazonBlocked* groupDisallow: / in * group
meta-externalagentMetaBlocked* groupDisallow: / in * group
cohere-aiCohereBlocked* groupDisallow: / in * group
DiffbotDiffbotBlocked* groupDisallow: / in * group
AI search
OAI-SearchBotOpenAIPartialnames this bot8 path(s) disallowed in OAI-SearchBot group, e.g. /api/
Claude-SearchBotAnthropicBlocked* groupDisallow: / in * group
PerplexityBotPerplexityBlocked* groupDisallow: / in * group
DuckAssistBotDuckDuckGoBlocked* groupDisallow: / in * group
User-requested fetch
ChatGPT-UserOpenAIPartialnames this bot8 path(s) disallowed in ChatGPT-User group, e.g. /api/
Claude-UserAnthropicBlocked* groupDisallow: / in * group
Perplexity-UserPerplexityBlocked* groupDisallow: / in * group
MistralAI-UserMistralBlocked* groupDisallow: / in * group
Meta-ExternalFetcherMetaBlocked* groupDisallow: / in * group

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots5121017No
The Times (thetimes.co.uk): Blocks all AI bots · Plumb robots.txt leaderboard