← Robots.txt leaderboard

The Sun

thesun.co.uk · GB · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
5/100 open
AI searchclosed
12/100 open
User-requested fetchclosed
10/100 open
robots.txt
HTTP 200 · 4.9 KB
Rule groups / sitemaps
3 / 12
AI bots named explicitly
9 of 20
llms.txt
no (HTTP 404)

https://thesun.co.uk/robots.txt → https://www.thesun.co.uk/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIPartialnames this bot101 path(s) disallowed in GPTBot group, e.g. /search/
ClaudeBotAnthropicBlocked* groupDisallow: / in * group
anthropic-aiAnthropicBlockednames this botDisallow: / in anthropic-ai group, 1 Allow exception(s) e.g. /sell-your-story-sun/$
Google-ExtendedGoogleBlockednames this botDisallow: / in Google-Extended group, 1 Allow exception(s) e.g. /sell-your-story-sun/$
Applebot-ExtendedAppleBlocked* groupDisallow: / in * group
CCBotCommon CrawlBlockednames this botDisallow: / in CCBot group, 1 Allow exception(s) e.g. /sell-your-story-sun/$
BytespiderByteDanceBlocked* groupDisallow: / in * group
AmazonbotAmazonBlocked* groupDisallow: / in * group
meta-externalagentMetaBlocked* groupDisallow: / in * group
cohere-aiCohereBlocked* groupDisallow: / in * group
DiffbotDiffbotBlocked* groupDisallow: / in * group
AI search
OAI-SearchBotOpenAIPartialnames this bot101 path(s) disallowed in OAI-SearchBot group, e.g. /search/
Claude-SearchBotAnthropicBlockednames this botDisallow: / in Claude-SearchBot group, 1 Allow exception(s) e.g. /sell-your-story-sun/$
PerplexityBotPerplexityBlockednames this botDisallow: / in PerplexityBot group, 1 Allow exception(s) e.g. /sell-your-story-sun/$
DuckAssistBotDuckDuckGoBlocked* groupDisallow: / in * group
User-requested fetch
ChatGPT-UserOpenAIPartialnames this bot101 path(s) disallowed in ChatGPT-User group, e.g. /search/
Claude-UserAnthropicBlockednames this botDisallow: / in Claude-User group, 1 Allow exception(s) e.g. /sell-your-story-sun/$
Perplexity-UserPerplexityBlocked* groupDisallow: / in * group
MistralAI-UserMistralBlocked* groupDisallow: / in * group
Meta-ExternalFetcherMetaBlocked* groupDisallow: / in * group

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots5121017No
The Sun (thesun.co.uk): Blocks all AI bots · Plumb robots.txt leaderboard