← Robots.txt leaderboard

Die Zeit

zeit.de · DE · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
0/100 open
AI searchclosed
0/100 open
User-requested fetchclosed
0/100 open
robots.txt
HTTP 200 · 10.0 KB
Rule groups / sitemaps
5 / 1
AI bots named explicitly
1 of 20
llms.txt
no (HTTP 403)

https://zeit.de/robots.txt → https://www.zeit.de/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIBlocked* groupDisallow: / in * group
ClaudeBotAnthropicBlocked* groupDisallow: / in * group
anthropic-aiAnthropicBlocked* groupDisallow: / in * group
Google-ExtendedGoogleBlockednames this botDisallow: / in Google-Extended group, 1 Allow exception(s) e.g. /*-gxe$
Applebot-ExtendedAppleBlocked* groupDisallow: / in * group
CCBotCommon CrawlBlocked* groupDisallow: / in * group
BytespiderByteDanceBlocked* groupDisallow: / in * group
AmazonbotAmazonBlocked* groupDisallow: / in * group
meta-externalagentMetaBlocked* groupDisallow: / in * group
cohere-aiCohereBlocked* groupDisallow: / in * group
DiffbotDiffbotBlocked* groupDisallow: / in * group
AI search
OAI-SearchBotOpenAIBlocked* groupDisallow: / in * group
Claude-SearchBotAnthropicBlocked* groupDisallow: / in * group
PerplexityBotPerplexityBlocked* groupDisallow: / in * group
DuckAssistBotDuckDuckGoBlocked* groupDisallow: / in * group
User-requested fetch
ChatGPT-UserOpenAIBlocked* groupDisallow: / in * group
Claude-UserAnthropicBlocked* groupDisallow: / in * group
Perplexity-UserPerplexityBlocked* groupDisallow: / in * group
MistralAI-UserMistralBlocked* groupDisallow: / in * group
Meta-ExternalFetcherMetaBlocked* groupDisallow: / in * group

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots00020No
Die Zeit (zeit.de): Blocks all AI bots · Plumb robots.txt leaderboard