← Robots.txt leaderboard

The New Yorker

newyorker.com · US · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
9/100 open
AI searchclosed
12/100 open
User-requested fetchclosed
10/100 open
robots.txt
HTTP 200 · 2.2 KB
Rule groups / sitemaps
3 / 7
AI bots named explicitly
16 of 20
llms.txt
no (HTTP 404)

https://newyorker.com/robots.txt → https://www.newyorker.com/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIPartial* group7 content path(s) disallowed for all bots in * group, e.g. /user-context
ClaudeBotAnthropicBlockednames this botDisallow: / in ClaudeBot group
anthropic-aiAnthropicPartial* group7 content path(s) disallowed for all bots in * group, e.g. /user-context
Google-ExtendedGoogleBlockednames this botDisallow: / in Google-Extended group
Applebot-ExtendedAppleBlockednames this botDisallow: / in Applebot-Extended group
CCBotCommon CrawlBlockednames this botDisallow: / in CCBot group
BytespiderByteDanceBlockednames this botDisallow: / in Bytespider group
AmazonbotAmazonBlockednames this botDisallow: / in Amazonbot group
meta-externalagentMetaBlockednames this botDisallow: / in meta-externalagent group
cohere-aiCohereBlockednames this botDisallow: / in cohere-ai group
DiffbotDiffbotBlockednames this botDisallow: / in Diffbot group
AI search
OAI-SearchBotOpenAIPartial* group7 content path(s) disallowed for all bots in * group, e.g. /user-context
Claude-SearchBotAnthropicBlockednames this botDisallow: / in Claude-SearchBot group
PerplexityBotPerplexityBlockednames this botDisallow: / in PerplexityBot group
DuckAssistBotDuckDuckGoBlockednames this botDisallow: / in DuckAssistBot group
User-requested fetch
ChatGPT-UserOpenAIPartial* group7 content path(s) disallowed for all bots in * group, e.g. /user-context
Claude-UserAnthropicBlockednames this botDisallow: / in Claude-User group
Perplexity-UserPerplexityBlockednames this botDisallow: / in Perplexity-User group
MistralAI-UserMistralBlockednames this botDisallow: / in MistralAI-User group
Meta-ExternalFetcherMetaBlockednames this botDisallow: / in Meta-ExternalFetcher group

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots9121016No
The New Yorker (newyorker.com): Blocks all AI bots · Plumb robots.txt leaderboard