← Robots.txt leaderboard

Mashable

mashable.com · US · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
5/100 open
AI searchclosed
0/100 open
User-requested fetchclosed
0/100 open
robots.txt
HTTP 200 · 6.3 KB
Rule groups / sitemaps
3 / 3
AI bots named explicitly
19 of 20
llms.txt
no (HTTP 404)

https://mashable.com/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIBlockednames this botDisallow: / in GPTBot group, 8 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
ClaudeBotAnthropicBlockednames this botDisallow: / in ClaudeBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
anthropic-aiAnthropicBlockednames this botDisallow: / in anthropic-ai group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
Google-ExtendedGooglePartial* group1 content path(s) disallowed for all bots in * group, e.g. /archive/
Applebot-ExtendedAppleBlockednames this botDisallow: / in Applebot-Extended group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
CCBotCommon CrawlBlockednames this botDisallow: / in CCBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
BytespiderByteDanceBlockednames this botDisallow: / in Bytespider group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
AmazonbotAmazonBlockednames this botDisallow: / in Amazonbot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
meta-externalagentMetaBlockednames this botDisallow: / in meta-externalagent group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
cohere-aiCohereBlockednames this botDisallow: / in cohere-ai group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
DiffbotDiffbotBlockednames this botDisallow: / in Diffbot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
AI search
OAI-SearchBotOpenAIBlockednames this botDisallow: / in OAI-SearchBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
Claude-SearchBotAnthropicBlockednames this botDisallow: / in Claude-SearchBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
PerplexityBotPerplexityBlockednames this botDisallow: / in PerplexityBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
DuckAssistBotDuckDuckGoBlockednames this botDisallow: / in DuckAssistBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
User-requested fetch
ChatGPT-UserOpenAIBlockednames this botDisallow: / in ChatGPT-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
Claude-UserAnthropicBlockednames this botDisallow: / in Claude-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
Perplexity-UserPerplexityBlockednames this botDisallow: / in Perplexity-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
MistralAI-UserMistralBlockednames this botDisallow: / in MistralAI-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo
Meta-ExternalFetcherMetaBlockednames this botDisallow: / in Meta-ExternalFetcher group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots50019No
Mashable (mashable.com): Blocks all AI bots · Plumb robots.txt leaderboard