← Robots.txt leaderboard

news.com.au

news.com.au · AU · scanned 21 November 2026

Blocks all AI bots

Model trainingclosed
0/100 open
AI searchclosed
0/100 open
User-requested fetchclosed
0/100 open
robots.txt
HTTP 200 · 3.5 KB
Rule groups / sitemaps
4 / 7
AI bots named explicitly
1 of 20
llms.txt
no (HTTP 404)

https://news.com.au/robots.txt → https://www.news.com.au/robots.txt

01Bot by bot

What the file says to each crawler

BotOperatorVerdictRule usedWhy
Model training
GPTBotOpenAIBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
ClaudeBotAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
anthropic-aiAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
Google-ExtendedGoogleBlockednames this botDisallow: / in Google-Extended group
Applebot-ExtendedAppleBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
CCBotCommon CrawlBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
BytespiderByteDanceBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
AmazonbotAmazonBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
meta-externalagentMetaBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
cohere-aiCohereBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
DiffbotDiffbotBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
AI search
OAI-SearchBotOpenAIBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
Claude-SearchBotAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
PerplexityBotPerplexityBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
DuckAssistBotDuckDuckGoBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
User-requested fetch
ChatGPT-UserOpenAIBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
Claude-UserAnthropicBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
Perplexity-UserPerplexityBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
MistralAI-UserMistralBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json
Meta-ExternalFetcherMetaBlocked* groupDisallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json

02History

Every scan of this file

ScannedStanceTrainingAI searchUser fetchBlockedllms.txt
21 November 2026Blocks all AI bots00020No
news.com.au (news.com.au): Blocks all AI bots · Plumb robots.txt leaderboard