The Australian
Blocks all AI bots
- Model trainingclosed
- 9/100 open
- AI searchclosed
- 25/100 open
- User-requested fetchclosed
- 20/100 open
- robots.txt
- HTTP 200 · 3.2 KB
- Rule groups / sitemaps
- 4 / 2
- AI bots named explicitly
- 4 of 20
- llms.txt
- no (HTTP 404)
01Bot by bot
What the file says to each crawler
| Bot | Operator | Verdict | Rule used | Why |
|---|---|---|---|---|
| Model training | ||||
GPTBot | OpenAI | Allowed | names this bot | GPTBot group allows everything |
ClaudeBot | Anthropic | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
anthropic-ai | Anthropic | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
Google-Extended | Blocked | names this bot | Disallow: / in Google-Extended group | |
Applebot-Extended | Apple | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
CCBot | Common Crawl | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
Bytespider | ByteDance | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
Amazonbot | Amazon | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
meta-externalagent | Meta | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
cohere-ai | Cohere | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
Diffbot | Diffbot | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
| AI search | ||||
OAI-SearchBot | OpenAI | Allowed | names this bot | OAI-SearchBot group allows everything |
Claude-SearchBot | Anthropic | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
PerplexityBot | Perplexity | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
DuckAssistBot | DuckDuckGo | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
| User-requested fetch | ||||
ChatGPT-User | OpenAI | Allowed | names this bot | ChatGPT-User group allows everything |
Claude-User | Anthropic | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
Perplexity-User | Perplexity | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
MistralAI-User | Mistral | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
Meta-ExternalFetcher | Meta | Blocked | * group | Disallow: / in * group, 1 Allow exception(s) e.g. /.well-known/adagents.json |
02History
Every scan of this file
| Scanned | Stance | Training | AI search | User fetch | Blocked | llms.txt |
|---|---|---|---|---|---|---|
| 21 November 2026 | Blocks all AI bots | 9 | 25 | 20 | 17 | No |