Mashable
Blocks all AI bots
- Model trainingclosed
- 5/100 open
- AI searchclosed
- 0/100 open
- User-requested fetchclosed
- 0/100 open
- robots.txt
- HTTP 200 · 6.3 KB
- Rule groups / sitemaps
- 3 / 3
- AI bots named explicitly
- 19 of 20
- llms.txt
- no (HTTP 404)
01Bot by bot
What the file says to each crawler
| Bot | Operator | Verdict | Rule used | Why |
|---|---|---|---|---|
| Model training | ||||
GPTBot | OpenAI | Blocked | names this bot | Disallow: / in GPTBot group, 8 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
ClaudeBot | Anthropic | Blocked | names this bot | Disallow: / in ClaudeBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
anthropic-ai | Anthropic | Blocked | names this bot | Disallow: / in anthropic-ai group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Google-Extended | Partial | * group | 1 content path(s) disallowed for all bots in * group, e.g. /archive/ | |
Applebot-Extended | Apple | Blocked | names this bot | Disallow: / in Applebot-Extended group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
CCBot | Common Crawl | Blocked | names this bot | Disallow: / in CCBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Bytespider | ByteDance | Blocked | names this bot | Disallow: / in Bytespider group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Amazonbot | Amazon | Blocked | names this bot | Disallow: / in Amazonbot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
meta-externalagent | Meta | Blocked | names this bot | Disallow: / in meta-externalagent group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
cohere-ai | Cohere | Blocked | names this bot | Disallow: / in cohere-ai group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Diffbot | Diffbot | Blocked | names this bot | Disallow: / in Diffbot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
| AI search | ||||
OAI-SearchBot | OpenAI | Blocked | names this bot | Disallow: / in OAI-SearchBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Claude-SearchBot | Anthropic | Blocked | names this bot | Disallow: / in Claude-SearchBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
PerplexityBot | Perplexity | Blocked | names this bot | Disallow: / in PerplexityBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
DuckAssistBot | DuckDuckGo | Blocked | names this bot | Disallow: / in DuckAssistBot group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
| User-requested fetch | ||||
ChatGPT-User | OpenAI | Blocked | names this bot | Disallow: / in ChatGPT-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Claude-User | Anthropic | Blocked | names this bot | Disallow: / in Claude-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Perplexity-User | Perplexity | Blocked | names this bot | Disallow: / in Perplexity-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
MistralAI-User | Mistral | Blocked | names this bot | Disallow: / in MistralAI-User group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
Meta-ExternalFetcher | Meta | Blocked | names this bot | Disallow: / in Meta-ExternalFetcher group, 5 Allow exception(s) e.g. /ad/tech/how-to-set-up-the-smartest-galaxy-ai-features-on-yo |
02History
Every scan of this file
| Scanned | Stance | Training | AI search | User fetch | Blocked | llms.txt |
|---|---|---|---|---|---|---|
| 21 November 2026 | Blocks all AI bots | 5 | 0 | 0 | 19 | No |