Robots.txt leaderboardMEASURED
Which publishers let AI bots in, read from their own robots.txt.
Plumb fetched the live robots.txt of 344 news and media publishers in 9 regions and graded 20 AI crawler tokens against each file. This is what each publisher has told the bots, not what the bots did.
- 53%
- block OpenAI's GPTBot outright
- 70%
- block at least one of the 11 training crawlers
- 4%
- block training but leave AI search open
- 8%
- publish an llms.txt file
01By bot
The average training bot is blocked by 52%. The average AI search bot, by 34%.
Share of publishers blocking each token, grouped by what its operator says the bot is for. A partial verdict means the publisher named the bot and closed some sections, not the whole site.
Model trainingavg openness 31/100
AI searchavg openness 44/100
User-requested fetchavg openness 45/100
02Stance
One grade hides the decision. The stance shows it.
Each publisher gets a label from three separate readings: training, AI search and user-requested fetches.
- Blocks all AI bots7020%
- Blocks training and AI search154%
- Blocks training, allows AI search144%
- Blocks some AI bots18955%
- Open to AI bots268%
- No robots.txt51%
- Could not fetch257%
| Region | Publishers | Block GPTBot | Training closed | AI search closed | User fetch closed | Could not fetch |
|---|---|---|---|---|---|---|
| United States | 97 | 56% | 59% | 35% | 30% | 9 |
| Canada | 12 | 73% | 64% | 36% | 36% | 1 |
| UK & Ireland | 33 | 71% | 55% | 42% | 26% | 2 |
| Europe | 57 | 77% | 70% | 39% | 34% | 1 |
| India | 23 | 12% | 0% | 0% | 0% | 6 |
| Asia-Pacific | 45 | 57% | 50% | 30% | 20% | 1 |
| Latin America | 29 | 26% | 22% | 19% | 15% | 2 |
| Africa | 26 | 40% | 28% | 4% | 12% | 1 |
| Middle East | 22 | 15% | 10% | 5% | 5% | 2 |
| Publisher type | Publishers | Block GPTBot | Training closed | AI search closed | User fetch closed | Could not fetch |
|---|---|---|---|---|---|---|
| National news | 136 | 48% | 46% | 31% | 28% | 4 |
| Regional / local | 43 | 74% | 55% | 42% | 32% | 5 |
| Magazine | 28 | 39% | 50% | 36% | 25% | 0 |
| Wire / agency | 16 | 57% | 43% | 14% | 14% | 2 |
| Broadcaster | 48 | 70% | 53% | 25% | 13% | 8 |
| B2B / trade | 17 | 47% | 47% | 0% | 0% | 2 |
| Digital-native | 56 | 44% | 42% | 21% | 21% | 4 |
03Changes
Who changed their stance since the last scan
This is the first scan on record. Changes show up here after the next weekly scan.
04Every publisher
Look up a publisher
Openness runs from 0 (every bot in that group blocked) to 100 (none blocked). The strip shows each of the 20 bots in order: training, then AI search, then user fetch.
| Per bot: training · search · user | llms.txt | ||||||
|---|---|---|---|---|---|---|---|
| ABCabc.es · Europe · National news | Blocks all AI bots | 14 | 0 | 0 | 17/20 | No | |
| Ars Technicaarstechnica.com · United States · Digital-native | Blocks all AI bots | 5 | 12 | 10 | 17/20 | No | |
| Asahi Shimbunasahi.com · Asia-Pacific · National news | Blocks all AI bots | 0 | 12 | 10 | 18/20 | No | |
| Australian Financial Reviewafr.com · Asia-Pacific · National news | Blocks all AI bots | 0 | 12 | 10 | 18/20 | No | |
| Chicago Tribunechicagotribune.com · United States · Regional / local | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| Cleveland.comcleveland.com · United States · Regional / local | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| CNNcnn.com · United States · Broadcaster | Blocks all AI bots | 9 | 0 | 10 | 17/20 | No | |
| Corriere della Seracorriere.it · Europe · National news | Blocks all AI bots | 9 | 0 | 10 | 18/20 | No | |
| Dagens Nyheterdn.se · Europe · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Daily Maildailymail.com · UK & Ireland · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | Yes | |
| De Standaardstandaard.be · Europe · National news | Blocks all AI bots | 9 | 25 | 0 | 18/20 | No | |
| De Telegraaftelegraaf.nl · Europe · National news | Blocks all AI bots | 9 | 25 | 0 | 18/20 | No | |
| de Volkskrantvolkskrant.nl · Europe · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Detroit Free Pressfreep.com · United States · Regional / local | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Die Zeitzeit.de · Europe · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| DRdr.dk · Europe · Broadcaster | Blocks all AI bots | 5 | 0 | 10 | 18/20 | No | |
| El Mundoelmundo.es · Europe · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| El Universaleluniversal.com.mx · Latin America · National news | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| Franceinfofranceinfo.fr · Europe · Broadcaster | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Haaretzhaaretz.com · Middle East · National news | Blocks all AI bots | 0 | 12 | 10 | 18/20 | No | |
| Helsingin Sanomaths.fi · Europe · National news | Blocks all AI bots | 0 | 12 | 0 | 19/20 | No | |
| HuffPosthuffpost.com · United States · Digital-native | Blocks all AI bots | 0 | 0 | 10 | 19/20 | No | |
| Irish Independentindependent.ie · UK & Ireland · National news | Blocks all AI bots | 9 | 25 | 0 | 18/20 | No | |
| Kompaskompas.com · Asia-Pacific · National news | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| La Terceralatercera.com · Latin America · National news | Blocks all AI bots | 9 | 0 | 20 | 18/20 | No | |
| Le Devoirledevoir.com · Canada · National news | Blocks all AI bots | 5 | 12 | 0 | 18/20 | No | |
| Le Soirlesoir.be · Europe · National news | Blocks all AI bots | 0 | 0 | 20 | 19/20 | Yes | |
| Les Echoslesechos.fr · Europe · National news | Blocks all AI bots | 9 | 0 | 0 | 19/20 | No | |
| Mashablemashable.com · United States · Digital-native | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| Milwaukee Journal Sentineljsonline.com · United States · Regional / local | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| NBC Newsnbcnews.com · United States · Broadcaster | Blocks all AI bots | 0 | 12 | 0 | 19/20 | No | |
| New York Magazinenymag.com · United States · Magazine | Blocks all AI bots | 18 | 12 | 0 | 17/20 | No | |
| news.com.aunews.com.au · Asia-Pacific · Digital-native | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| News24news24.com · Africa · Digital-native | Blocks all AI bots | 0 | 0 | 10 | 19/20 | No | |
| Nikkei Asiaasia.nikkei.com · Asia-Pacific · Magazine | Blocks all AI bots | 0 | 12 | 10 | 18/20 | No | |
| NJ.comnj.com · United States · Regional / local | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| NRCnrc.nl · Europe · National news | Blocks all AI bots | 0 | 0 | 10 | 19/20 | No | |
| NRKnrk.no · Europe · Broadcaster | Blocks all AI bots | 0 | 0 | 10 | 19/20 | No | |
| O Globooglobo.globo.com · Latin America · National news | Blocks all AI bots | 9 | 0 | 10 | 17/20 | No | |
| Ouest-Franceouest-france.fr · Europe · Regional / local | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Peoplepeople.com · United States · Magazine | Blocks all AI bots | 5 | 12 | 10 | 17/20 | No | |
| Politico Europepolitico.eu · Europe · Digital-native | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Publicopublico.pt · Europe · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Rapplerrappler.com · Asia-Pacific · Digital-native | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| Reutersreuters.com · UK & Ireland · Wire / agency | Blocks all AI bots | 0 | 12 | 10 | 18/20 | No | |
| Semaforsemafor.com · United States · Digital-native | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| The Arizona Republicazcentral.com · United States · Regional / local | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| The Australiantheaustralian.com.au · Asia-Pacific · National news | Blocks all AI bots | 9 | 25 | 20 | 17/20 | No | |
| The Baltimore Sunbaltimoresun.com · United States · Regional / local | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| The Denver Postdenverpost.com · United States · Regional / local | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| The Globe and Mailtheglobeandmail.com · Canada · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| The Heraldheraldscotland.com · UK & Ireland · Regional / local | Blocks all AI bots | 5 | 0 | 10 | 18/20 | No | |
| The i Paperinews.co.uk · UK & Ireland · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| The New York Timesnytimes.com · United States · National news | Blocks all AI bots | 5 | 0 | 10 | 18/20 | No | |
| The New Yorkernewyorker.com · United States · Magazine | Blocks all AI bots | 9 | 12 | 10 | 16/20 | No | |
| The New Zealand Heraldnzherald.co.nz · Asia-Pacific · National news | Blocks all AI bots | 0 | 0 | 0 | 20/20 | No | |
| The Oregonianoregonlive.com · United States · Regional / local | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| The Sunthesun.co.uk · UK & Ireland · National news | Blocks all AI bots | 5 | 12 | 10 | 17/20 | No | |
| The Telegraphtelegraph.co.uk · UK & Ireland · National news | Blocks all AI bots | 5 | 0 | 0 | 19/20 | No | |
| The Timesthetimes.co.uk · UK & Ireland · National news | Blocks all AI bots | 5 | 12 | 10 | 17/20 | No |
05Methodology
How the grades are worked out
Fetching
One request at a time, about one per second, as
PlumbLeaderboard/1.0with a link back to this page. Redirects are followed only within the publisher's own domain. A 404 or 410 means no robots.txt, which under RFC 9309 restricts nothing. A 401, 403, 429, 5xx, timeout or off-site redirect is recorded as “could not fetch” and left out of every share. Some publishers block unfamiliar user agents at the edge, so that is a fact about their firewall, not their robots.txt.Which rules apply to a bot
The group that names the bot's token exactly, case-insensitive; if several groups name it, they are merged. If none does, the
*group. AGooglebotgroup does not coverGoogle-Extended, and aClaudeBotgroup does not coverClaude-User, because the operators say their tokens are separate.Blocked, partial, allowed
Blocked: the applicable group has
Disallow: /(or/*) and the root stays disallowed, using longest-match with Allow winning ties. Allowing only the homepage (Allow: /$) still counts as blocked. Allow exceptions under a full block, such as a sponsored section, are listed on the publisher page but do not change the verdict. Partial: the applicable group, named or*, disallows some paths but not the root. Housekeeping rules in the*group (search pages, accounts, admin, feeds, scripts, query strings) are not counted, so a*group with only those is allowed. Allowed: everything else.Openness and stance
For training, AI search and user fetch separately, openness = (allowed + half of partial) ÷ bots in the group × 100. A group is closed when at least 70% of its bots are blocked, open when none is blocked or partial. Stance: all three closed is “blocks all AI bots”; training and search closed is “blocks training and AI search”; training closed with search fully open is “blocks training, allows AI search”; all open is “open”; anything else is “blocks some AI bots”.
Bots graded
Model training: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, DiffbotAI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBotUser-requested fetch: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher
llms.txt and the rest
llms.txt counts when
/llms.txtreturns 200 with a non-HTML body. Robots.txt size is read up to 500 KB, the limit Google documents. Sitemap lines are counted as written.What this does not tell you
A robots.txt is a request. It does not show whether a bot obeyed it, whether the publisher also blocks at the edge, or whether a licensing deal sits behind an open door. The seed list is a curated sample of major publishers, not a census; it is versioned in the repository (2026-11-21.2).