Robots.txt leaderboardMEASURED

Which publishers let AI bots in, read from their own robots.txt.

Plumb fetched the live robots.txt of 344 news and media publishers in 9 regions and graded 20 AI crawler tokens against each file. This is what each publisher has told the bots, not what the bots did.

Scanned 21 November 2026 · 319 files read, 25 could not be fetched · rescanned weekly · user agent PlumbLeaderboard/1.0 (+https://plumbtrace.com/leaderboard)

Not on the list? Audit any site · what each bot is: the AI agent directory

53%
block OpenAI's GPTBot outright
70%
block at least one of the 11 training crawlers
4%
block training but leave AI search open
8%
publish an llms.txt file

Shares are of the 319 publishers whose robots.txt could be read (including 5 with no file, which block nothing).

01By bot

The average training bot is blocked by 52%. The average AI search bot, by 34%.

Share of publishers blocking each token, grouped by what its operator says the bot is for. A partial verdict means the publisher named the bot and closed some sections, not the whole site.

Model trainingavg openness 31/100

  • CCBotCommon Crawl60%
  • ClaudeBotAnthropic57%
  • BytespiderByteDance56%
  • GPTBotOpenAI53%
  • anthropic-aiAnthropic53%
  • Applebot-ExtendedApple51%
  • meta-externalagentMeta50%
  • DiffbotDiffbot50%
  • cohere-aiCohere49%
  • Google-ExtendedGoogle47%
  • AmazonbotAmazon47%

AI searchavg openness 44/100

  • PerplexityBotPerplexity48%
  • OAI-SearchBotOpenAI30%
  • Claude-SearchBotAnthropic29%
  • DuckAssistBotDuckDuckGo28%

User-requested fetchavg openness 45/100

  • ChatGPT-UserOpenAI38%
  • Meta-ExternalFetcherMeta33%
  • Perplexity-UserPerplexity30%
  • Claude-UserAnthropic28%
  • MistralAI-UserMistral24%
BlockedPartialAllowed or not mentioned

02Stance

One grade hides the decision. The stance shows it.

Each publisher gets a label from three separate readings: training, AI search and user-requested fetches.

  • Blocks all AI bots7020%
  • Blocks training and AI search154%
  • Blocks training, allows AI search144%
  • Blocks some AI bots18955%
  • Open to AI bots268%
  • No robots.txt51%
  • Could not fetch257%
RegionPublishersBlock GPTBotTraining closedAI search closedUser fetch closedCould not fetch
United States9756%59%35%30%9
Canada1273%64%36%36%1
UK & Ireland3371%55%42%26%2
Europe5777%70%39%34%1
India2312%0%0%0%6
Asia-Pacific4557%50%30%20%1
Latin America2926%22%19%15%2
Africa2640%28%4%12%1
Middle East2215%10%5%5%2
Publisher typePublishersBlock GPTBotTraining closedAI search closedUser fetch closedCould not fetch
National news13648%46%31%28%4
Regional / local4374%55%42%32%5
Magazine2839%50%36%25%0
Wire / agency1657%43%14%14%2
Broadcaster4870%53%25%13%8
B2B / trade1747%47%0%0%2
Digital-native5644%42%21%21%4

03Changes

Who changed their stance since the last scan

This is the first scan on record. Changes show up here after the next weekly scan.

04Every publisher

Look up a publisher

Openness runs from 0 (every bot in that group blocked) to 100 (none blocked). The strip shows each of the 20 bots in order: training, then AI search, then user fetch.

344 of 344 publishersDownload all as CSV
Per bot: training · search · userllms.txt
ABCabc.es · Europe · National newsBlocks all AI bots140017/20No
Ars Technicaarstechnica.com · United States · Digital-nativeBlocks all AI bots5121017/20No
Asahi Shimbunasahi.com · Asia-Pacific · National newsBlocks all AI bots0121018/20No
Australian Financial Reviewafr.com · Asia-Pacific · National newsBlocks all AI bots0121018/20No
Chicago Tribunechicagotribune.com · United States · Regional / localBlocks all AI bots50019/20No
Cleveland.comcleveland.com · United States · Regional / localBlocks all AI bots50019/20No
CNNcnn.com · United States · BroadcasterBlocks all AI bots901017/20No
Corriere della Seracorriere.it · Europe · National newsBlocks all AI bots901018/20No
Dagens Nyheterdn.se · Europe · National newsBlocks all AI bots00020/20No
Daily Maildailymail.com · UK & Ireland · National newsBlocks all AI bots00020/20Yes
De Standaardstandaard.be · Europe · National newsBlocks all AI bots925018/20No
De Telegraaftelegraaf.nl · Europe · National newsBlocks all AI bots925018/20No
de Volkskrantvolkskrant.nl · Europe · National newsBlocks all AI bots00020/20No
Detroit Free Pressfreep.com · United States · Regional / localBlocks all AI bots00020/20No
Die Zeitzeit.de · Europe · National newsBlocks all AI bots00020/20No
DRdr.dk · Europe · BroadcasterBlocks all AI bots501018/20No
El Mundoelmundo.es · Europe · National newsBlocks all AI bots00020/20No
El Universaleluniversal.com.mx · Latin America · National newsBlocks all AI bots50019/20No
Franceinfofranceinfo.fr · Europe · BroadcasterBlocks all AI bots00020/20No
Haaretzhaaretz.com · Middle East · National newsBlocks all AI bots0121018/20No
Helsingin Sanomaths.fi · Europe · National newsBlocks all AI bots012019/20No
HuffPosthuffpost.com · United States · Digital-nativeBlocks all AI bots001019/20No
Irish Independentindependent.ie · UK & Ireland · National newsBlocks all AI bots925018/20No
Kompaskompas.com · Asia-Pacific · National newsBlocks all AI bots50019/20No
La Terceralatercera.com · Latin America · National newsBlocks all AI bots902018/20No
Le Devoirledevoir.com · Canada · National newsBlocks all AI bots512018/20No
Le Soirlesoir.be · Europe · National newsBlocks all AI bots002019/20Yes
Les Echoslesechos.fr · Europe · National newsBlocks all AI bots90019/20No
Mashablemashable.com · United States · Digital-nativeBlocks all AI bots50019/20No
Milwaukee Journal Sentineljsonline.com · United States · Regional / localBlocks all AI bots00020/20No
NBC Newsnbcnews.com · United States · BroadcasterBlocks all AI bots012019/20No
New York Magazinenymag.com · United States · MagazineBlocks all AI bots1812017/20No
news.com.aunews.com.au · Asia-Pacific · Digital-nativeBlocks all AI bots00020/20No
News24news24.com · Africa · Digital-nativeBlocks all AI bots001019/20No
Nikkei Asiaasia.nikkei.com · Asia-Pacific · MagazineBlocks all AI bots0121018/20No
NJ.comnj.com · United States · Regional / localBlocks all AI bots50019/20No
NRCnrc.nl · Europe · National newsBlocks all AI bots001019/20No
NRKnrk.no · Europe · BroadcasterBlocks all AI bots001019/20No
O Globooglobo.globo.com · Latin America · National newsBlocks all AI bots901017/20No
Ouest-Franceouest-france.fr · Europe · Regional / localBlocks all AI bots00020/20No
Peoplepeople.com · United States · MagazineBlocks all AI bots5121017/20No
Politico Europepolitico.eu · Europe · Digital-nativeBlocks all AI bots00020/20No
Publicopublico.pt · Europe · National newsBlocks all AI bots00020/20No
Rapplerrappler.com · Asia-Pacific · Digital-nativeBlocks all AI bots00020/20No
Reutersreuters.com · UK & Ireland · Wire / agencyBlocks all AI bots0121018/20No
Semaforsemafor.com · United States · Digital-nativeBlocks all AI bots00020/20No
The Arizona Republicazcentral.com · United States · Regional / localBlocks all AI bots00020/20No
The Australiantheaustralian.com.au · Asia-Pacific · National newsBlocks all AI bots9252017/20No
The Baltimore Sunbaltimoresun.com · United States · Regional / localBlocks all AI bots50019/20No
The Denver Postdenverpost.com · United States · Regional / localBlocks all AI bots50019/20No
The Globe and Mailtheglobeandmail.com · Canada · National newsBlocks all AI bots00020/20No
The Heraldheraldscotland.com · UK & Ireland · Regional / localBlocks all AI bots501018/20No
The i Paperinews.co.uk · UK & Ireland · National newsBlocks all AI bots00020/20No
The New York Timesnytimes.com · United States · National newsBlocks all AI bots501018/20No
The New Yorkernewyorker.com · United States · MagazineBlocks all AI bots9121016/20No
The New Zealand Heraldnzherald.co.nz · Asia-Pacific · National newsBlocks all AI bots00020/20No
The Oregonianoregonlive.com · United States · Regional / localBlocks all AI bots50019/20No
The Sunthesun.co.uk · UK & Ireland · National newsBlocks all AI bots5121017/20No
The Telegraphtelegraph.co.uk · UK & Ireland · National newsBlocks all AI bots50019/20No
The Timesthetimes.co.uk · UK & Ireland · National newsBlocks all AI bots5121017/20No

05Methodology

How the grades are worked out

  • Fetching

    One request at a time, about one per second, as PlumbLeaderboard/1.0with a link back to this page. Redirects are followed only within the publisher's own domain. A 404 or 410 means no robots.txt, which under RFC 9309 restricts nothing. A 401, 403, 429, 5xx, timeout or off-site redirect is recorded as “could not fetch” and left out of every share. Some publishers block unfamiliar user agents at the edge, so that is a fact about their firewall, not their robots.txt.

  • Which rules apply to a bot

    The group that names the bot's token exactly, case-insensitive; if several groups name it, they are merged. If none does, the * group. A Googlebot group does not cover Google-Extended, and a ClaudeBot group does not cover Claude-User, because the operators say their tokens are separate.

  • Blocked, partial, allowed

    Blocked: the applicable group has Disallow: / (or /*) and the root stays disallowed, using longest-match with Allow winning ties. Allowing only the homepage (Allow: /$) still counts as blocked. Allow exceptions under a full block, such as a sponsored section, are listed on the publisher page but do not change the verdict. Partial: the applicable group, named or *, disallows some paths but not the root. Housekeeping rules in the * group (search pages, accounts, admin, feeds, scripts, query strings) are not counted, so a * group with only those is allowed. Allowed: everything else.

  • Openness and stance

    For training, AI search and user fetch separately, openness = (allowed + half of partial) ÷ bots in the group × 100. A group is closed when at least 70% of its bots are blocked, open when none is blocked or partial. Stance: all three closed is “blocks all AI bots”; training and search closed is “blocks training and AI search”; training closed with search fully open is “blocks training, allows AI search”; all open is “open”; anything else is “blocks some AI bots”.

  • Bots graded

    Model training: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, DiffbotAI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBotUser-requested fetch: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher

  • llms.txt and the rest

    llms.txt counts when /llms.txt returns 200 with a non-HTML body. Robots.txt size is read up to 500 KB, the limit Google documents. Sitemap lines are counted as written.

  • What this does not tell you

    A robots.txt is a request. It does not show whether a bot obeyed it, whether the publisher also blocks at the edge, or whether a licensing deal sits behind an open door. The seed list is a curated sample of major publishers, not a census; it is versioned in the repository (2026-11-21.2).