← AI agent directory

Meta · User-triggered fetch

Meta-ExternalFetcher

Operator
Meta
Purpose
User-triggered fetch
Verification
None
Follows robots.txt
Operator says it may ignore robots.txt

Operator documentation: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers

01Identify and verify

How it shows up in your logs

User-agent contains

  • meta-externalfetcher

Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.

Is it really Meta-ExternalFetcher?

The operator publishes no ranges or keys, so any request using this name is unverified.

Meta documents no crawler IP ranges.

02Publishers

How major publishers treat it

33%of 319 publishers on the leaderboard block Meta-ExternalFetcher outright (104). 150 more close some sections. 99 name it explicitly in their robots.txt.

From the scan of 21 November 2026 · see every publisher

03robots.txt

A rule you can paste

The group names meta-externalfetcher exactly. A group for a different token from the same company does not cover it.

Meta says this fetcher may bypass robots.txt because a person asked for the page. A rule here records your preference; enforcing it takes a block at your CDN or firewall.

# Meta-ExternalFetcher (Meta)
User-agent: meta-externalfetcher
Disallow: /

04Same operator

Meta's other agents

Meta-ExternalFetcher (Meta): what it is and how to block it · Plumb