Google · Model training
Google-Extended
A robots.txt token, not a separate crawler. Google fetches with Googlebot; this token controls whether that content may be used for its Gemini models.
- Operator
- Purpose
- Model training
- Verification
- Published IP ranges
- Follows robots.txt
- Operator says it follows robots.txt
01Identify and verify
How it shows up in your logs
User-agent contains
Google-Extended
Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.
Is it really Google-Extended?
Plumb checks the request IP against the operator's published ranges.
Ranges checked: google:special, google:googlebot, google:user-fetchers, google:user-fetchers-google
02Publishers
How major publishers treat it
47%of 319 publishers on the leaderboard block
Google-Extended outright (151). 116 more close some sections. 174 name it explicitly in their robots.txt.03robots.txt
A rule you can paste
The group names Google-Extended exactly. A group for a different token from the same company does not cover it.
# Google-Extended (Google) User-agent: Google-Extended Disallow: /
04Same operator
Google's other agents
- GoogleAgent-MarinerUser-triggered fetch
- Google-AgentUser-triggered fetch
- Google-CloudVertexBotOther
- Googlebot-ExtendedAI search indexing
- GoogleOtherOther
- GooglebotSearch engine