← AI agent directory

Google · Model training

Google-Extended

A robots.txt token, not a separate crawler. Google fetches with Googlebot; this token controls whether that content may be used for its Gemini models.

Operator
Google
Purpose
Model training
Verification
Published IP ranges
Follows robots.txt
Operator says it follows robots.txt

Operator documentation: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

01Identify and verify

How it shows up in your logs

User-agent contains

  • Google-Extended

Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.

Is it really Google-Extended?

Plumb checks the request IP against the operator's published ranges.

Ranges checked: google:special, google:googlebot, google:user-fetchers, google:user-fetchers-google

02Publishers

How major publishers treat it

47%of 319 publishers on the leaderboard block Google-Extended outright (151). 116 more close some sections. 174 name it explicitly in their robots.txt.

From the scan of 21 November 2026 · see every publisher

03robots.txt

A rule you can paste

The group names Google-Extended exactly. A group for a different token from the same company does not cover it.

# Google-Extended (Google)
User-agent: Google-Extended
Disallow: /

04Same operator

Google's other agents

Google-Extended (Google): what it is and how to block it · Plumb