← AI agent directory

Google · Search engine

Googlebot

Operator
Google
Purpose
Search engine
Verification
Published IP ranges
Follows robots.txt
Operator says it follows robots.txt

Operator documentation: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

01Identify and verify

How it shows up in your logs

User-agent contains

  • Googlebot

Matching is case-insensitive. Anyone can send this string, so a match alone says only what the request claims to be.

Is it really Googlebot?

Plumb checks the request IP against the operator's published ranges.

Ranges checked: google:googlebot, google:special, google:user-fetchers, google:user-fetchers-google

02Publishers

How major publishers treat it

The leaderboard does not grade Googlebot yet, so there is no publisher figure for it.

03robots.txt

A rule you can paste

The group names Googlebot exactly. A group for a different token from the same company does not cover it.

# Googlebot (Google)
User-agent: Googlebot
Disallow: /

04Same operator

Google's other agents

Googlebot (Google): what it is and how to block it · Plumb