comiza

Agentwatch › Crawlers › Google-Extended

Google-Extended

Google-Extended is not a crawler. It is a name that exists only inside robots.txt, so that you can tell Google whether your content may be used to train its models. Google never sends a request under this name, so anything in your access log carrying it is an impostor by definition. It is a preference you record, not a visitor you receive, and no firewall rule can act on it.

Run byGoogle
robots.txt tokenGoogle-Extended
Purposetraining only
Cost of blockingNothing. Your visibility is unaffected.

What blocking Google-Extended actually costs

Blocking it costs you no visibility whatsoever, because it governs training and nothing else. It is also the safest entry in this whole list to add to your robots.txt: since nothing ever requests anything under this name, the rule cannot accidentally shut a door you wanted open.

The one thing to get right is not to confuse it with the crawler that does the actual fetching for that vendor. They are separate names on purpose.

Google runs two of these, and they do different things

This is the part that costs people money. One Disallow aimed at the wrong name here is the difference between refusing to be trained on and vanishing from the answers.

robots.txt tokenWhat it is forWhat blocking it costs
Googlebotfeeds answersYou disappear from its answers.
Google-Extended ← this pagetraining onlyNothing. Your visibility is unaffected.

The rules, exactly

Because nothing ever requests a page under this name, the rule below is the entire mechanism. There is no server side to configure and nothing to verify in your logs.

To allow it

User-agent: Google-Extended Allow: /

An explicit Allow is only needed when a broader rule would otherwise catch it. If your robots.txt does not disallow anything, this crawler is already allowed and you need no rule at all.

To block it

User-agent: Google-Extended Disallow: /

Put it in its own group. A named group replaces the * group entirely for that crawler and inherits nothing from it, which surprises almost everyone.

Three things about robots.txt that catch people out

A crawler obeys exactly one group
It picks the group whose User-agent value is the longest one that matches its name, and ignores every other group, including *. If you write a rule under * and a separate group for Google-Extended, the rules under * do not apply to it at all.
Matching is by prefix, not by exact name
A group headed User-agent: Google matches Googlebot, and a group headed with a partial name matches more than you intended. Write the full token.
A server error on robots.txt blocks everything
If /robots.txt returns a 5xx, the documented behaviour is that crawlers stop crawling the whole site until it recovers. A missing file returning 404 is safe; a broken one is not.

Making sure it is really them

A User-Agent is a claim, not proof. Anyone can send any name, so a rule that trusts the name alone can be walked straight through.

Anything arriving under this name is fake by definition, because the real thing never makes requests. Treat it as a sign to look closer, not as a crawler to allow.

Official documentation: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

Every crawler in the catalogue