comiza

Agentwatch › Crawlers › Applebot-Extended

Applebot-Extended

Applebot-Extended is not a crawler. It is a name that exists only inside robots.txt, so that you can tell Apple whether your content may be used to train its models. Apple never sends a request under this name, so anything in your access log carrying it is an impostor by definition. It is a preference you record, not a visitor you receive, and no firewall rule can act on it.

Run byApple
robots.txt tokenApplebot-Extended
Purposetraining only
Cost of blockingNothing. Your visibility is unaffected.

What blocking Applebot-Extended actually costs

Blocking it costs you no visibility whatsoever, because it governs training and nothing else. It is also the safest entry in this whole list to add to your robots.txt: since nothing ever requests anything under this name, the rule cannot accidentally shut a door you wanted open.

The one thing to get right is not to confuse it with the crawler that does the actual fetching for that vendor. They are separate names on purpose.

The rules, exactly

Because nothing ever requests a page under this name, the rule below is the entire mechanism. There is no server side to configure and nothing to verify in your logs.

To allow it

User-agent: Applebot-Extended Allow: /

An explicit Allow is only needed when a broader rule would otherwise catch it. If your robots.txt does not disallow anything, this crawler is already allowed and you need no rule at all.

To block it

User-agent: Applebot-Extended Disallow: /

Put it in its own group. A named group replaces the * group entirely for that crawler and inherits nothing from it, which surprises almost everyone.

Three things about robots.txt that catch people out

A crawler obeys exactly one group
It picks the group whose User-agent value is the longest one that matches its name, and ignores every other group, including *. If you write a rule under * and a separate group for Applebot-Extended, the rules under * do not apply to it at all.
Matching is by prefix, not by exact name
A group headed User-agent: Google matches Googlebot, and a group headed with a partial name matches more than you intended. Write the full token.
A server error on robots.txt blocks everything
If /robots.txt returns a 5xx, the documented behaviour is that crawlers stop crawling the whole site until it recovers. A missing file returning 404 is safe; a broken one is not.

Making sure it is really them

A User-Agent is a claim, not proof. Anyone can send any name, so a rule that trusts the name alone can be walked straight through.

Anything arriving under this name is fake by definition, because the real thing never makes requests. Treat it as a sign to look closer, not as a crawler to allow.

Official documentation: https://support.apple.com/en-us/119829

Every crawler in the catalogue