comiza

Agentwatch › Crawlers › PerplexityBot

PerplexityBot

PerplexityBot is how Perplexity builds the index that Perplexity answers from. When it cannot reach your pages, they are not in that index, and an assistant cannot cite a page it has never read. This is the crawler people block by accident while meaning to block something else.

Run byPerplexity
robots.txt tokenPerplexityBot
Purposefeeds answers
Cost of blockingYou disappear from its answers.

What blocking PerplexityBot actually costs

Your pages stop appearing in Perplexity. Not immediately and not visibly: what happens is that the index goes stale and then thin, and the traffic that would have come never arrives. There is no notification, no warning in any console, and nothing in your analytics that says why.

If you want to refuse training but stay quotable, this is precisely the crawler you must keep allowed.

Perplexity runs two of these, and they do different things

This is the part that costs people money. One Disallow aimed at the wrong name here is the difference between refusing to be trained on and vanishing from the answers.

robots.txt tokenWhat it is forWhat blocking it costs
PerplexityBot ← this pagefeeds answersYou disappear from its answers.
Perplexity-Userlive fetchIt cannot look at your page when somebody asks about you.

The rules, exactly

To allow it

User-agent: PerplexityBot Allow: /

An explicit Allow is only needed when a broader rule would otherwise catch it. If your robots.txt does not disallow anything, this crawler is already allowed and you need no rule at all.

To block it

User-agent: PerplexityBot Disallow: /

Put it in its own group. A named group replaces the * group entirely for that crawler and inherits nothing from it, which surprises almost everyone.

Three things about robots.txt that catch people out

A crawler obeys exactly one group
It picks the group whose User-agent value is the longest one that matches its name, and ignores every other group, including *. If you write a rule under * and a separate group for PerplexityBot, the rules under * do not apply to it at all.
Matching is by prefix, not by exact name
A group headed User-agent: Google matches Googlebot, and a group headed with a partial name matches more than you intended. Write the full token.
A server error on robots.txt blocks everything
If /robots.txt returns a 5xx, the documented behaviour is that crawlers stop crawling the whole site until it recovers. A missing file returning 404 is safe; a broken one is not.

Why your robots.txt may say yes while PerplexityBot still gets nothing

robots.txt is a request. A firewall is not. Bot protection at your CDN answers before your site does, and it has never read your robots.txt.

Every one of these crawlers arrives from a data centre, which is exactly what bot rules are tuned to refuse. The result is a site with a perfectly permissive robots.txt sitting behind a wall, and no checker that only reads robots.txt can see it, because it never makes the request.

The free scan on this site asks for your page as PerplexityBot, from a real server, and compares the answer against what an ordinary browser gets. If they differ, you have found the wall.

Check your site Free, no account, no email.

Making sure it is really them

A User-Agent is a claim, not proof. Anyone can send any name, so a rule that trusts the name alone can be walked straight through.

Perplexity documents its user agents and address ranges in the guide linked below.

Official documentation: https://docs.perplexity.ai/guides/bots

Every crawler in the catalogue