comiza

Agentwatch › Crawlers › Bytespider

Bytespider

Bytespider collects text so ByteDance can train models on it. It does not decide whether you appear in any answer, and refusing it is a normal, uncontroversial choice that many publishers make.

Run byByteDance
robots.txt tokenBytespider
Purposetraining only
Cost of blockingNothing. Your visibility is unaffected.

What blocking Bytespider actually costs

Nothing. Blocking Bytespider does not remove you from ByteDance's products, does not affect whether you are cited, and does not change your search ranking. It only means your text is not collected for training.

This is worth stating plainly because the opposite belief is widespread, and it runs in both directions: people block it expecting to disappear from the assistant and are confused when they do not, and people allow it believing that is how they get cited, which it is not.

The rules, exactly

To allow it

User-agent: Bytespider Allow: /

An explicit Allow is only needed when a broader rule would otherwise catch it. If your robots.txt does not disallow anything, this crawler is already allowed and you need no rule at all.

To block it

User-agent: Bytespider Disallow: /

Put it in its own group. A named group replaces the * group entirely for that crawler and inherits nothing from it, which surprises almost everyone.

Three things about robots.txt that catch people out

A crawler obeys exactly one group
It picks the group whose User-agent value is the longest one that matches its name, and ignores every other group, including *. If you write a rule under * and a separate group for Bytespider, the rules under * do not apply to it at all.
Matching is by prefix, not by exact name
A group headed User-agent: Google matches Googlebot, and a group headed with a partial name matches more than you intended. Write the full token.
A server error on robots.txt blocks everything
If /robots.txt returns a 5xx, the documented behaviour is that crawlers stop crawling the whole site until it recovers. A missing file returning 404 is safe; a broken one is not.

Why your robots.txt may say yes while Bytespider still gets nothing

robots.txt is a request. A firewall is not. Bot protection at your CDN answers before your site does, and it has never read your robots.txt.

Every one of these crawlers arrives from a data centre, which is exactly what bot rules are tuned to refuse. The result is a site with a perfectly permissive robots.txt sitting behind a wall, and no checker that only reads robots.txt can see it, because it never makes the request.

The free scan on this site asks for your page as Bytespider, from a real server, and compares the answer against what an ordinary browser gets. If they differ, you have found the wall.

Check your site Free, no account, no email.

Making sure it is really them

A User-Agent is a claim, not proof. Anyone can send any name, so a rule that trusts the name alone can be walked straight through.

ByteDance does not publish address ranges in a form site owners can easily check, which is one reason this crawler is widely blocked.

Every crawler in the catalogue