comizaAgentwatch › Crawlers › CCBot
CCBot
CCBot collects text so Common Crawl can train models on it. It does not decide whether you appear in any answer, and refusing it is a normal, uncontroversial choice that many publishers make.
| Run by | Common Crawl |
| robots.txt token | CCBot |
| Purpose | training only |
| Cost of blocking | Nothing. Your visibility is unaffected. |
What blocking CCBot actually costs
Nothing. Blocking CCBot does not remove you from the open datasets most models are trained on, does not affect whether you are cited, and does not change your search ranking. It only means your text is not collected for training.
This is worth stating plainly because the opposite belief is widespread, and it runs in both directions: people block it expecting to disappear from the assistant and are confused when they do not, and people allow it believing that is how they get cited, which it is not.
The rules, exactly
To allow it
User-agent: CCBot
Allow: /
An explicit Allow is only needed when a broader rule would
otherwise catch it. If your robots.txt does not disallow anything, this
crawler is already allowed and you need no rule at all.
To block it
User-agent: CCBot
Disallow: /
Put it in its own group. A named group replaces the
* group entirely for that crawler and inherits nothing
from it, which surprises almost everyone.
Three things about robots.txt that catch people out
- A crawler obeys exactly one group
-
It picks the group whose
User-agentvalue is the longest one that matches its name, and ignores every other group, including*. If you write a rule under*and a separate group forCCBot, the rules under*do not apply to it at all. - Matching is by prefix, not by exact name
-
A group headed
User-agent: Googlematches Googlebot, and a group headed with a partial name matches more than you intended. Write the full token. - A server error on robots.txt blocks everything
-
If
/robots.txtreturns a 5xx, the documented behaviour is that crawlers stop crawling the whole site until it recovers. A missing file returning 404 is safe; a broken one is not.
Why your robots.txt may say yes while CCBot still gets nothing
robots.txt is a request. A firewall is not. Bot protection at your CDN answers before your site does, and it has never read your robots.txt.
Every one of these crawlers arrives from a data centre, which is exactly what bot rules are tuned to refuse. The result is a site with a perfectly permissive robots.txt sitting behind a wall, and no checker that only reads robots.txt can see it, because it never makes the request.
The free scan on this site asks for your page as CCBot, from a real server, and compares the answer against what an ordinary browser gets. If they differ, you have found the wall.
Making sure it is really them
A User-Agent is a claim, not proof. Anyone can send any name, so a rule that trusts the name alone can be walked straight through.
Common Crawl is a non-profit and publishes its crawling practice on the page below.
Official documentation: https://commoncrawl.org/ccbot
Every crawler in the catalogue
OAI-SearchBotfeeds answersChatGPT-Userlive fetchGPTBottraining onlyClaude-SearchBotfeeds answersClaude-Userlive fetchClaudeBottraining onlyPerplexityBotfeeds answersPerplexity-Userlive fetchGoogle-Extendedtraining onlyGooglebotfeeds answersbingbotfeeds answersApplebot-Extendedtraining onlymeta-externalagenttraining onlyBytespidertraining only