comizaAgentwatch › Study
Only 5.9% of the traffic claiming to be an OpenAI crawler came from OpenAI
Between 26 July 2026 to 09 August 2026, this server logged 112,144 requests. We took every one that claimed to be an AI crawler and checked the address it came from against the list the vendor itself publishes for that crawler. Where the claim could be checked at all, 42.3% of it was genuine.
What this is not, and where it is weakest
One server, one small site, 15 days. It is not a survey of the web and nothing here should be read as a global figure. It is a measurement anyone can repeat on their own logs, and the method is written out below so that you can.
The honest weakness is concentration. A handful of busy scanners can move these percentages on a site this size, and they do: the single busiest unconfirmed address accounts for 48% of all unconfirmed requests, and the top five for 75%. This is a real picture of what one server receives, not a population estimate.
The results
Requests are counted per claimed identity. The percentage is the share that could be traced to an address the vendor publishes for that exact crawler — each OpenAI crawler against its own file, not a pooled one.
| Claimed to be | Requests | Addresses | Confirmed | Checked against |
|---|---|---|---|---|
Googlebot | 4,442 | 102 | 86.0% | address list, then reverse DNS |
GPTBot | 3,272 | 61 | 3.7% | published address list |
OAI-SearchBot | 1,685 | 84 | 12.0% | published address list |
ChatGPT-User | 509 | 27 | 0.2% | published address list |
bingbot | 316 | 129 | 84.8% | published address list |
PerplexityBot | 213 | 13 | 0.0% | published address list |
| PerplexityBot: Read this row carefully. In August 2025 Cloudflare published that Perplexity was crawling from addresses outside its own published list, rotating networks and changing its User-Agent; Perplexity disputed this publicly and attributed the traffic to a third party. So 0% confirmed here means one of two very different things — impersonators wearing the name, or Perplexity crawling from outside the range it declares. Our method cannot separate them, and we do not claim to. | ||||
ClaudeBot | 802 | 47 | no verification method published | |
Google-Extended | 154 | 8 | no verification method published | |
meta-externalagent | 119 | 70 | no verification method published | |
CCBot | 45 | 9 | no verification method published | |
Bytespider | 21 | 8 | no verification method published | |
Perplexity-User | 12 | 2 | no verification method published | |
Claude-User | 2 | 1 | no verification method published | |
Rows without a percentage are not accusations. Anthropic, Meta, ByteDance and Common Crawl publish neither an address list nor a documented reverse DNS convention, so their traffic can be neither confirmed nor denied. Those rows are excluded from every percentage on this page.
The part that removes any doubt
A low percentage could, in principle, mean a vendor crawling from addresses it forgot to publish. This cannot.
21 addresses wore more than one crawler's name. The 5 below each cycled through 8 to 10 different AI crawler identities while requesting the same kind of thing: environment files, cloud credentials, admin panels and process memory.
| Address | Requests | Identities | Names used | Most requested paths |
|---|---|---|---|---|
34.56.220.217 | 104 | 10 | GPTBot (19), ClaudeBot (18), PerplexityBot (13), OAI-SearchBot (12), ChatGPT-User (12), Perplexity-User (7), meta-externalagent (6), Google-Extended (6), Bytespider (6), CCBot (5) | /env.js/admin/.env/.env.local |
104.28.227.187 | 103 | 10 | GPTBot (19), ChatGPT-User (17), PerplexityBot (16), ClaudeBot (14), OAI-SearchBot (11), Bytespider (9), Perplexity-User (5), meta-externalagent (4), CCBot (4), Google-Extended (4) | /.env.local/.env.production/.env.development |
23.161.169.114 | 57 | 8 | ClaudeBot (11), GPTBot (9), Googlebot (9), bingbot (8), ChatGPT-User (6), CCBot (6), PerplexityBot (5), OAI-SearchBot (3) | /graphql//.well-known/security.txt |
23.161.169.113 | 55 | 8 | bingbot (13), Googlebot (11), GPTBot (10), CCBot (5), ChatGPT-User (5), ClaudeBot (4), PerplexityBot (4), OAI-SearchBot (3) | /graphql//.well-known/security.txt |
154.58.229.19 | 45 | 8 | CCBot (8), PerplexityBot (7), ChatGPT-User (7), ClaudeBot (6), bingbot (5), GPTBot (4), OAI-SearchBot (4), Googlebot (4) | //.git/HEAD/products.js |
No real crawler changes its name 10 times. This is one scanner wearing whatever gets it through the door — and AI crawler names get through the door precisely because site owners have spent two years being told to let them in.
What follows from it
- Allowing a crawler by its User-Agent is not a security decision, it is a hope
- The name in a request is typed by whoever sent it. If your firewall exempts anything calling itself GPTBot, you have written an exemption that anyone can claim, and on this server most of what claimed it was not it.
- Your crawl statistics are probably wrong
- Analytics and log tools group by User-Agent because that is all they have. If you have been reassured by how much AI crawler traffic you get, check it. We counted claims first too, and got the answer wrong by an order of magnitude until we checked the addresses.
- Verification is cheap, and the vendors made it possible
- OpenAI, Google, Microsoft and Perplexity publish the addresses their crawlers use, as machine readable files. Google and Microsoft also support reverse DNS. Checking takes milliseconds and needs no relationship with anyone.
Method, so you can repeat it
- Take every access log line and pull out the source address and the User-Agent.
- Match the User-Agent against a catalogue of known crawler tokens,
longest token first — otherwise
Claude-SearchBotis misread asClaudeBot. - For each claimed crawler, fetch the vendor's published address list. Follow redirects: Google has moved its file and Perplexity has moved its host, and a client that does not follow gets an empty list and silently concludes the vendor publishes nothing.
- Check the address against that crawler's own list.
- If it is not there and the vendor documents reverse DNS, resolve the address to a name, then resolve that name back. Both halves are required; a reverse record on its own can say anything.
- Only if every available method fails, call it unconfirmed — and if the vendor publishes no method at all, say that instead, because it is a different statement.
Our implementation is the same code that answers the free check on this site, so the numbers above and the tool agree by construction. Measured on 26 July 2026 to 09 August 2026; regenerated whenever the logs are.