Only 5.3% of the traffic claiming to be an OpenAI crawler came from OpenAI

The results

Requests are counted per claimed identity. The percentage is the share that could be traced to an address the vendor publishes for that exact crawler — each OpenAI crawler against its own file, not a pooled one.

Rows without a percentage are not accusations. Anthropic, Meta, ByteDance and Common Crawl publish neither an address list nor a documented reverse DNS convention, so their traffic can be neither confirmed nor denied. Those rows are excluded from every percentage on this page.

How this sits next to the larger measurements

Two security networks have measured the same thing at a scale this server will never see, and their headline number is far lower than the one above. Both can be right at once, and the reason they differ is the most useful thing on this page.

HUMAN Security's Satori team analysed traffic from 16 well-known AI crawlers and found that one request in eighteen carrying an AI crawler user agent was fake — about 5.6%. DataDome recorded 7.9 billion AI agent requests across its network in January and February 2026, and reports that the best known agent names are precisely the ones most often worn by something else. On this server, measured over 28 July 2026 to 9 August 2026, 62.1% of checkable claims could not be confirmed — roughly 11 times their figure. That is a fixed measurement of a fixed window, not a live counter: quote it and it will still say the same thing when someone checks.

Three things account for the gap. None of them requires either side to be wrong.

• The thresholds are not the same. They assert fraud: traffic they have determined is not what it claims to be. This page asserts something weaker — that a claim could not be confirmed against the vendor's own published list. Every fake request is unconfirmed; not every unconfirmed request is fake. The number here is the looser of the two claims and catches more by construction.
• The denominators are not the same. Their share is diluted by enormous volumes of genuine crawling across thousands of commercial sites. On a small and obscure server, real crawlers arrive rarely and automated scanners arrive constantly, so the same absolute amount of forged traffic lands as a far larger proportion.
• Concentration. As stated at the top, a single address accounts for 50% of the unconfirmed traffic here. One busy scanner cannot move a network-wide figure. It moves this one.

So the reading to take away is not that the web is 62% forged — it is not. It is that the ratio arriving at your door depends on how big your door is, and the published industry figure understates it for a small site, because the genuine traffic that dilutes that figure is traffic a small site never receives. Worth knowing before writing a rule that admits AI crawlers on the strength of the name alone.

The part that removes any doubt

A low percentage could, in principle, mean a vendor crawling from addresses it forgot to publish. This cannot.

17 addresses wore more than one crawler's name. The 5 below each cycled through 6 to 10 different AI crawler identities while requesting the same kind of thing: environment files, cloud credentials, admin panels and process memory.

No real crawler changes its name 10 times. This is one scanner wearing whatever gets it through the door — and AI crawler names get through the door precisely because site owners have spent two years being told to let them in.

Is the method itself any good?

The strongest objection to this page is not about the data, it is about the instrument. If the check produces false negatives, every "unconfirmed" number above is soft — and the honest answer cannot be to assert that the method is good.

So here is a test of the method against itself. A real crawler reads pages. A scanner wearing a crawler's name hunts for login forms and configuration files. If the two groups behave differently, the check is tracking something real rather than failing at random.

The confirmed group asks for /robots.txt and ordinary pages. The unconfirmed group asks for wp-login.php, /wp-json/ and index.php. Google does not hunt for WordPress login forms on a site that has never run WordPress.

What this does not prove. It does not establish a false negative rate, and we do not claim one. A genuine crawler using an address its own vendor has not published would land in the second group and look like an impostor to us. The behavioural gap makes that unlikely to explain much of it; it does not make it impossible.

What follows from it

Allowing a crawler by its User-Agent is not a security decision, it is a hope

The name in a request is typed by whoever sent it. If your firewall exempts anything calling itself GPTBot, you have written an exemption that anyone can claim, and on this server most of what claimed it was not it.

Your crawl statistics are probably wrong

Analytics and log tools group by User-Agent because that is all they have. If you have been reassured by how much AI crawler traffic you get, check it. We counted claims first too, and got the answer wrong by an order of magnitude until we checked the addresses.

Verification is cheap, and the vendors made it possible

OpenAI, Google, Microsoft and Perplexity publish the addresses their crawlers use, as machine readable files. Google and Microsoft also support reverse DNS. Checking takes milliseconds and needs no relationship with anyone.

Method, so you can repeat it

1. Take every access log line and pull out the source address and the User-Agent.
2. Match the User-Agent against a catalogue of known crawler tokens, longest token first — otherwise Claude-SearchBot is misread as ClaudeBot.
3. For each claimed crawler, fetch the vendor's published address list. Follow redirects: Google has moved its file and Perplexity has moved its host, and a client that does not follow gets an empty list and silently concludes the vendor publishes nothing.
4. Check the address against that crawler's own list.
5. If it is not there and the vendor documents reverse DNS, resolve the address to a name, then resolve that name back. Both halves are required; a reverse record on its own can say anything.
6. Only if every available method fails, call it unconfirmed — and if the vendor publishes no method at all, say that instead, because it is a different statement.

Our implementation is the same code that answers the free check on this site, so the numbers above and the tool agree by construction. Measured on 28 July 2026 to 9 August 2026; regenerated whenever the logs are.