comiza

Agentwatch › Crawlers › bingbot

bingbot

bingbot is how Microsoft builds the index that Bing and Copilot answers from. When it cannot reach your pages, they are not in that index, and an assistant cannot cite a page it has never read. This is the crawler people block by accident while meaning to block something else.

Run byMicrosoft
robots.txt tokenbingbot
Purposefeeds answers
Cost of blockingYou disappear from its answers.

What blocking bingbot actually costs

Your pages stop appearing in Bing and Copilot. Not immediately and not visibly: what happens is that the index goes stale and then thin, and the traffic that would have come never arrives. There is no notification, no warning in any console, and nothing in your analytics that says why.

If you want to refuse training but stay quotable, this is precisely the crawler you must keep allowed.

The rules, exactly

To allow it

User-agent: bingbot Allow: /

An explicit Allow is only needed when a broader rule would otherwise catch it. If your robots.txt does not disallow anything, this crawler is already allowed and you need no rule at all.

To block it

User-agent: bingbot Disallow: /

Put it in its own group. A named group replaces the * group entirely for that crawler and inherits nothing from it, which surprises almost everyone.

Three things about robots.txt that catch people out

A crawler obeys exactly one group
It picks the group whose User-agent value is the longest one that matches its name, and ignores every other group, including *. If you write a rule under * and a separate group for bingbot, the rules under * do not apply to it at all.
Matching is by prefix, not by exact name
A group headed User-agent: Google matches Googlebot, and a group headed with a partial name matches more than you intended. Write the full token.
A server error on robots.txt blocks everything
If /robots.txt returns a 5xx, the documented behaviour is that crawlers stop crawling the whole site until it recovers. A missing file returning 404 is safe; a broken one is not.

Why your robots.txt may say yes while Bingbot still gets nothing

robots.txt is a request. A firewall is not. Bot protection at your CDN answers before your site does, and it has never read your robots.txt.

Every one of these crawlers arrives from a data centre, which is exactly what bot rules are tuned to refuse. The result is a site with a perfectly permissive robots.txt sitting behind a wall, and no checker that only reads robots.txt can see it, because it never makes the request.

The free scan on this site asks for your page as Bingbot, from a real server, and compares the answer against what an ordinary browser gets. If they differ, you have found the wall.

Check your site Free, no account, no email.

Making sure it is really them

A User-Agent is a claim, not proof. Anyone can send any name, so a rule that trusts the name alone can be walked straight through.

Microsoft's method is reverse DNS, then a forward lookup back to the same address. The page below walks through it.

Official documentation: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26

Every crawler in the catalogue