See your site the way an AI assistant sees it

What gets checked

Each of these has silently removed a real site from AI answers, and none of them announce themselves.

Whether robots.txt lets the answer crawlers in

Not one rule but the one that actually applies. A crawler obeys a single group, chosen by the longest matching User-agent value, and it inherits nothing from *. We report the exact line number that decided it, for every crawler separately.

Whether the server serves them once they ask

We request your home page as each crawler, from a real server, and compare against what an ordinary browser gets. Bot protection at the CDN answers before your site does and never reads your robots.txt, so a perfectly permissive file can sit in front of a wall. No robots.txt checker can see this, because it never makes the request.

Whether there is anything to read when they arrive

Crawlers read the document as delivered. They do not run your framework. We count the words present in the HTML itself, and flag the case where the page is an empty container waiting for JavaScript, which to every assistant is a blank page with a title on it.

Whether the page forbids being quoted

noindex keeps a page out entirely. nosnippet and max-snippet:0 let it be indexed but forbid quoting any of it, which amounts to the same silence from an assistant. These arrive by accident, usually from a staging configuration.

Structured data, sitemap, llms.txt, certificate

JSON-LD is the part of a page a machine reads without interpreting anything, and broken JSON-LD is worse than none because you believe it works. A lapsed certificate stops every crawler days before anyone notices the browser warning.

The crawlers, and which ones actually cost you something

The expensive mistake is treating these as one group. Blocking the training crawlers is a legitimate choice that costs no visibility. Blocking the search and live-fetch crawlers removes you from the answers themselves. They are deliberately separate tokens.

The part that does not fit in a one-off check

Nothing above stays true. A deploy changes a header, somebody turns on bot protection, a certificate lapses, a plugin rewrites robots.txt. It breaks quietly and the traffic that never arrives leaves no trace. The scan above tells you where you stand today; the only useful version of it is the one that runs while you are not looking.

For agents

The same scanner is an MCP server, because this is a question an agent should be able to answer for itself before advising anyone about their visibility. No key, no account; the free allowance applies per address.

check_site returns the verdict, the findings with their fixes, and a link to the readable report. list_ai_crawlers returns the table above as data, which is the thing to read before writing a robots.txt on someone's behalf.

Questions people actually ask

Does blocking GPTBot remove me from ChatGPT?

No, and this is the single most common misunderstanding. GPTBot collects text for training. OAI-SearchBot builds the index ChatGPT answers from, and ChatGPT-User fetches your page live at the moment somebody asks about you. You can refuse the first and keep the other two, which is what most publishers now do.

Do I need an llms.txt file?

Google Search says plainly that it does not use it. Claude, Perplexity and coding assistants do fetch it, and Chrome's Lighthouse moved the audit for it into its agentic browsing category in May 2026. Worth having, not worth much on its own, and no substitute for the page being readable.

My robots.txt allows everything and I still get nothing.

Then look at the second layer. Bot protection at your CDN answers before your site does, and it has never read your robots.txt. Every AI crawler arrives from a data centre, which is exactly what those rules are tuned to refuse. The scan above makes the request and shows you the answer.

Will this show me whether ChatGPT mentions my brand?

No, and anything promising that is sampling prompts and extrapolating. This measures something you can verify and act on: whether the machines can reach, read and quote your pages. Nothing you do about rankings matters while the answer to that is no.

Is scanning someone else's site allowed?

We fetch a small number of pages, once, and cache the result. That is the same footprint as any link checker. Daily watching only ever runs against a site whose owner confirmed it by email.