Public pages only. Anything behind a login, on a private network, or not reachable over HTTP is refused.
What this checks, and what it doesn't
It checks: whether the page answers a crawler at all, whether it carries a noindex or nofollow directive, whether robots.txt permits each of ten named AI crawlers on that path, where the canonical points, and whether the title, headings, body text and JSON-LD can be read from the HTML your server sends.
It does not check whether your content is any good, whether you rank, or whether an AI has actually cited you. Those are worth knowing and this is not the tool for them. This answers the narrower question that has to be true first: could a crawler read this page if it wanted to?
Every finding shows the evidence it came from, so you can disagree with it.
Why there's no score
A number out of a hundred hides the only thing that matters: which specific problem, and where. A page with one noindex tag and a page with a slightly short meta description could score the same, and only one of them is invisible.
So findings are ranked blocker, warning, info and pass. A blocker means an AI crawler cannot use this page at all. Everything else is a matter of degree.
Blocking AI crawlers is a legitimate choice. Plenty of publishers do it deliberately. When this tool finds crawlers blocked it says so as a fact, not a failure — the point is that it should be a choice you made rather than one you inherited.
Questions
What does this actually check?
Whether an AI crawler can reach the page at all, whether the page tells crawlers not to index it, whether robots.txt permits each of ten named AI crawlers on that path, and whether the main content and structured data can be read from the HTML the server sends. It is a technical diagnostic, not a content or ranking review.
Do I need an llms.txt file?
Reality check: no. No major AI crawler is documented as reading llms.txt. It costs little to publish and proves nothing. This tool reports whether you have one as information only, and never counts its absence as a problem. Be wary of anyone selling it as required.
Does blocking AI crawlers hurt my search ranking?
Blocking GPTBot, ClaudeBot or Google-Extended does not affect Google Search ranking. Google-Extended controls Gemini grounding, not Search. Blocking Googlebot or Bingbot is a different matter and does affect search — those two are listed separately here for that reason.
Why does it say my page returned 403 when it loads fine in my browser?
Many CDNs and bot-protection services refuse requests from user agents they do not recognise. A human browser gets the page; an unknown crawler gets a 403. That is worth knowing, because AI crawlers are treated the same way. If you see this, check your CDN or WAF bot rules. Everything else in the report was then read from the error response rather than your real page.
What is the difference between noindex and robots.txt?
robots.txt asks a crawler not to fetch a URL. A noindex directive lets it fetch the page but asks it not to keep it. They are independent, and a page can pass one and fail the other. Worth knowing: multiple robots meta tags are legal, and crawlers combine them by taking the most restrictive value. A permissive first tag does not cancel a restrictive later one, which is exactly how a noindex gets missed.
Is the URL I scan stored anywhere?
The URL is sent to a Cloudflare Worker that fetches it and returns the result. The Worker has no database attached and stores nothing. Cloudflare keeps ordinary request logs as it does for any site. This page also loads ads and analytics like the rest of StashGrid, so it is not a no-network tool and does not claim to be.
Can I scan a page behind a login?
No. The scanner fetches the URL as an anonymous crawler, which is the whole point — that is what an AI crawler sees. Private networks, IP addresses and non-HTTP schemes are refused outright.