AI Crawlability Lens

Fetch your page exactly as GPTBot, PerplexityBot and ClaudeBot see it — raw and unrendered. 100% local, no account, no server.

Googlebot renders your JavaScript. GPTBot, PerplexityBot and ClaudeBot mostly do not. Your page can rank in Google, pass every SEO audit, and still be invisible to AI answer engines — because they read only the raw HTML your server returns. If your content is drawn by JavaScript after load, they see an empty shell. AI Crawlability Lens fetches the page you are on as each crawler, using that crawler's real User-Agent, and shows you exactly which content never reaches it. WHAT IT CHECKS Per-crawler visibility. Your page is fetched nine times — once as each major search and AI crawler, and once with an ordinary browser User-Agent as a baseline — and each raw response is compared against what you actually see on screen. robots.txt. Every crawler is evaluated against your rules using proper group precedence, including the AI-usage tokens that are separate from the search crawler they appear related to. Allowing a search engine to crawl your site does not always allow it to use your content in AI answers, and that distinction surprises almost everyone. Server-level blocking. A refusal aimed at one crawler is reported separately from a low score, because the fix is completely different. Cloaking. Whether your infrastructure quietly serves different HTML to different crawlers. Framework-aware advice. The recommendation is matched to the stack the page is actually built on, so you get the specific thing to change rather than generic guidance. WHAT YOU GET A visibility score out of 100 for each crawler, with headings weighted double. A plain-English verdict for every crawler and the reasoning behind it: blocked in robots.txt, refused by your server, or simply absent from the HTML. An on-page overlay that outlines every invisible block in red. Switch crawlers and the map redraws. It draws over the page and never modifies it. A self-contained HTML report to hand to a developer — one file, no external references, and it makes no network requests when opened. A local history of your last 20 checks. PRIVACY Everything runs locally. There is no account, no server and no telemetry. The only requests made are to the page you are on and its robots.txt, and only when you click Check. Site access is requested per-site at the moment you click, never for all sites at install time, and you can revoke it whenever you like. No page content is ever stored or transmitted. HONEST LIMITS Requests come from your own IP address, not from a crawler's published address range. So the robots.txt findings and the question of whether your content exists in the raw HTML are exact — but treat a server refusal as a lead to confirm in your access logs, not a conclusion. Built for search and content professionals who need to know whether their pages can be cited by AI answer engines. That is a different question from whether a page is server-rendered, and a more urgent one.