RobotSense Crawl Intelligence

Verify crawl and index control — with real bot intelligence.

Instantly verify robots.txt, meta robots, X-Robots-Tag headers, canonical tags, and AI bot access for any page you visit. RobotSense extension puts real-time crawl and index diagnostics right inside your browser. Open the side panel on any page to instantly see how search engines and AI crawlers interpret your directives — no switching to external tools, no copy-pasting URLs. 🔍 WHAT IT CHECKS • Robots.txt — Fetches and parses the site's robots.txt, then shows whether a specific bot (Googlebot, Bingbot, etc.) is allowed or blocked for the current URL. Wildcard and path-matching rules are fully supported. • Meta Robots Tags — Reads <meta name="robots"> and bot-specific meta tags directly from the live DOM and reports every directive (noindex, nofollow, noarchive, max-snippet, and more). • X-Robots-Tag HTTP Headers — Intercepts real HTTP response headers so you see the exact X-Robots-Tag values the server sends — including ones hidden from JavaScript by CORS. • Canonical Tags — Detects the page's canonical URL, flags self-referencing vs. cross-domain canonicals, warns about mismatches, and checks for multiple conflicting tags. • AI Bot Access — Checks robots.txt rules for 14 major AI crawlers (including GPTBot, ClaudeBot, and Anthropic-AI) and reports how many are allowed vs. blocked. 📊 PAGE SUMMARY DASHBOARD Every analysis opens with an at-a-glance dashboard: – Indexability verdict (Indexable / Not Indexable / Conditional) – HTTP status code (captured from the real network response) – Current URL and Canonical URL (with one-click copy) – Word count and page language – Alert count highlighting potential SEO issues – AI Bots allowed/blocked summary 🎯 TOOLBAR ICON The toolbar icon changes color automatically: – Green: page is indexable – Amber: conditional issues detected – Red: page is not indexable – Default: no data yet / restricted page 🤖 USER-AGENT SIMULATION Switch between Googlebot, Bingbot, Yahoo Slurp, or enter any custom user-agent string to check how different bots see the page. Set your preferred default in Settings. ⚙️ CONFIGURABLE SETTINGS – Default user-agent for analysis – Cache timeout (how long intercepted data is retained) – Custom user-agent list (add, edit, remove) – Toggle word count and language display 👤 BUILT FOR SEOs, web developers, content strategists, and anyone who needs to verify that crawl and index directives are working correctly — without leaving the page. 🔒 PRIVACY RobotSense works entirely in your browser. No data is sent to external servers. All analysis happens locally using the page's own robots.txt, meta tags, and HTTP headers.