RobotSense Crawl Intelligence

Verify crawl and index control — with real bot intelligence.

Instantly verify robots.txt, meta robots, X-Robots-Tag headers, canonical tags, and AI bot access for any page you visit. RobotSense extension puts real-time crawl and index diagnostics right inside your browser. Open the side panel on any page to instantly see how search engines and AI crawlers interpret your directives — no switching to external tools, no copy-pasting URLs. 🔍 WHAT IT CHECKS • Robots.txt — Fetches and parses the site's robots.txt, then shows whether a specific bot (Googlebot, Bingbot, etc.) is allowed or blocked for the current URL. Wildcard and path-matching rules are fully supported. • Meta Robots Tags — Reads <meta name="robots"> and bot-specific meta tags directly from the live DOM and reports every directive (noindex, nofollow, noarchive, max-snippet, and more). • X-Robots-Tag HTTP Headers — Intercepts real HTTP response headers so you see the exact X-Robots-Tag values the server sends — including ones hidden from JavaScript by CORS. • Canonical Tags — Detects the page's canonical URL, flags self-referencing vs. cross-domain canonicals, warns about mismatches, and checks for multiple conflicting tags. • AI Bot Access — Checks robots.txt rules for 14 major AI crawlers (including GPTBot, ClaudeBot, and Anthropic-AI) and reports how many are allowed vs. blocked. 📊 PAGE SUMMARY DASHBOARD Every analysis opens with an at-a-glance dashboard: – Indexability verdict (Indexable / Not Indexable / Conditional) – HTTP status code (captured from the real network response) – Current URL and Canonical URL (with one-click copy) – Word count and page language – Alert count highlighting potential SEO issues – AI Bots allowed/blocked summary 🎯 TOOLBAR ICON The toolbar icon changes color automatically: – Green: page is indexable – Amber: conditional issues detected – Red: page is not indexable – Default: no data yet / restricted page 🤖 USER-AGENT SIMULATION Switch between Googlebot, Bingbot, Yahoo Slurp, or enter any custom user-agent string to check how different bots see the page. Set your preferred default in Settings. ⚙️ CONFIGURABLE SETTINGS – Default user-agent for analysis – Cache timeout (how long intercepted data is retained) – Custom user-agent list (add, edit, remove) – Toggle word count and language display 👤 BUILT FOR SEOs, web developers, content strategists, and anyone who needs to verify that crawl and index directives are working correctly — without leaving the page. 🔒 PRIVACY RobotSense works entirely in your browser. No data is sent to external servers. All analysis happens locally using the page's own robots.txt, meta tags, and HTTP headers. What's new in 1.3.0 More accurate word count – Word count now matches the text actually displayed on the page, including on modern sites with compressed HTML. – New: see how many words are hidden on the page (inactive tabs, collapsed sections, hidden menus). Search engines can still read this text, so a large hidden count is worth checking. – Better support for languages written without spaces, such as Chinese, Japanese and Thai. Smarter indexability checks – The toolbar icon and the side panel now always agree. – Bot-specific rules are respected: a noindex aimed at one crawler no longer marks the page as blocked for others. – New warning when robots.txt returns a server error, which can stop search engines crawling the whole site. – Pages whose canonical points to another URL are now flagged as "Canonicalised". – More accurate robots.txt results for specialised crawlers such as Googlebot-News. – Hover over the status or Alerts badge to see every issue found. AI bot access, by purpose – Now checks 20 AI crawlers, grouped into AI training, AI search and user-triggered assistants, including OAI-SearchBot, Claude-SearchBot, Claude-User, Perplexity-User and DuckAssistBot. – See at a glance whether you block AI training while staying visible in AI search. – Each bot is checked against robots.txt, meta robots tags and X-Robots-Tag headers. – Expand the list to see every bot's status and the rule behind it. Redirect insights – See the redirect chain that led to the page, with temporary redirects, HTTPS upgrades and JavaScript redirects labelled. – New warning when the page's canonical URL redirects or returns an error. – New warning when a page redirects visitors with a meta refresh. Works better on modern web apps – The panel updates automatically when you move between pages in single-page apps. – HTTP status is now shown for pages restored with the back button or opened before the extension was installed. See what crawlers get without JavaScript – Most AI crawlers never run JavaScript. On pages where JavaScript changes what they receive, RobotSense now shows exactly what differs from the HTML the server sends. – Catches noindex tags that JavaScript adds or removes, including the case where a page looks indexable but search engines treat it as noindex. – Flags canonicals and titles changed by JavaScript, and content that only appears after JavaScript runs. – Only appears when something differs. On most sites nothing changes. Fixes – Settings are no longer reset when the extension updates. – Faster analysis on sites with many AI-crawler rules. – Links with #anchors no longer trigger false canonical warnings. – General stability and security improvements.