Overview
Audit which paths a site disallows in robots.txt and how many of them Google has indexed anyway.
This extension fetches the site's robots.txt, extracts the Disallow rules that apply to Googlebot, converts each one into a Google site: query, and shows how many URLs Google keeps in their index under each disallowed pattern. Because: A site can ask Googlebot not to crawl certain paths, but those URLs can still end up in the index. This tool makes it easy to spot where that's happening at scale. Click the icon on any site and the extension: 1. Fetches the site's robots.txt 2. Extracts every Disallow rule that applies to Googlebot 3. Builds a Google site: query for each rule 4. Reports how many URLs Google still has indexed under each disallowed pattern A site can ask Googlebot not to crawl certain paths, but those URLs can still end up in the index. This tool makes it easy to spot where that's happening at scale. FEATURES • Handles every robots.txt pattern: plain prefixes, wildcards, end-of-URL anchors, filetype suffixes, query-string patterns, URL-encoded UTF-8 paths, and Allow exceptions • Tokenization and grouping match Google's open-source robots.txt parser • Allow rules carve out exceptions from Disallows via combined Google queries • 7-day local cache — re-running an audit is instant for cached rules • Sortable table, copy-as-TSV / Markdown (e.g., for client deliverables) • Google CAPTCHA recovery flow for large audits PRIVACY No data collection. No telemetry. No remote logging. The extension only fetches the public robots.txt and runs Google search queries through your normal browser session. All run state and cached counts live on your device only.
Details
- Version1.0.1
- UpdatedMay 1, 2026
- Size35.27KiB
- LanguagesEnglish
- Developerblackvine GmbHWebsite
Bienenweide 1 Eystrup 27324 DEEmail
chrome.dev@blackvine.orgPhone
+49 172 4497717 - TraderThis developer has identified itself as a trader per the definition from the European Union and committed to only offer products or services that comply with EU laws.
- D-U-N-S312832894
Privacy
Robots Disallow Checker has disclosed the following information regarding the collection and usage of your data. More detailed information can be found in the developer's privacy policy.
Robots Disallow Checker handles the following:
This developer declares that your data is
- Not being sold to third parties, outside of the approved use cases
- Not being used or transferred for purposes that are unrelated to the item's core functionality
- Not being used or transferred to determine creditworthiness or for lending purposes
Support
For help with questions, suggestions, or problems, visit the developer's support site
