Overview
AI crawler diagnostic tool. See which AI crawlers a site allows or blocks, from its robots.txt, llms.txt and page-level rules.
AI Crawler Diagnostic Tool shows which AI crawlers a website allows or blocks. Open any site, click the icon, and get a result for 16 crawlers, including GPTBot and ClaudeBot, in a couple of seconds. It is an AI crawler checker for the page you already have open. There is nothing to paste, upload or sign up for. WHAT IT CHECKS 1. robots.txt. The extension reads the site's robots.txt and works out the result for each crawler the way the crawler does: it finds the group that names the crawler, falls back to the * group if there is none, then applies the longest matching rule (RFC 9309). 2. llms.txt. It shows whether the site has an llms.txt file. A "page not found" page served at that address does not count as found. 3. Page-level rules. It reads the page's robots meta tag and its X-Robots-Tag header, and flags noindex, nosnippet, noai and noimageai. The last two are labeled non-standard. WHICH CRAWLERS It checks 16 crawlers from ten operators, including OpenAI, Anthropic and Google. They are grouped by what they do: crawlers that collect training data, crawlers that build an AI search index, and agents that fetch a page when a person asks them to. The full list is in the popup. WHAT YOU SEE - Each crawler marked Allowed, Blocked or Partly blocked - A count of each at the top, so you can read the result at a glance - The exact robots.txt line that decided each result, for the site root and for the page you are on - Any other paths that are blocked for that crawler - A warning on the agents that may fetch pages regardless of robots.txt COPY THE REPORT One button copies the full result. Pasted into Google Docs, Word or an email, it arrives as a table. Pasted anywhere else, it arrives as plain text. Use it in an SEO audit, a client report or a ticket for your developer. WHO IT IS FOR - SEOs checking AI crawlability during a technical audit - Site owners who want to confirm that AI search crawlers can reach their pages - Publishers who want to block AI crawlers used for training and check that the block works - Developers testing a robots.txt change before and after it ships COMMON QUESTIONS IT ANSWERS - Is GPTBot blocked on this site? - Can AI search crawlers reach this page? - Did my robots.txt change block the crawlers I meant to block, and only those? - Does this site have an llms.txt file? - Is this page set to noindex or nosnippet? WHAT IT CANNOT SEE This shows what the site declares. It cannot see blocks set at the CDN or firewall level, so a crawler that robots.txt allows may still be blocked by a service such as Cloudflare. It checks the site you are on and the page you have open. It does not crawl the rest of the site. PRIVACY The extension runs entirely in your browser. When you click the icon, it requests robots.txt, llms.txt and the current page's headers from the site you are on. It sends nothing to any other server and collects no data. There are no accounts, no analytics and no ads. Free, from joshblackburn.co.
0 out of 5No ratings
Details
- Version1.0.0
- UpdatedNovember 16, 2026
- Size22.13KiB
- LanguagesEnglish
- DeveloperWebsite
Email
josh@joshblackburn.co - Non-traderThis developer has not identified itself as a trader. For consumers in the European Union, please note that consumer rights do not apply to contracts between you and this developer.
Privacy
This developer declares that your data is
- Not being sold to third parties, outside of the approved use cases
- Not being used or transferred for purposes that are unrelated to the item's core functionality
- Not being used or transferred to determine creditworthiness or for lending purposes
Support
For help with questions, suggestions, or problems, visit the developer's support site