AI Crawler Check

Instantly see which AI crawlers are blocked or allowed on any website's robots.txt, with one click.

AI Crawler Check lets you instantly see which AI crawlers are allowed or blocked on any website, with a single click. Click the extension icon while browsing any site, and get an instant report: • Which AI crawlers are allowed, blocked, or partially blocked in that site's robots.txt — covering the crawlers used by major AI companies for training and search, like OpenAI's GPTBot and Anthropic's ClaudeBot • Whether the page has a meta robots tag opting out of AI training • Whether the site publishes an llms.txt file Why it matters: more AI assistants and search tools crawl the web to answer questions or train models. Site owners can choose to allow or block each one individually in robots.txt — but checking this manually means reading long, technical files. AI Crawler Check does it for you, in one click. Built for SEO professionals, website owners, developers, and anyone curious whether their content is accessible to AI crawlers. No sign-up, no tracking, no data collection — everything runs locally in your browser.