AI Crawler Check

See if a site lets GPTBot, ClaudeBot, PerplexityBot and Google-Extended in, and whether it has an llms.txt.

Is your site blocking ChatGPT or Claude? Click once and see. AI Crawler Check reads the robots.txt of the site you're on and shows, for each major AI and search crawler, whether it's allowed or blocked. It covers the crawlers from OpenAI, Anthropic, Perplexity, Google and others: the ones that collect training data, the ones behind AI search, and the ones that fetch a page when a user asks. For each crawler you see which rule decided it: the bot's own group or the catch-all * group. It also tells you whether the page you're on is blocked even when the rest of the site isn't, and whether the site has an llms.txt file. What it can't see: a CDN or firewall (Cloudflare blocks some AI bots by default) can stop a crawler that robots.txt allows. Check your server logs for that. No account, no tracking, nothing leaves your browser. By Saeed Shahriar: https://saeedseo.com/