Is your site blocking AI crawlers?
We fetch your robots.txt and test it against 12 AI user agents, from GPTBot to PerplexityBot. See exactly which engines can read your site and the rule that blocks the ones that can't.
How it works
We fetch your robots.txt and parse it the way Google does: user-agent groups, Allow and Disallow with longest-match precedence, and the*and$wildcards. Each of the 12 AI user agents is tested against your homepage path, falling back to the wildcard group when it has no group of its own. "Partial" means the homepage is allowed but some paths are not. Nothing you enter is stored.
Not sure which crawler belongs to which engine? See the AEO & SEO glossary or read how crawler access fits into an answer engine optimisation plan.
AI Crawler Checker: common questions
Which AI crawlers does the checker test?
GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini training), PerplexityBot, CCBot (Common Crawl), Applebot-Extended, Meta-ExternalAgent, Bytespider (TikTok) and DuckAssistBot.
Should I block AI crawlers?
If you want customers to find you through ChatGPT, Perplexity or Gemini, no. Blocking GPTBot or OAI-SearchBot removes you from ChatGPT search results. Many hosts and security plugins add these blocks by default, which is why this check exists.
My robots.txt has no AI rules. Am I blocked?
No. With no matching User-agent group and no wildcard Disallow, every crawler is allowed. The checker shows 'allowed' in that case. Blocks usually come from a 'User-agent: *' group with 'Disallow: /', or from a Cloudflare or WAF setting that never appears in robots.txt.
The checker says allowed but ChatGPT still can't read my page. Why?
robots.txt is only the first gate. Cloudflare Bot Fight Mode, a WAF, IP blocking, or a JavaScript-only page can still stop the crawler. Run the AI Visibility Audit to see whether your HTML is readable without JavaScript.
Related free tools
Robots.txt Generator
Build a correct robots.txt in seconds. Toggle GPTBot, ClaudeBot, PerplexityBot and other AI crawlers, set crawl rules per bot, add your sitemap. Copy and upload.
AI Visibility Audit
Real audit of any website's AI search readiness. Crawler access, llms.txt, schema, sitemap, on-page citability, security. Letter grade in 10 seconds.
llms.txt Generator
Generate a working llms.txt from your sitemap in seconds. Crawls your pages, extracts real titles and descriptions, and outputs a file ready for your site root.