Orkkid
Free tool · AI search visibility

Robots.txt checker: is your site blocking AI crawlers?

This AI crawler checker fetches your robots.txt and tests it against 12 AI user agents, from GPTBot to PerplexityBot. See exactly which engines can read your site and the robots.txt rule that blocks the ones that can't.
12 AI crawlers
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more.
Real robots.txt
Fetched live and parsed to Google's spec, wildcards included.
Exact rule shown
See the group and line that blocks each bot.

How the robots.txt checker tests AI crawlers

We fetch your robots.txt and parse it the way Google does: user-agent groups, Allow and Disallow with longest-match precedence, and the*and$wildcards. Each of the 12 AI user agents is tested against your homepage path, falling back to the wildcard group when it has no group of its own. "Partial" means the homepage is allowed but some paths are not. It is a robots.txt checker built for AI search: every AI crawler side by side, with the exact rule behind each verdict. Nothing you enter is stored.

Crawler access is step one. Read how ChatGPT recommends businesses or see how it fits into a generative engine optimization plan.

Questions

AI Crawler Checker: common questions

What is GPTBot?

GPTBot is OpenAI's web crawler. It collects public pages that may be used to train OpenAI's models, and it obeys robots.txt. OpenAI runs two other agents: OAI-SearchBot, which indexes pages for ChatGPT search, and ChatGPT-User, which fetches a page when a user asks. Blocking OAI-SearchBot keeps you out of ChatGPT search results.

Which AI crawlers does the checker test?

GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini training), PerplexityBot, CCBot (Common Crawl), Applebot-Extended, Meta-ExternalAgent, Bytespider (TikTok) and DuckAssistBot.

Should I block AI crawlers?

If you want customers to find you through ChatGPT, Perplexity or Gemini, no. Blocking GPTBot or OAI-SearchBot removes you from ChatGPT search results. Many hosts and security plugins add these blocks by default, which is why this check exists.

My robots.txt has no AI rules. Am I blocked?

No. With no matching User-agent group and no wildcard Disallow, every crawler is allowed. The checker shows 'allowed' in that case. Blocks usually come from a 'User-agent: *' group with 'Disallow: /', or from a Cloudflare or WAF setting that never appears in robots.txt.

The checker says allowed but ChatGPT still can't read my page. Why?

robots.txt is only the first gate. Cloudflare Bot Fight Mode, a WAF, IP blocking, or a JavaScript-only page can still stop the crawler. Run the AI Visibility Audit to see whether your HTML is readable without JavaScript.

Related free tools

Browse all free GEO tools

Robots.txt Generator

Free robots.txt generator: build a valid robots.txt file in seconds, toggle GPTBot, ClaudeBot, PerplexityBot and other AI crawlers, and add your sitemap.

AI Visibility Audit

Free AI visibility checker: audit any site's AI search readiness (crawler access, llms.txt, schema, citability) and get a letter grade in 10 seconds.

llms.txt Generator

Free llms.txt generator: crawl your sitemap, pull real page titles and descriptions, and download a working llms.txt file for your site root in seconds.

Be the business AI recommends.

The free GEO audit runs this check and the rest on your site and your competitors, then shows what is stopping ChatGPT and Google from sending you customers.

Get my free GEO auditBook a call with Ned