Orkkid
Free tool · AI search visibility

Free robots.txt generator that AI can read.

Pick what search engines and AI crawlers may access, add your sitemap, and copy a valid robots.txt file. The robots.txt file generator has presets for allowing or blocking AI training bots, so you don't have to remember the user agents.

1. Pick a starting point

2. Crawler access

Search engines

Search engines

Never block these unless you want the site out of Google and Bing.

  • Googlebot

    Google · Google Search

  • Bingbot

    Microsoft · Bing and Copilot

AI search

AI search

These fetch pages to answer questions. Blocking them removes you from AI recommendations.

  • OAI-SearchBot

    OpenAI · ChatGPT search

  • ChatGPT-User

    OpenAI · ChatGPT browsing

  • PerplexityBot

    Perplexity · Perplexity answers

  • DuckAssistBot

    DuckDuckGo · DuckAssist

AI training

AI training

Used to train models. Blocking these does not affect Google Search rankings.

  • GPTBot

    OpenAI · GPT training

  • ClaudeBot

    Anthropic · Claude crawler

  • anthropic-ai

    Anthropic · Legacy crawler

  • Google-Extended

    Google · Gemini training

  • CCBot

    Common Crawl · Open dataset

  • Applebot-Extended

    Apple · Apple Intelligence

  • Meta-ExternalAgent

    Meta · Llama training

  • Bytespider

    ByteDance · TikTok / Doubao

3. Paths to keep out of search (applies to all crawlers)

4. Sitemap URL

Don't have one? Generate a sitemap.xml. Enter just the domain and we assume /sitemap.xml.

How the robots.txt generator works

Every crawler you block gets its own group withDisallow: /. Everything else falls through to theUser-agent: *group, which carries the paths you ticked, and the Sitemap line goes last. That is the exact structure Google, Bing and the AI engines document, so there is nothing to interpret. The file is built in your browser; nothing is sent to us.

Check the result with the AI crawler checker. Pair it with an llms.txt file so AI engines know both what they may crawl and what is worth reading.

Questions

Robots.txt Generator: common questions

What should a basic robots.txt contain?

A 'User-agent: *' group with any paths you want kept out of search (admin, cart, internal search), and a 'Sitemap:' line pointing at your XML sitemap. Everything else is allowed by default. Never disallow '/' unless you want the whole site out of Google.

Can I allow ChatGPT search but block AI training?

Yes. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Allow OAI-SearchBot and ChatGPT-User, disallow GPTBot. Google-Extended controls Gemini training without affecting Google Search. The generator has a preset for exactly this.

Where does robots.txt go?

At the root of the domain: https://yourdomain.com/robots.txt. It must be plain text, served with a 200 status. Subfolders are ignored by crawlers.

Will robots.txt remove a page from Google?

Not reliably. Disallow stops crawling, but a page can still be indexed from links. To remove a page use a 'noindex' meta tag or an HTTP header and keep it crawlable so Google can see the tag.

Related free tools

Browse all free GEO tools

AI Crawler Checker

Free robots.txt checker for AI crawlers: test GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 8 more bots and see the exact rule behind each verdict.

llms.txt Generator

Free llms.txt generator: crawl your sitemap, pull real page titles and descriptions, and download a working llms.txt file for your site root in seconds.

AI Visibility Audit

Free AI visibility checker: audit any site's AI search readiness (crawler access, llms.txt, schema, citability) and get a letter grade in 10 seconds.

Be the business AI recommends.

The free GEO audit runs this check and the rest on your site and your competitors, then shows what is stopping ChatGPT and Google from sending you customers.

Get my free GEO auditBook a call with Ned