Orkkid
Free tool · AI search visibility

Build a robots.txt that AI can read.

Pick what search engines and AI crawlers may access, add your sitemap, and copy a valid robots.txt. Presets for allowing or blocking AI training bots so you don't have to remember the user agents.

1. Pick a starting point

2. Crawler access

Search engines

Search engines

Never block these unless you want the site out of Google and Bing.

  • Googlebot

    Google · Google Search

  • Bingbot

    Microsoft · Bing and Copilot

AI search

AI search

These fetch pages to answer questions. Blocking them removes you from AI recommendations.

  • OAI-SearchBot

    OpenAI · ChatGPT search

  • ChatGPT-User

    OpenAI · ChatGPT browsing

  • PerplexityBot

    Perplexity · Perplexity answers

  • DuckAssistBot

    DuckDuckGo · DuckAssist

AI training

AI training

Used to train models. Blocking these does not affect Google Search rankings.

  • GPTBot

    OpenAI · GPT training

  • ClaudeBot

    Anthropic · Claude crawler

  • anthropic-ai

    Anthropic · Legacy crawler

  • Google-Extended

    Google · Gemini training

  • CCBot

    Common Crawl · Open dataset

  • Applebot-Extended

    Apple · Apple Intelligence

  • Meta-ExternalAgent

    Meta · Llama training

  • Bytespider

    ByteDance · TikTok / Doubao

3. Paths to keep out of search (applies to all crawlers)

4. Sitemap URL

Don't have one? Generate a sitemap.xml. Enter just the domain and we assume /sitemap.xml.

How it works

Every crawler you block gets its own group withDisallow: /. Everything else falls through to theUser-agent: *group, which carries the paths you ticked, and the Sitemap line goes last. That is the exact structure Google, Bing and the AI engines document, so there is nothing to interpret. The file is built in your browser; nothing is sent to us.

Check the result with the AI crawler checker. Pair it with an llms.txt file so AI engines know both what they may crawl and what is worth reading.

Questions

Robots.txt Generator: common questions

What should a basic robots.txt contain?

A 'User-agent: *' group with any paths you want kept out of search (admin, cart, internal search), and a 'Sitemap:' line pointing at your XML sitemap. Everything else is allowed by default. Never disallow '/' unless you want the whole site out of Google.

Can I allow ChatGPT search but block AI training?

Yes. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Allow OAI-SearchBot and ChatGPT-User, disallow GPTBot. Google-Extended controls Gemini training without affecting Google Search. The generator has a preset for exactly this.

Where does robots.txt go?

At the root of the domain: https://yourdomain.com/robots.txt. It must be plain text, served with a 200 status. Subfolders are ignored by crawlers.

Will robots.txt remove a page from Google?

Not reliably. Disallow stops crawling, but a page can still be indexed from links. To remove a page use a 'noindex' meta tag or an HTTP header and keep it crawlable so Google can see the tag.

Related free tools

Browse all free GEO tools

AI Crawler Checker

Check whether your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 8 other AI crawlers, with the exact rule behind each verdict.

llms.txt Generator

Generate a working llms.txt from your sitemap in seconds. It crawls your pages, pulls real titles and descriptions, and outputs a file for your site root.

AI Visibility Audit

Real audit of any website's AI search readiness. Crawler access, llms.txt, schema, sitemap, on-page citability, security. Letter grade in 10 seconds.

Be the business AI recommends.

The free GEO audit runs this check and the rest on your site and your competitors, then shows what is stopping ChatGPT and Google from sending you customers.

Get my free GEO auditBook a call with Ned