Build a robots.txt that AI can read.
Pick what search engines and AI crawlers may access, add your sitemap, and copy a valid robots.txt. Presets for allowing or blocking AI training bots so you don't have to remember the user agents.
1. Pick a starting point
2. Crawler access
3. Paths to keep out of search (applies to all crawlers)
4. Sitemap URL
Don't have one? Generate a sitemap.xml. Enter just the domain and we assume /sitemap.xml.
How it works
Every crawler you block gets its own group withDisallow: /. Everything else falls through to theUser-agent: *group, which carries the paths you ticked, and the Sitemap line goes last. That is the exact structure Google, Bing and the AI engines document, so there is nothing to interpret. The file is built in your browser; nothing is sent to us.
robots.txt is one item on the technical SEO audit checklist. Pair it with an llms.txt file so AI engines know both what they may crawl and what is worth reading.
Robots.txt Generator: common questions
What should a basic robots.txt contain?
A 'User-agent: *' group with any paths you want kept out of search (admin, cart, internal search), and a 'Sitemap:' line pointing at your XML sitemap. Everything else is allowed by default. Never disallow '/' unless you want the whole site out of Google.
Can I allow ChatGPT search but block AI training?
Yes. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search. Allow OAI-SearchBot and ChatGPT-User, disallow GPTBot. Google-Extended controls Gemini training without affecting Google Search. The generator has a preset for exactly this.
Where does robots.txt go?
At the root of the domain: https://yourdomain.com/robots.txt. It must be plain text, served with a 200 status. Subfolders are ignored by crawlers.
Will robots.txt remove a page from Google?
Not reliably. Disallow stops crawling, but a page can still be indexed from links. To remove a page use a 'noindex' meta tag or an HTTP header and keep it crawlable so Google can see the tag.
Related free tools
AI Crawler Checker
Check whether your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 8 other AI crawlers. Instant per-bot verdict with the exact rule that blocks each one.
XML Sitemap Generator
Enter your homepage and get a valid sitemap.xml. Crawls internal links up to 200 pages, skips noindex and non-HTML URLs, and outputs a file ready for Search Console.
llms.txt Generator
Generate a working llms.txt from your sitemap in seconds. Crawls your pages, extracts real titles and descriptions, and outputs a file ready for your site root.