AI Robots.txt Generator
Choose which AI crawlers can read your site. You get a robots.txt that keeps you in AI answers, with training blocked if you want.
AI search
Keep these allowed to be cited in answers.
AI training
Your call. Blocking these doesn't change AI answers.
# AI search crawlers: they fetch pages to answer questions, and cite them. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Googlebot User-agent: Bingbot Allow: / # Everyone else. User-agent: * Allow: /
Publish it at yoursite.com/robots.txt, then run the AI Crawler Checker to confirm.
Guide
Allow AI search, block AI training
Search crawlers and training crawlers
AI companies run separate crawlers. Search crawlers like OAI-SearchBot and Claude-SearchBot fetch pages to answer a question now, and cite them. Training crawlers like GPTBot and ClaudeBot collect data for future models. You can allow the first and block the second, and your AI search visibility stays the same.
Why private paths repeat in each group
A crawler follows only the group of rules that names it, and ignores the rules under User-agent: * once it has its own. So a path you want kept out has to be listed in every group. The generator does that for you.
Before you publish
robots.txt lives at the root of your domain, like yoursite.com/robots.txt, and covers only that host, so each subdomain needs its own. Once it's live, run the AI Crawler Checker to confirm every crawler reads it the way you meant.
Questions
Good to know.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls whether Google uses your content for Gemini training. AI Overviews and AI Mode use Googlebot, like the rest of Google Search.
Is robots.txt enforced?
It's a request, not a lock. The major AI companies say their crawlers follow it, but anything truly private belongs behind a login.
Should I block training crawlers?
It's a business choice. Blocking them keeps your content out of future models' training data without changing AI search answers. Allowing them can help models know your brand before they search.
See who AI recommends instead of you.
Citepoint finds the buyer questions you're losing, writes the pages that win them, and tracks every engine daily.
Run the free scan