AI Crawler Checker
Enter your domain to see which AI search crawlers your robots.txt lets in. A blocked crawler can't cite you, however good your pages are.
Guide
Can ChatGPT, Claude and Perplexity read your site?
Why crawler access decides AI visibility
ChatGPT, Claude and Perplexity answer with live web search. Each one fetches pages with its own crawler, and each crawler obeys robots.txt. If your robots.txt blocks OAI-SearchBot, ChatGPT search can't read your pages, so it recommends whoever it can read. Many sites block these crawlers by accident, usually through a blanket rule added to stop AI training.
Search crawlers versus training crawlers
Blocking training is a reasonable choice, and it doesn't have to cost you visibility. GPTBot and Google-Extended collect training data. OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot fetch pages to answer a question right now. You can block the first group and allow the second.
What llms.txt and sitemaps add
A sitemap tells every crawler which pages exist and when they changed. llms.txt is a short Markdown index of your most useful pages, written for language models. Neither replaces good pages, but both make them easier to find.
Questions
Good to know.
Which user agent does ChatGPT search use?
OAI-SearchBot for search results and ChatGPT-User when a person asks ChatGPT to open a page. GPTBot is OpenAI's training crawler.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot only collects training data. ChatGPT search results depend on OAI-SearchBot, which you can allow separately.
How do I allow AI search crawlers?
Add a User-agent group for each crawler with Allow: /, or remove the Disallow rule that matches it. The checker shows which rule applies to each one.
See who AI recommends instead of you.
Citepoint finds the buyer questions you're losing, writes the pages that win them, and tracks every engine daily.
Run the free scan