Free tool, no sign-up
AI crawler check
Enter a domain. We read its robots.txt and show how 16 AI and search crawlers are treated.
How to read the result
- Allowed
- robots.txt names the crawler and does not block the home page.
- Blocked
- A rule disallows the home page for that crawler.
- Not mentioned
- No group names the crawler, so it follows the * group. We show whether that group blocks it.
- Some paths restricted
- The home page is open, but a Disallow rule covers other paths.
- Which rule wins
- The longest matching rule wins, and Allow beats Disallow on a tie, as RFC 9309 specifies.
We test the home page path. Crawlers that are named in robots.txt are labelled Allowed or Blocked, and the rule line that decided it is shown with its line number.
What to change
To appear in AI answers, let the search and answer crawlers in: OAI-SearchBot, PerplexityBot, Googlebot and Bingbot. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers.
Training crawlers are a separate decision. Disallowing GPTBot, ClaudeBot or Google-Extended asks those vendors not to train on your pages. Search-based answers use the retrieval crawlers.
# Let answer engines read the site
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of model training
User-agent: GPTBot
Disallow: /User-triggered fetchers such as ChatGPT-User and Perplexity-User open a page when a person’s request needs it. Their vendors say robots.txt rules may not apply to them, so a Disallow line is a request, not a lock. A firewall or bot protection can also block crawlers that robots.txt allows. After you fix access, publish an llms.txt to give agents a clear starting point.
Questions about the check
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of it are not shown in ChatGPT search answers.
What if my site has no robots.txt?
Crawlers treat a missing file as permission to fetch everything, so every crawler counts as allowed. The result says so.
robots.txt allows everything, so why am I not cited?
Crawl permission is only the first step. Pages also need to be indexed, relevant to the question and worth citing, and a firewall or CDN setting can block crawlers whatever robots.txt says. The scan shows what the engines actually answer.
Do you store the domains I check?
No. We fetch the public files, return the result and keep only a daily counter keyed by a hash of your IP address. The limit is 60 checks a day.
Which crawlers does the check cover?
16 of them, taken from each vendor’s documentation. Newer tokens such as Claude-User, Claude-SearchBot and Amzn-User are not in the list yet, and Claude-Web and anthropic-ai are older Anthropic names that many sites still list.
Allowed is only the first step.
Run a free scan to see whether Perplexity and Claude actually mention your brand.
Scan my brand