AI crawler checker
Paste your robots.txt, name a path, and see which of 18 AI and search crawlers may fetch it.
Findings
Copy it from https://yoursite.com/robots.txt. This page cannot fetch other sites (browsers block that), so paste it. To audit the live site including noindex, llms.txt and body text, run the AdRails aeo_site_audit tool.
Runs entirely in your browser. Nothing you enter is sent anywhere.
Can ChatGPT, Claude and Perplexity read my website?
Only when the crawler that engine actually uses is allowed on the page you care about.
Three gates decide whether an AI answer engine can use your page. The crawler has to be allowed by robots.txt, the URL has to return real HTML, and the engine has to choose you as a source. robots.txt is the gate you can check in ten seconds, and the checker above does exactly that for the 18 crawlers AdRails tracks: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Applebot-Extended, DuckAssistBot, Meta-ExternalAgent, Amazonbot, CCBot and Bytespider. selftest: aeo.crawl_audit
Each engine runs more than one crawler, and every crawler is a separate token in the file. A rule aimed at one does nothing to the others. That is why a site can be fully blocked from training datasets and still be quoted in an answer, or fully open to training and invisible in answers because one search token was disallowed months ago and forgotten.
What the table tells you
- Crawler and owner. The exact token to write in robots.txt and the company behind it, so a GPTBot rule is never mistaken for a Googlebot rule. selftest: aeo.crawl_audit
- Purpose. Each crawler is labelled training, search or user-fetch — the label that decides whether a block costs you answers. selftest: aeo.crawl_audit
- Allowed on your path. The whole rule set applied with RFC 9309 longest-match, not just the first line that matches. selftest: aeo.crawl_audit
How the verdict is decided
The result is good only when every crawler labelled search or user-fetch may fetch the path you typed. Training crawlers are listed in the table but they do not flip the verdict, because a training crawl is not what fetches a page when a person asks a question at that moment.
What is the difference between training, search and user-fetch bots?
Same file, same syntax, three different jobs — and only two of them affect whether you get cited.
robots.txt knows nothing about "AI". It knows user-agent tokens, and each token is operated for a specific job. Splitting the 18 crawlers by job is what turns a wall of rules into a decision.
Training crawlers
Fetch pages to build model weights — GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, Meta-ExternalAgent. Blocking these is a licensing decision about your text. It does not remove a search crawler's or a user-fetch agent's right to fetch the same URL.
Search crawlers
Build the index an assistant retrieves from at answer time — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, DuckAssistBot. Disallow these and the engine has nothing of yours to quote from.
User-fetch agents
Fetch one page because a person asked about it just now — ChatGPT-User, Claude-User, Perplexity-User. They arrive holding a specific URL, so a block here breaks the exact link a user clicked.
The checker labels every crawler from its own table and shows the label in the result above, so you never have to guess which column a token belongs to. selftest: aeo.crawl_audit AdRails treats all three identically when it applies the rules — a Disallow is a Disallow. The distinction is a policy one: it separates the blocks that change what your text is used for from the blocks that remove you from answers.
Should I block GPTBot?
A licensing decision about training data, not an SEO decision.
Blocking GPTBot is a choice about whether your text trains a model. GPTBot is OpenAI's training crawler. It is not the token that fetches your page when a ChatGPT user asks about your product — that is OAI-SearchBot for retrieval and ChatGPT-User for a live fetch. Disallow GPTBot and leave those two allowed and your pages can still be retrieved and cited.
Three common patterns, in plain English. Pick one on purpose rather than by accident.
Allow everything
One User-agent: * group with an Allow: / line, plus a sitemap. Nothing to maintain, no risk of blocking an engine you wanted. The trade-off is that your text is also training data.
Training off, answers on
Disallow the training tokens — GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended — and keep the search and user-fetch tokens open. Your pages stay retrievable in answers while the training crawl is refused.
Block everything
A blanket Disallow: / for all 18 tokens removes your site from AI answers wherever the engine honours robots.txt, and from training too. The checker will show every search and user-fetch crawler red, which is the point.
One detail worth knowing: Google-Extended is a separate token from Googlebot. Disallowing Google-Extended opts out of Google's AI training use without touching Google Search crawling; disallowing Googlebot takes you out of Search as well. Test both paths in the checker before you commit the file. selftest: aeo.crawl_audit
How do robots.txt rules decide whether a bot may fetch a page?
Longest match wins. Allow wins ties. No Disallow line means everything is allowed.
robots.txt is read in groups. A group starts with one or more User-agent lines and continues with Allow and Disallow lines until the next user-agent line. A crawler uses the groups whose user-agent matches its token, plus the * group when no specific group exists for it.
Inside the matching groups, RFC 9309 applies the longest match. Every Allow and Disallow pattern that matches the path is a candidate; the longest matching pattern wins. If an Allow and a Disallow pattern are exactly the same length, Allow wins. First-match does not exist in the standard, and that is where hand-written files usually go wrong.
Take a file that disallows /blog/ for everyone but allows /blog/public/. For the path /blog/public/post both patterns match, but the Allow pattern is longer, so the page is allowed — for every crawler that inherits the * group. The checker reports it as allowed for all 18. selftest: aeo.crawl_audit
Three more rules decide the odd cases. A group with no Disallow lines allows everything. A missing robots.txt allows everything, and so does an empty file. And a robots.txt rule only binds crawlers that choose to respect it — it is a request, not a lock, which is why the checker also reports what it found rather than only yes or no.
What else stops AI answers from citing me?
An allowed crawler is necessary, not sufficient.
The crawl audit in AdRails scores a site from 0 to 100 across every gate, not just robots.txt: the 18-crawler rules, llms.txt, sitemap, noindex and nosnippet directives, canonical tags, and how much body text the page actually serves. selftest: aeo.crawl_audit
- noindex and nosnippet. A noindex directive drops the page from an index even when robots.txt permits the crawl; nosnippet removes the text an engine would quote. Both are checked separately from the crawl rules. selftest: aeo.crawl_audit
- JavaScript-only content. A "body text only" check measures what the page serves without running scripts. If the answer lives in client-side rendering, an extractor sees an empty shell. selftest: aeo.crawl_audit
- No llms.txt. The validator checks a
/llms.txtagainst the llmstxt.org format, and the builder returns a file that passes its own validation. selftest: aeo.llms_txt - Missing or broken sitemap. The audit reports the sitemap lines it finds in robots.txt alongside the rule results, so a stale URL surfaces next to a blocked crawler. selftest: aeo.crawl_audit
- A page that never answers the question. Answer-readiness is a separate score: answer-first paragraph, question headings, FAQ, structured data, specifics, freshness and citations. selftest: aeo.page_audit
Fix robots.txt first. It is a single file, it takes a minute, and every other improvement is wasted while the crawler is refused.
How do I run this check on the live site instead of pasting?
A browser cannot read another origin's robots.txt. The AdRails MCP server can.
AdRails is an open-source (Apache-2.0) MCP server that runs on your machine with your credentials. It fetches the files itself through an SSRF-safe fetcher that resolves hosts and refuses private, loopback, link-local and reserved addresses on every redirect hop. selftest: aeo_tools
aeo_site_audit returns the same 18-crawler table for a live domain, plus llms.txt, sitemap, noindex/nosnippet, canonical and body-text findings and a 0-100 score. selftest: aeo.crawl_audit Around it, the AEO toolkit is read-only except IndexNow submit: aeo_page_audit for answer-readiness, aeo_llms_txt to validate or build /llms.txt, aeo_schema_generate for JSON-LD that passes its own audit, aeo_visibility_record for share of answer, and aeo_ai_traffic to split real AI referrals from spoofed ones in your own server logs. selftest: aeo.visibility selftest: aeo.ai_traffic
Read the AEO toolkit overview for how the pieces fit together, or browse the full tool reference. The source is on GitHub.
Straight answers.
Can ChatGPT read my website if robots.txt blocks GPTBot?
Only if a different crawler is still allowed. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, and ChatGPT-User fetches a page when a person asks about a URL. A rule that disallows GPTBot alone leaves those tokens untouched, so your pages can still be retrieved and cited in answers.
Should I block GPTBot?
That is a licensing choice about training data, not an SEO one. Blocking GPTBot stops OpenAI's training crawl but does not hide you from ChatGPT answers while OAI-SearchBot and ChatGPT-User are allowed. Set the rule deliberately, paste the file into the checker above, and do not try to express two different policies with a single User-agent line.
How does robots.txt decide between an Allow and a Disallow line?
RFC 9309 uses the longest match, not the first one. For a path, the crawler collects every Allow and Disallow pattern in its matching user-agent groups and keeps the longest pattern that matches. If an Allow and a Disallow pattern are the same length, Allow wins. A group with no Disallow lines allows everything.
What is the difference between training, search and user-fetch crawlers?
Training crawlers fetch pages to build model weights. Search crawlers build the index an assistant retrieves from when someone asks a question. User-fetch agents grab a single page because a person asked about that URL just now. The three use different tokens, so you can opt out of training while staying visible in answers.
Why can't this page fetch my robots.txt automatically?
Browsers block a page from reading another site's files, so a checker running on adrails.dev cannot pull your robots.txt without a proxy. Pasting it keeps the whole check inside your browser and nothing you enter leaves the page. For a live fetch that also tests noindex, llms.txt and body text, run the AdRails aeo_site_audit tool.
What else can stop AI answers from citing my page?
robots.txt is only the first gate. A noindex or nosnippet directive drops the page from an index, content rendered only by JavaScript leaves little body text to quote, a missing sitemap slows discovery, and a canonical pointing elsewhere hands the credit to another URL. aeo_site_audit scores all of these together from 0 to 100.