AI assistants can only cite content their crawlers can fetch — and one robots.txt line can silently shut them all out. Run a free AI crawler check: enter your website and instantly see which of 14 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) your robots.txt allows, blocks, or partially restricts, plus whether your llms.txt and sitemap are in place.
The checker takes your URL, resolves the host, and fetches your live public robots.txt — the same file every crawler reads before touching your pages. Nothing is installed and nothing on your site changes.
Rules are grouped by user-agent the way crawlers apply them: the most specific matching group wins, longest rule takes precedence, and Allow beats Disallow on ties. No robots.txt at all simply means every crawler is allowed.
Each crawler — training bots like GPTBot and ClaudeBot, search indexers like OAI-SearchBot and PerplexityBot, and live-browsing agents like ChatGPT-User — gets an Allowed, Blocked, or Partial verdict, with the exact robots.txt rule that caused it.
You get a per-crawler verdict list plus two extra checks: whether you publish an llms.txt file and whether a sitemap is declared or reachable. From there, the next question is whether AI actually mentions you — which the free AI visibility check answers.
All 14 major AI crawlers with an Allowed, Blocked, or Partial verdict each — and what each one powers, so you know whether a block costs you training presence, AI search results, or live browsing.
Blocked or Partial verdicts show the matching robots.txt line, so fixing an unintended block is a one-line edit instead of a guessing game.
Whether your site publishes an llms.txt — the plain-text file that gives AI assistants a curated guide to what your site is about and which pages matter most.
Whether a sitemap is declared in your robots.txt or reachable at /sitemap.xml, so crawlers — including AI search bots — can discover every page you want indexed.
Run the check above to find out in seconds. GPTBot is OpenAI’s training crawler, and it obeys robots.txt — so a "User-agent: GPTBot / Disallow: /" group, or a blanket "User-agent: * / Disallow: /", keeps it out. The report shows GPTBot’s verdict and the exact rule that produced it.
14 of the major ones: GPTBot, OAI-SearchBot, and ChatGPT-User (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot and Perplexity-User, Google-Extended and Googlebot, Bingbot, CCBot (Common Crawl), Meta-ExternalAgent, Applebot-Extended, and Bytespider (ByteDance).
It depends on which crawler. Blocking training bots like GPTBot or CCBot keeps your content out of future model training — but blocking search and browsing crawlers like OAI-SearchBot, PerplexityBot, or ChatGPT-User means AI assistants can’t cite or recommend you when your buyers ask questions. Many sites unintentionally block everything with one wildcard rule; the report shows exactly which category each verdict affects.
Your robots.txt lets that crawler reach your homepage but disallows some real paths — the report shows the widest matching Disallow rule. That’s often deliberate (keeping crawlers out of /admin or /cart), but it’s worth confirming the blocked paths aren’t content you want AI assistants to see.
No — it reads your public robots.txt, llms.txt, and sitemap, which covers the standard, voluntary crawl rules. It does not detect server-side or network-level blocking such as Cloudflare bot rules, WAF filters, or user-agent blocks in your server config. A crawler can be "Allowed" in robots.txt and still be turned away at the door by those layers, so treat this as the robots.txt half of the picture.
llms.txt is a proposed convention: a plain-text file at /llms.txt that gives AI assistants a curated summary of your site and its most important pages. It’s early and adoption is still growing, so it isn’t required — but it’s a low-effort way to guide how AI systems read your site, and the checker tells you whether yours exists.
This check is one signal. The free audit grades your whole site for AI search and shows the plan Magister would run to fix what it finds.