AI crawler checker

Who is actually allowed to crawl your site?

Since 2023, a whole new population of crawlers has arrived on the web: training bots like GPTBot (OpenAI) and ClaudeBot (Anthropic) that collect pages to train models, and assistant fetchers like PerplexityBot, OAI-SearchBot, and ChatGPT-User that retrieve your pages live to answer a user's question — and cite you. They all read the same robots.txt file that Googlebot does, but they use different user-agent tokens, so a rule written for one says nothing about the others. A single User-agent: * + Disallow: / copied from an old template can silently remove you from ChatGPT answers, Perplexity citations, and Google at the same time.

This checker fetches your live robots.txt and evaluates it the way crawlers do (RFC 9309: longest user-agent prefix wins, longest path rule wins, allow beats disallow on ties) for 60+ known tokens across six categories — search engines, AI training bots, AI assistants, SEO tools, social preview fetchers, and archives. For every blocked bot you see the exact line responsible, so you can fix or confirm it in seconds. It also checks whether you publish an llms.txt file — the emerging convention for telling AI assistants what matters on your site — and whether the page itself carries a noindex directive that overrides everything else.

Should you block AI crawlers?

It depends on what you sell. If your content is the product (paywalled journalism, original datasets), blocking training bots is a defensible choice. If you sell anything else, AI answers are a distribution channel: assistants that can't read your site can't recommend it. A common middle ground is to block training bots (GPTBot, CCBot, Bytespider…) while keeping assistant bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) allowed — you stay citable without feeding training corpora. The snippets above are split along exactly that line.

FAQ

My site has no robots.txt — is that bad?

No. A missing robots.txt means every crawler is allowed everywhere, which is the right default for most marketing sites. Only add one when you have something to restrict.

Does robots.txt actually stop AI crawlers?

The major operators (OpenAI, Anthropic, Google, Apple, Meta) document and honor their tokens. Some aggressive scrapers ignore robots.txt entirely — for those you need server-side blocking (WAF or user-agent rules), not a text file.

What's the difference between GPTBot and ChatGPT-User?

GPTBot crawls at scale to train models. ChatGPT-User fetches a single page because a human asked ChatGPT about it — blocking it hides you from live answers but has nothing to do with training.