Which AI crawlers should you allow?

The three categories, and why the distinction matters

Search and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Applebot) build the indexes that assistants consult when answering. Blocking these removes you from AI answers. There is almost never a good reason to do it.

User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) load your page because a specific person asked a question that required it. Blocking these blocks a prospective customer from reading your site through an assistant. Note that Perplexity’s documentation states its user fetcher generally ignores robots.txt, so blocking it is partly aspirational.

Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, Amazonbot, Bytespider) collect content used to develop models. This is the only category where a genuine trade-off exists, and it turns on what your content is for.

The training decision, argued both ways

The case for blocking training crawlers is straightforward if your content is your product. News organisations, research publishers and subscription businesses are being substituted for by models trained on their work, and receive nothing in return. Blocking is a rational defence of an asset.

The case for allowing them applies to almost everyone else. If your content is marketing — pages that exist to make people aware of your company — then training inclusion is the mechanism by which a model can name you without retrieving anything. Model priors are what produce a recommendation when no live search happens. Blocking training means you can only ever be found by live retrieval, which is a smaller and less durable surface.

There is no middle setting for this. A company that wants to be recommended by AI and blocks training crawlers has chosen a policy that works against its stated objective, usually without having decided to.

The trap: the robots.txt you serve is not the one you wrote

This is the most common misconfiguration we find, and it is nearly invisible. Cloudflare offers a "managed robots.txt" feature that prepends its own block to your file — Disallow rules for ClaudeBot, GPTBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider and Amazonbot, along with a Content-Signal line declaring ai-train=no. Other hosts and CDNs offer similar defaults, sometimes enabled without an explicit choice.

The result is a site whose owner believes AI crawlers are welcome, serving a file that tells them the opposite. Adding a permissive block further down the file is not a reliable fix: RFC 9309 requires groups with the same user-agent to be merged, and on an exact tie an Allow should win — but implementations vary, and a policy that depends on a parser resolving a contradiction correctly is not a policy. Turn the managed feature off at the edge instead.

Two more checks belong here. Bot-management and WAF rules can return 403 to AI crawlers regardless of what robots.txt says — a network-level block that no robots file will override. And robots.txt only governs crawling: a page can be crawled and still be useless if its content only appears after JavaScript runs.

A reasonable default policy

For a company whose content is marketing and whose objective is to be discovered, described accurately and recommended: allow everything, and make it explicit rather than relying on a wildcard.

Name the search and retrieval crawlers, the user-triggered fetchers and the training crawlers individually with Allow: /. Declare a sitemap. Then verify by fetching the live file from outside your network, because that is the only version that matters.

The one bot worth a separate thought is Bytespider, which offers little discovery value in US B2B markets and has a contested record on compliance. Allowing or blocking it is a low-stakes call — the point is to make it deliberately.