Should you block or allow AI crawlers? The GPTBot and ClaudeBot decision

Mostly: allow the crawlers that answer questions, and block the ones that only train, if you block anything at all. The move to avoid is blocking everything and quietly deleting yourself from AI answers, which is exactly what some setups, including Cloudflare’s newer defaults, now do on your behalf. The whole decision turns on one distinction most guides skip, so start there.

01 · The frameworkThe distinction that decides everything

AI crawlers do not all do the same job, and that is the entire decision. They split three ways:

• Training crawlers fetch your pages to train a model. You get nothing back. Blocking these keeps your content out of a model you do not control.
• Answer crawlers fetch your pages to build the index an assistant answers from. Allowing these is how you get cited in ChatGPT, Claude, or Perplexity, with a link back to you.
• User-initiated fetches happen when a person pastes your URL into an assistant and asks about it. Blocking these mostly just frustrates someone who already wanted your page.

So “should I block AI crawlers” is the wrong question. The right one is “which of those three jobs do I want to allow?” For most people the answer is the same: block training if you like, keep the answer crawlers open, because the answer crawlers are your visibility.

02 · The bot mapWho is crawling you in 2026

03 · The decisionPick a posture

There are really only three coherent positions. Match yourself to one.

The selective posture is the popular middle ground, and in robots.txt it looks like this:

# Block training-only crawlers
User-agent: GPTBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: ClaudeBot
Disallow: /

# Allow the crawlers that cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Never block this one
User-agent: Googlebot
Allow: /

A note on that file: in robots.txt, a missing directive already means allow, so those answer crawlers are permitted by default. The Allow lines are there to document intent, which matters when someone inherits the file a year later. Whatever posture you pick, pair it with an llms.txt so the crawlers you do allow can find your best pages in one fetch instead of guessing.

04 · The default trapCloudflare may have decided for you

If your site is behind Cloudflare, you may have already made this decision without realising it.

On July 1, 2025, Cloudflare became the first major infrastructure provider to block AI crawlers by default, and “block on all pages” is now the default for newly created domains. New sites are asked up front whether to allow AI crawlers, and if you clicked through setup without thinking about it, the answer may be no. The result is blunt: a product that wants to be found in AI answers can be invisible to them straight out of the box, while competitors who opted in become the sources those assistants cite instead.

Cloudflare’s AI Crawl Control dashboard is where you change this. It shows which crawlers are hitting you, whether they actually respect your robots.txt, and lets you allow or block each one individually. It also offers Pay Per Crawl, a third option that charges crawlers through a 402 response, aimed at publishers with content worth licensing rather than at most indie products.

The action item is simple: do not accept whatever the default left you on. Open the dashboard, see who is allowed, and make it a deliberate choice.

05 · Enforcement realityrobots.txt is a request, not a wall

Even a perfect robots.txt only works on crawlers that choose to obey it. The major labs say theirs do, but a large share of AI traffic ignores the file entirely, and robots.txt has no technical means to stop them. If you genuinely need to block a bot, that lives at the server or firewall layer, where a rule is evaluated before robots.txt is ever read, so a firewall block overrides any Allow.

It is worth being honest about why blocking tempts people. The exchange that made crawling fair, where search indexes you and sends visitors in return, has broken for AI.

That imbalance is the real argument for blocking training crawlers: they take much more than they return. But it cuts the other way for anyone without an audience yet. If nobody is visiting, there is no referral relationship to protect, and the content a cold product guards is rarely a moat someone is racing to copy. Invisibility costs a new product more than scraping does.

One more reality check: if your pages are a client-side JavaScript app, this whole debate may be moot. Most AI crawlers do not run JavaScript, so they receive an empty shell no matter what your robots.txt says. Fix that first, because it is part of being parseable at all.

06 · The recommendationSo what should you actually do?

Stripped down:

• If you are launching something and want to be found, allow the answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and stop worrying about the rest. Optionally block the training-only ones. Above all, do not let a default block delete you from AI search.
• If you run a content business with a real audience and real IP, take the selective posture: be cited, do not feed training.
• If your content is genuinely proprietary or premium, block it all and accept you will not be cited. That is a valid choice, as long as it is a choice.

Whatever you pick, the failure mode is the same: not choosing, and letting an inherited robots.txt or a platform default decide for you. Remember which half of the launch picture this sits in too. Being absent from AI answers is a Footprint problem, not a content one. The worst posture is an accidental one.