Should you block AI crawlers from your website?

Somewhere in the last two years, “block the AI bots” became a default recommendation in certain corners of the web, and robots.txt files everywhere filled up with GPTBot disallows. For a publisher whose content is the product, that can be a rational position. For a business whose website exists to be found and chosen, it is usually self-harm performed as a matter of principle.

The decision is not about how you feel about AI companies. It is about what your website is for.

What blocking actually does, and does not do

A robots.txt disallow for GPTBot, ClaudeBot, PerplexityBot and the rest asks those crawlers not to fetch your pages. The compliant ones honour it. Consequences follow in two layers:

Training: your content contributes less to future model training. This is the layer publishers care about, and for them the argument is genuine: their words are their asset, and models that ingest them may substitute for visits.

Retrieval: this is the layer businesses forget. Modern assistants do not only answer from training data; they search and read pages live to ground their answers. Block the retrieval crawlers and you are absent at the exact moment a model is assembling an answer to “who should I hire for this near me.” Your competitors, unblocked, fill the space. No error message, no notification, just a recommendation list you are never on.

The question that decides it

What does your website exist to do?

If content is the product, you monetise attention on your pages, subscriptions, ads, licensing, then AI answers that substitute for visits take something real from you, and blocking, or negotiating licensed access, is a defensible commercial position.

If the website exists to win customers, every surface where buyers ask questions is a surface you want to be on. AI assistants are now such a surface, growing, and the businesses named in their answers took no special action except being readable, resolvable and cited. Blocking removes you from consideration in a channel where you never see the loss happen. It is turning away customers to spite the doorman.

Almost every business we work with is the second kind. Their content is not the asset; it is the brochure. Guarding a brochure from readers is a strange use of a robots.txt file.

The middle positions

The choice is not binary, and two nuances are worth knowing.

You can split the layers. Google-Extended controls Gemini training without touching Google Search. Some operators block training-specific agents while allowing the user-triggered retrieval ones. The roster of bots keeps changing, so this needs occasional review rather than a set-and-forget line.

You can also block selectively by section: allow the pages that describe what you sell, disallow the proprietary research or gated resources. Most business sites have little that genuinely warrants protection, but where something does, section-level rules beat a blanket ban.

One caution: check what you are already blocking by accident. CDN bot-protection products and boilerplate robots.txt files sometimes block AI crawlers by default, and plenty of businesses that would never choose invisibility have it configured anyway. An afternoon’s audit of robots.txt and the CDN’s bot rules is worth it just to make the current position a chosen one.

Where we land

For publishers: a real decision with money on both sides; take it seriously, maybe take it to a lawyer.

For everyone else: allow the crawlers, and put the effort into being worth citing, clear entity data, specific quotable answers, presence in the sources models trust. Visibility in AI answers is becoming ordinary marketing surface area. The robots.txt line that keeps you out of it protects nothing you were selling and costs you customers you never knew asked.