Should you block AI crawlers like GPTBot on your company website?

Not if you want AI assistants to recommend you, with one exception that is a genuine judgment call. The AI companies now run separate bots for separate jobs, and the documentation each one publishes makes the split clear: one agent collects training data, another indexes pages for the assistant's search feature, and a third fetches a page when a user asks a question about it. Blocking the training bot is a defensible choice. Blocking the search and user-fetch agents removes your site from the answers, and a publisher-style default on your CDN may be doing that already without anyone on your team having decided it.

This is a different question from whether an llms.txt file helps. That file is a request. A robots.txt disallow is an instruction the compliant bots obey, so getting it wrong has consequences.

What do the different AI bots actually do?

Each major AI company documents its own agents, and the distinctions are the whole story.

OpenAI runs three that matter. According to OpenAI's bot documentation, GPTBot "is used to crawl content that may be used in training our generative AI foundation models," and disallowing it "indicates a site's content should not be used in training." OAI-SearchBot is different: it "is used to surface websites in search results in ChatGPT's search features," and the documentation states that sites which opt out of it "will not be shown in ChatGPT search answers, though can still appear as navigational links." The third, ChatGPT-User, handles "certain user actions in ChatGPT and Custom GPTs," is "not used for crawling the web in an automatic fashion," and is "not used to determine whether content may appear in Search."

Anthropic splits its agents the same way. Per Anthropic's crawler documentation, ClaudeBot collects "web content that could potentially contribute to their training," and restricting it "signals that the site's future materials should be excluded from our AI model training datasets." Claude-SearchBot "navigates the web to improve search result quality," and disabling it "prevents our system from indexing your content for search optimization, which may reduce your site's visibility." Claude-User fetches pages "when individuals ask questions to Claude," and blocking it "prevents our system from retrieving your content in response to a user query."

Google's control is narrower than most people assume. Google's crawler documentation describes Google-Extended as a token publishers use "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models," and for grounding in Gemini Apps and Vertex AI. The same page states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." AI Overviews and AI Mode are Search features, and Google's AI features documentation says that to be eligible to appear as a supporting link, "a page must be indexed and eligible to be shown in Google Search with a snippet," with "no additional technical requirements." So blocking Google-Extended does nothing to AI Overviews either way, and the controls that do affect them are the ordinary ones: noindex, nosnippet and max-snippet.

Perplexity has one wrinkle worth knowing. Its bot documentation says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and respects robots.txt, while Perplexity-User, which fetches a page when a user asks a question, "generally ignores robots.txt rules" because a person requested the fetch. Apple draws the cleanest line of all: Apple's documentation says Applebot-Extended "does not crawl webpages" but governs whether Applebot's data can train Apple's models, and "webpages that disallow Applebot-Extended can still be included in search results."

So which bots is it safe to block, and which ones cost you?

Sort them by job, not by company.

The training-only agents are GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Blocking those, according to each company's own documentation, keeps your future content out of model training sets and does not remove you from ChatGPT search, Claude's search, Google Search, AI Overviews or Apple's search features. If your legal team or your board wants a position on training, this is where that position lives, and it costs you nothing in search visibility today.

The search-index agents are OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking these takes you out of the citations. OpenAI says so in plain words, and Anthropic says the same. A company that wants to get cited by ChatGPT or appear in Perplexity's answers and has these disallowed has made its own job impossible, however good the content is.

The user-fetch agents are ChatGPT-User, Claude-User and Perplexity-User. These fire when a person asks the assistant about your company by name or pastes your URL. Blocking them means the assistant cannot read your pricing page, your About page or your latest announcement when someone asks about you, and will answer from whatever third parties say instead. For most companies, that is the worst of the available outcomes.

The exception, the genuine judgment call, is the training block. There is a real argument for it: a model that cannot train on your pages learns about you only from everyone else's. There is also a real argument against: if the model's memory of your category was formed without your site, its unsourced answers, the ones with no live search behind them, will not know you. Neither side is wrong. What is wrong is treating a blanket "block all AI bots" rule as a free, consequence-free default.

Could a CDN setting be blocking you without anyone deciding to?

Yes, and this is the part to check this week. On July 1, 2025, Cloudflare announced that it would change its default settings to block AI crawlers unless they pay creators for their content, framed around publisher economics: the post argued that compared with the Google of old, it had become 750 times harder to get a visit back from OpenAI's crawling and 30,000 times harder from Anthropic's. That is a publisher's problem. A software company that wants to be recommended has the opposite problem, and the same default.

The policy has since become more granular. Cloudflare's current documentation sorts AI bots into Search, which collects content to answer questions later, Training, which takes content to train or fine-tune models, and Agent, automated activity acting in real time on someone's behalf. From September 15, 2026, the default blocks Training and Agent bots on pages that display ads while allowing Search bots, and a mixed-purpose crawler that combines Search and Training is blocked by every configuration. The older blanket "Block AI bots" switch is being deprecated on the same date.

For a company site that carries no advertising, the ad-page default may not bite. For one that does, or for one where somebody flipped the old blanket switch in 2025 and forgot, the Agent category is the one to look at: by Cloudflare's own definition it covers the user-fetch behavior described above. Check the setting rather than assuming. The test is simple: ask ChatGPT, Claude and Perplexity about a page on your site by URL and see whether they can read it.

Does blocking crawlers protect you from being misdescribed?

No, and in practice it does the reverse. Muck Rack's study of 25 million AI-cited links found that 84% of citations across ChatGPT, Claude and Gemini came from earned media, with paid and advertorial content at 0.3%. The assistants are mostly reading what journalists and third-party sites wrote about you, not your own pages. Those sites set their own robots policies, and most of the major ones have commercial reasons to stay visible to search-class bots. Your block changes none of that.

What it does change is the correction mechanism. When the assistant can read your site, a wrong third-party claim about your pricing or your product can be checked against the source. When it cannot, the third-party claim stands alone. The same logic applies to Google AI Mode, which draws on the ordinary Search index and treats your indexed pages as one voice among the sources it fans out across.

What should a company actually put in robots.txt?

For a company whose goal is to be found and recommended, the default should be to allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-fetch agents, and to make a deliberate, documented decision about GPTBot, ClaudeBot, Google-Extended and Applebot-Extended rather than inheriting one. Write the decision down, note who made it and when, and revisit it when the documentation above changes.

Then check the CDN layer separately, because robots.txt is only read by bots that are allowed to reach it. If Cloudflare or another provider is blocking at the edge, the file is irrelevant. And put the pages you most want an assistant to read, pricing, product, leadership, the press page, on the open side of any rule you keep.

The direct answer, for anyone who read this far

Blocking the training-only bots, GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, is a legitimate choice that, according to each company's own documentation, does not remove you from ChatGPT search, Claude's search, Google Search or AI Overviews. Blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot or the user-fetch agents takes you out of the answers. Cloudflare's defaults have blocked AI bots by category since September 15, 2026, so check what your CDN is doing before you decide what your robots file should say. And remember that the citations mostly come from coverage on other people's sites, which no setting of yours controls.

If you want an honest read on where your company stands in AI answers today, request a quotation and we will show you what the assistants currently say and where it comes from.