AI crawlers and robots.txt — should you let GPTBot in or block it

Should AI crawlers be allowed onto your site or blocked? For a company pursuing visibility the answer is nearly always: let them in. A blocked crawler cannot bring your pages into AI answers, and today that is the same as shutting the door on a search engine. There are exceptions, and the decision is worth making deliberately rather than leaving it to a plugin’s default setting. Here are the main crawlers, how to manage them in robots.txt and when blocking is justified.

The main AI crawlers and what they do

The crawlers have different jobs, and the distinction matters for the decision:

Crawler	Who	Job
GPTBot	OpenAI	collects material for training models
OAI-SearchBot	OpenAI	indexes pages for ChatGPT’s search
ChatGPT-User	OpenAI	fetches a page when a user asks for it in a conversation
PerplexityBot	Perplexity	indexes pages as sources for answers
ClaudeBot	Anthropic	collects material for training
Google-Extended	Google	controls whether content may be used to train Gemini models — does not affect search or AI Overviews visibility

The split is roughly in two. Training crawlers (GPTBot, ClaudeBot, Google-Extended) affect whether future models learn from your content. Search crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User) affect whether your pages can become sources in today’s AI answers. For visibility the search crawlers are critical and the training crawlers are strategic.

Why blocking is poison for a visibility goal

When PerplexityBot cannot reach your page, Perplexity cannot cite you. It cites the competitor whose page it could reach. When GPTBot is blocked, your content does not end up in the training material for future models, and in the models’ ”memory” your industry is represented by others. Blocking is not a neutral choice but an active withdrawal from exactly the competition GEO work is trying to win.

The most common problem is not a deliberate block but an accident: a security plugin or firewall blocks ”unknown” crawlers by default, and nobody notices until someone wonders why the company never appears in AI answers. That is why this check belongs in every GEO audit.

When blocking is justified

In fairness, blocking has its use cases. If your business is the content itself — paid media, course material, a database — blocking the training crawlers is a legitimate way to protect it, so the value of your paid content does not drain free into a model’s memory. Many media companies block GPTBot for exactly this reason. Even then it is worth considering letting the search crawlers in: they bring referral traffic without taking content for training.

For a service company, an online store or anyone whose content is marketing, blocking is sabotaging your own visibility for nothing in return.

How to manage crawlers with robots.txt

Everything allowed: if the file contains no blocks, there is nothing to do. A selective approach, allowing search and blocking training, looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Two reminders. robots.txt is a request, not a lock: reputable operators respect it, but it is not technical protection. And a block is always worth documenting: who decided, why, and when it will be reviewed. A cautious decision made a year ago can cost visibility today that nobody thinks to connect to it.

Check what robots.txt does not show

A crawler can be blocked in three layers: in robots.txt, in the server’s bot protection, and in CDN or firewall rules. Looking at robots.txt alone is therefore not enough. The most reliable test is to fetch your own page using the crawler’s user agent string and see whether you get a 200 or an error page. If technical testing is not your thing, this is a quick job to hand to a partner; for us it is standard content in a GEO audit.

Training crawlers and search crawlers are different things

This distinction settles the whole blocking debate, and it is frequently muddled. AI companies run two kinds of bots in practice, and they have entirely different effects on your visibility.

Training crawlers gather material used to teach models. Their effect shows only in the next generation of models, months later, and any single site’s share of that is not large. Blocking these is the decision the copyright debate is about.

Search crawlers fetch a page at the moment a user asks something. They bring your site into an answer immediately, and blocking them removes you from answers at once. The search functions in Perplexity and ChatGPT belong to this group.

The practical conclusion: if you want to appear in AI answers but do not want your content in training data, you can block the training crawlers and allow the search ones. This middle path works and several media companies use it. It does require knowing which bot is which, and the list changes several times a year.

Identifying bots in server logs

robots.txt tells you what you have allowed. It does not tell you what actually happens. Only the logs do that.

Filter the logs for user agents containing the word bot and see which of them are AI services. Three figures matter: how many requests each bot makes per month, which pages they target, and what response code they receive. The last is the important one. If a bot receives a 403 or 429, it is blocked or throttled even though robots.txt permits it.

This is a more common problem than people assume. Firewalls, CDN services and bot protection block AI crawlers by default, because their traffic pattern resembles vulnerability scanning. The site owner believes the bots are welcome, but the protection layer turns them away before the request reaches the server. If no AI crawlers appear in your logs at all, this is the likely cause rather than a lack of interest in your site.

An example of a working robots.txt

For a company pursuing visibility, a file that allows everything essential and excludes only functional URLs is enough.

User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /thank-you/

Sitemap: https://example.fi/sitemap_index.xml

Note what is absent: no AI crawler is named, because there is no need to single any out when everything is allowed. Search result pages and thank-you pages are excluded because they add nothing to an answer and consume crawl budget.

Remember also that robots.txt is public. Competitors can read it, and it reveals a fair amount about what exists on your site and what is treated as sensitive.

What blocking actually costs

A blocking decision feels free, because its effect never shows in any metric. That is exactly why it is worth putting into numbers.

When you block a search crawler, you lose every mention the service would have made of your company while answering questions. You do not see the loss in analytics, because those visits never arrive. You do not see it in search visibility either, because your Google rankings do not change. The only sign is that competitors appear in answers and you do not.

In return you get the assurance that your content is not used to train models. If your business rests on selling content, that is a genuine and defensible reason. If your business rests on being found, the trade is a poor one. For most small and mid-sized companies the latter applies, which is why the default recommendation is to let the bots in.

A checklist for the site owner

Work through these five points once and you will know where you stand. The whole check takes about half an hour.

• Open your own robots.txt in a browser. Look for blocks you did not add yourself; many platforms and plugins write lines into the file automatically.
• Find the AI crawlers in your logs. If none appear, the cause is probably the protection layer rather than robots.txt.
• Check the response codes. Bots should receive a 200. A 403 or 429 means a block that happens before robots.txt matters.
• Review your CDN’s bot settings. Cloudflare and similar services often ship with protections that turn away AI crawlers as unknown.
• Make sure content does not require JavaScript. Some bots do not run scripts, so text loaded by script is invisible to them even when access is allowed.

Frequently asked questions

Should GPTBot be blocked?

If your goal is visibility in AI search, no. Blocking is mainly justified when the content itself is a paid product you do not want used as training material.

Does blocking Google-Extended affect search visibility?

No. Google-Extended only controls the use of content for training Gemini models. Googlebot and the search results, including AI Overviews, work regardless.

How do I check whether my site blocks AI crawlers?

Open yourdomain.com/robots.txt and look for crawler names on Disallow lines. Also check your firewall and security plugin settings, because they can block crawlers regardless of robots.txt. The most reliable test is to fetch a page using the crawler’s user agent.

Do AI crawlers obey robots.txt?

The major operators — OpenAI, Anthropic, Google, Perplexity — state that they do. robots.txt is still not technical protection, but an agreement that reputable parties follow.

Why do no AI crawlers appear in my logs even though robots.txt allows them?

The usual cause is a firewall, CDN or bot protection turning the request away before it reaches the server. Check what response code the bots receive: a 403 or 429 means a block at the protection layer, where the contents of robots.txt make no difference.

Can training crawlers be blocked while allowing search crawlers?

Yes. Some services run separate bots for training data collection and for real-time search, and they can be told apart by name in robots.txt. The list changes several times a year, so the setup needs maintenance rather than being a one-off task.

Checking crawler access is part of our GEO audit, where we test access with real crawler identifiers across all the layers. For a quick view of where you stand, run the AI visibility test.