The setting on your website that can block ChatGPT

Most people who own a website have never thought about which robots are allowed to read it. For twenty years there was no reason to. If a page was public, crawlers could see it, Google indexed it, and that was the end of the conversation.

That deal has quietly changed, and on 15 September it changes again.

What is actually happening

Cloudflare sits in front of a very large share of the web. It handles security and performance for millions of sites, including a lot of small business sites where it was set up years ago by a developer and never touched since.

In July, Cloudflare announced that from 15 September 2026 it will sort AI crawlers into three categories, search, agent and training, and apply different rules to each. Training and agent crawlers will be blocked by default on pages that carry advertising. Search crawlers stay allowed.

The new defaults apply to new Cloudflare customers, to new sites added by existing customers, and to everyone on the free plan. Paying customers with existing configurations keep what they have.

There is one detail in there that catches people out. Some crawlers do more than one job. Googlebot, Applebot and Bingbot all perform search and training functions, and under the new rules a multi-purpose crawler is judged by the strictest policy that applies to it. Block training and you can end up blocking the whole crawler, Google included.

If your site does not carry ads, and most small business sites do not, this particular change probably will not catch you. That is worth saying plainly rather than making it sound more dramatic than it is.

Why it still matters if you are not affected

The reason to pay attention is not the 15 September deadline on its own. It is what the deadline reveals.

A setting nobody consciously chose can decide whether AI tools are able to read your website. It can be switched by a hosting migration, by a security default, or by a policy change at a company you have never dealt with directly. And when it happens, nothing tells you.

There is no warning email. Nothing appears in Google Search Console. Your organic traffic carries on looking completely normal, because Googlebot is usually still getting through. You could be entirely absent from the answers ChatGPT and Perplexity give about your industry for months and have no way of knowing.

That is a very different failure mode to the ones businesses are used to. A broken contact form gets noticed. A site that has gone down gets noticed within the hour. Being quietly unreadable to the tools your customers are increasingly asking first does not announce itself at all.

We have written before about why AI search visibility matters and what is at stake if you wait. This is the technical floor underneath all of it. None of the content work matters if the crawlers cannot reach the page in the first place, which is also the practical answer to the question we tackled in is AI damaging your search engine rankings.

The two checks worth doing this week

Neither of these requires a developer.

The first takes about thirty seconds. Open ChatGPT and ask it to summarise a specific page on your website, giving it the full URL. If it comes back and tells you it cannot access the page, you have your answer immediately. Try it with two or three pages rather than just the homepage, because access can vary.

The second is the definitive version. Log in to Cloudflare, go to AI Crawl Control, and look at the allowed and blocked request counts over the last 24 hours. The panel lists each crawler by name, and OpenAI publishes its own list so you can check which is which. If the major ones are showing high blocked counts and zero allowed, that is your problem, and the fix is a toggle rather than a project.

If you do not have Cloudflare login details, whoever built or hosts your site will. It is a reasonable thing to ask them.

The mistake almost everyone makes

When business owners do finally open these settings, the instinct is usually to block everything. Nobody likes the idea of their content being hoovered up to train a model that then competes with them for attention.

That instinct is understandable, and acting on it wholesale is a mistake.

Training and search run on separate bots. Block GPTBot and you can still be cited in ChatGPT Search, because that runs through OAI-SearchBot. Block ClaudeBot and Claude-SearchBot still works. Perplexity splits its crawlers the same way.

So the position that makes sense for most businesses is to block the training crawlers and allow the search crawlers. You are not handing over your content to train models for free, but you still turn up when somebody asks an AI tool for a recommendation in your sector. Those are two separate decisions and they deserve to be made separately.

There is a third option that is mostly relevant to publishers rather than SMEs. Cloudflare has been developing a paid access model, moving from charging per crawl to compensating publishers when their content actually surfaces in an AI answer. If your business model depends on content itself rather than on content bringing people to a service, it is worth watching.

Robots.txt is not enough

A lot of the advice online tells you to add rules to your robots.txt file and leave it there. That is half a solution.

Robots.txt is a request. It is a sign taped to the door saying please do not come in. Well behaved crawlers read it and comply, and several major AI companies have been sued precisely because they carried on crawling despite explicit disallow rules. Some user-triggered fetchers openly state that robots.txt may not apply to them, on the basis that a human asked for the page. Google’s own crawler documentation sets out how its various bots behave and is worth reading if you want to get the rules right.

Cloudflare’s rules operate at network level. The request gets a 403 and never reaches your site, whether the crawler likes it or not.

You want both layers, and you want them saying the same thing. A robots.txt that welcomes search crawlers while your firewall blocks them is a configuration that will confuse everyone including you.

What to do before the 15th

Run the two checks above. Set explicit allow rules for the crawlers your business actually depends on rather than relying on category defaults, because defaults are exactly what caused the problem in the first place. Then check your logs again a week after the 15th to confirm nothing moved underneath you.

This is precisely the sort of thing that should be picked up as routine rather than discovered by accident, which is part of why we argue that a monthly website maintenance package is no longer optional. Crawler access has joined SSL certificates, plugin updates and backups on the list of things that quietly break without telling anyone.

If you would rather someone else looked at it, our digital support and optimisation team can audit crawler access across your site and tell you exactly what is getting through, and if the audit turns into something bigger our websites and digital platforms team can pick it up from there. Either way it is a short job and you get a straight answer.