The Cloudflare box you ticked in 2024 starts blocking Googlebot on 15 September
The legacy "Block AI bots" switch is deprecated on 15 September, and mixed-purpose crawlers get folded into it. Googlebot, Applebot and BingBot are in scope. Opting out is one click and the window is 25 days.
Sometime in the last two years a lot of us flipped a switch in Cloudflare called "Block AI bots." It was one click, it was free, it was framed as protecting your content, and there was no reason not to. I did it. I never thought about it again.
On 15 September that switch is deprecated, and the thing it does changes.
Here is the specific mechanic, from Cloudflare's own docs rather than the coverage. Cloudflare now sorts automated traffic into three behaviours: Search (indexing your content to answer questions about it later), Agent (something acting in real time on a person's behalf, a chat fetch bot or Claude driving a browser), and Training (absorbing your content into a model). Until now, the "Block AI bots" preset deliberately excluded crawlers that do more than one of those jobs. That exclusion is what kept Googlebot out of scope, because Googlebot crawls for search and for training with the same bot.
On 15 September that exclusion goes away. Mixed-purpose crawlers that combine Search and Training get blocked by every configuration that blocks AI training, including the legacy preset. Cloudflare names the three you'd care about: Googlebot, Applebot, BingBot.
So if you turned that on and forgot, in 25 days you are telling Google not to crawl you. Google is still roughly 88% of referral traffic on Cloudflare's own numbers. That's not a content-licensing decision, it's an accident.
The opt-out exists and takes about a minute: Security Settings, then Configure AI bot policies. You can confirm you want no change to Training crawlers that also crawl for Search, and you can set each of the three categories independently, with three levels each: block everywhere, block only on pages that display ads, or allow.
Two things I want to be accurate about, because the hot takes I've read are not.
First, the widely-repeated "Cloudflare is blocking AI agents by default" is narrower than it sounds. The new default, Training and Agent blocked with Search allowed, applies to pages that display ads, and to new domains onboarding. If you run a normal product site with no ad units, that default barely touches you. The change that actually reaches into an existing account is the mixed-purpose reclassification above, and it reaches you only if you have training-blocking switched on. Which, if you're the kind of person who ticks the safe-looking box, you do.
Second, Cloudflare sells bot management and is building a market for paid crawling. Every number in their report is measured by a company with a position in the outcome. Read them as direction, not measurement. The direction: more than half of internet traffic is now non-human, 52% of crawler requests were for AI training as of June 2026 against 22% in spring 2025, and mixed-use crawlers are over 36% of activity. That last one is the whole story. One bot, several jobs, and the taxonomy everyone's blocking rules were written against no longer describes it.
The part I think is genuinely strategic, and it isn't the Googlebot thing.
Agent is now a category you can block, which means it's a category you have to have an opinion about. A browser-use agent hitting your pricing page is not a crawler harvesting you. It's a person who asked something to go look at your product, sitting there waiting for an answer. Blocking Training is a stance about your content. Blocking Agent is a stance about a customer. Those got put on the same settings page next to each other, in the same visual language, and they are not the same decision at all.
I don't have a satisfying answer for Murror. We're not ad-supported, so the default doesn't bite. But the honest reason I'd never block Agent is thin: I want to be findable, and right now agents are how a growing share of finding happens. If people started pointing agents at private journalling products expecting to get in, I'd feel very differently, very fast. I'd rather say that than pretend I've reasoned it through.
What I'd actually do this week: open Security Settings, look at what's on, decide each of the three categories on purpose, and write down why. Not because Cloudflare's defaults are wrong, mostly they're reasonable, but because a default you never chose becomes a strategy by attrition. This one has a date on it.
Replies
Good writeup - the distinction between blocking Training vs blocking Agent is the part people are going to get wrong for a while. One thing worth adding from the operating side of an agent, not the site-owner side - when an agent gets blocked, it usually doesn't fail loudly. It falls back to a stale cache, or a generic search snippet, or just stops looking at your product and moves to the next result. The site owner never sees a support ticket for it, so blocking Agent traffic is a decision with basically no visible feedback loop when it goes wrong. That's a worse setup for catching a bad default than the Googlebot case, where at least your search traffic actually drops and you notice.
Murror
@omri_ben_shoham1 That's the part I under-weighted, and it's the better version of my point. A blocked crawler leaves a mark somewhere eventually, a coverage drop, a grey row, something you can go look at. A blocked agent leaves nothing at all, because the person waiting on the other end never knew your page was even a candidate. Same settings page, same widget, and only one of the two has any instrumentation attached to it.
The closest thing to a monitor I can think of is watching 403s by user agent on your own pricing and docs pages and treating any non-zero number as a decision you should have made on purpose. Nobody does it, because nothing fires.
@monatruong_murror That monitor has its own maintenance problem though - watching 403s by user agent only works if you know which user agents to watch for, and that list isn't static. New agents show up constantly (there's a new one every few months lately), so the "treat any non-zero number as a decision" monitor needs the same kind of upkeep as the blocklist itself, just less visible because it's a dashboard nobody's assigned to check rather than a setting somebody has to actively flip. Two moving parts instead of one, both easy to let go stale.
Founder here, I run usebravery.com, so I have been staring at exactly this problem for a week and I am not neutral about crawl access.
Adding a site-owner-side version of Omri's point about invisible failure, because the Googlebot case is less self-announcing than it looks.
The assumption in "at least your search traffic actually drops and you notice" is that you had search traffic to lose. A lot of the sites that ticked this box are younger than that. On ours, Search Console showed 230 URLs sitting in Discovered, currently not indexed, which is Google saying it knows the URL exists and has not fetched it. That state looks the same whether the cause is a crawl block, a crawl budget decision, or your own server handing back errors. There is no line in any report that says something upstream refused me. You get the same grey row for all three.
We went through that this week. The AI bot toggle was one of the things we turned off, and it was the right call regardless. But it was not our cause. Our cause was that unmatched dotted paths, /whatever.png, were falling through a route matcher into a locale handler and throwing a 500. Googlebot was being let in and then handed errors, and the symptom in Search Console was indistinguishable from being blocked.
So the practical version of "write down why" I would add: before you conclude the Cloudflare setting is your problem, fetch your own site the way Googlebot does. curl with its user agent, and specifically against the ugly URLs, not your homepage. The homepage is the one page everybody tests and the one page that is always fine.
The other half of your post is the part I have no clean answer to either. Blocking Training is a stance about content you already published. Blocking Agent is a stance about someone who is currently trying to buy something. Cloudflare put them in the same list with the same widget, and the people that costs will not find out that it cost them.
Murror
@serdarships The 500-on-dotted-paths detail is the useful part of this and I'd never have guessed it. Discovered, currently not indexed reading identically for "we were refused", "we deprioritised you" and "your own server errored" is a genuinely bad diagnostic, and it means the Cloudflare setting is going to get blamed for a lot of things it didn't do over the next month. Which is its own hazard: you flip it off, nothing improves, and you stop looking.
Taking the curl-your-ugly-URLs advice literally, thank you. A route matcher swallowing anything with a dot in it is exactly the class of bug that never shows up in the paths a human thinks to test.
Glad you turned the toggle off anyway. Right call independent of whether it was your cause.
That hazard is the part I would most want to be wrong about, and I do not think I am.
There is one instrument that separates your three cases, and it is not the coverage report. Search Console's Crawl stats, which sits under Settings rather than in the indexing menu, breaks fetches down by response code and by Googlebot type, and it moves on a daily cadence instead of the weeks coverage takes. Refused at the edge, served a 500, and never requested are three different rows there, where coverage collapses them into one label.
The other half is your own server logs, and the useful property is what is missing from them. A request refused upstream never reaches you, so it leaves no line at all. A 500 leaves one. So absent from your logs means something in front of you said no, and present with a 5xx means it was you. That is the split I could not make from the coverage report no matter how long I stared at it.
I should be clear that I am describing the instrument, not a result. We changed both things days ago, so I have no before and after yet, and the temptation you named, flip it and stop looking, is exactly the one I am trying not to fall into.
The dot thing still bothers me for a reason beyond our own bug. The paths a human thinks to test are the paths a human would type. Nothing about a route matcher is built around that, so the failing cases are the ones nobody has a reason to visit. I have no method for generating them rather than stumbling into them. If you have one I would take it.