Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk being blocked by default on many publisher sites. That deadline is today. On September 15, 2026, Cloudflare is setting new defaults for each of these three classifications, with the categories of Training and Agent blocked by default on pages that display ads, while Search remains allowed. The Cloudflare announcement landed in July. The policy is real. The infrastructure change shipped. The separation it assumes has not.
I run ten platforms that hit APIs, scrape public data sources, and call models to generate content. The construction ERP at coenconstruction.com pulls permit data from municipal sites. The review pipeline on valhalla-k9.com scrapes competitor pricing twice a day. The change-order parser on thepunchlist.ai ingests PDFs from contractor portals. None of those systems care whether Cloudflare categorizes a bot as Search, Training, or Agent. They care whether the request returns a 200 or a 403, and whether the payload contains the data the next function in the pipeline expects.
Multi-purpose crawlers like Googlebot are affected, and Cloudflare states that a multi-purpose crawler is handled under the most restrictive rule that applies to it, naming Googlebot, Applebot and Bingbot as crawlers that will be blocked by customers who chose to block Training. That is the part that matters. Google operates one bot. It indexes pages for search. It also trains models. The policy targets crawlers like Googlebot, Bingbot and Applebot that combine search indexing with AI training data collection. If you block Training, you block Googlebot. If you block Googlebot, you disappear from Google Search. Publishers have been reluctant to block these bots despite concerns about unpaid content usage, fearing removal from search results.
The policy shipped. The bots are still mixed-use.
Cloudflare announced this policy in July. It gave the industry two and a half months to split crawlers into three declared purposes. How AI providers react — separating their agents, or not — will determine the real scale of the effect on multi-purpose crawlers. As of this morning, Googlebot is still Googlebot. It does not send a header that says "I am indexing this page for search only, not for training." It does not offer a separate user-agent string for Gemini training versus Google Search indexing. The technical mechanism Cloudflare's policy depends on does not exist in the bots it is trying to regulate.
The exact definition of a page that displays advertising is not spelled out in detail: it is unclear whether it relies on detecting ad scripts, on a customer declaration, or on another method. That ambiguity is load-bearing. If Cloudflare detects ads by scanning for Google AdSense tags, then every page monetized through Google is now in scope. If it relies on customer declarations, then the default only applies when a site owner explicitly flags a page as ad-supported. If it relies on heuristics, then false positives will block legitimate traffic and false negatives will let training crawlers through. The policy does not say. The documentation does not clarify. Site owners are left to infer from server logs whether the block is working or not.
I have written before about data having a buyer before you price it and agents reading everything and deploying nothing. This is the infrastructure version of that problem. Cloudflare ships a policy that assumes crawlers will separate their purposes. The crawlers do not separate. The policy still applies. Site owners are now in a position where they either allow all uses or block all uses, and the block includes the search traffic they depend on for monetization.
Cloudflare CEO Matthew Prince said bot traffic now makes up the majority of internet traffic, a milestone the company did not expect until the following year. That acceleration is real. The construction ERP sees it in the logs. Forty percent of requests to the estimating interface at estimate.pro are bots. Half of those bots do not respect robots.txt. A quarter of those bots rotate IPs fast enough that rate limiting by IP is useless. The ones that declare a user-agent are the minority. The ones that declare a purpose are rarer still.
The default flipped today. The crawlers are still mixed-use. The policy is real, the separation is not, and every site owner behind Cloudflare now owns a choice the infrastructure made for them.
A new Pay Per Use program lets publishers get compensated when their content actually surfaces in an AI answer. That monetization path is the carrot. The block is the stick. The problem is that the carrot requires a deal with Cloudflare, a deal with the AI company, and a content pipeline that can attribute which answer came from which page. The construction vertical does not have that infrastructure. The dog training vertical does not have that infrastructure. The residential service industries I build software for do not have the leverage to negotiate those deals, and they do not have the traffic volume to make Pay Per Use financially viable even if the deals existed.
The policy that went live today is not theoretical. It changed the default for millions of sites. It blocked crawlers that did not separate their purposes. It left site owners to decide whether to override the default and allow training, or accept the default and risk losing search traffic. The decision is infrastructure now. It runs at the CDN layer. It does not wait for you to file a ticket or adjust a setting. The default is the decision, and the decision shipped this morning.