On 15 September, Cloudflare (which sits in front of around a quarter of all websites) will be changing its default settings for AI crawlers. When a company at that scale changes a default, a meaningful share of the web changes with it, mostly without anyone at those sites pressing a button.
The change has been public since 1 July, but it has been reported in ways that blur who is affected. So here is the plain version, followed by what it means for the people who make content, the people trying to be visible in AI answers, and the quality of the answers themselves.
What happens on Monday
The mechanics: one "block AI bots" switch becomes three categories, available to every customer including free accounts since 1 July.
Search crawlers index content and send referral traffic back. Agent crawlers act in real time on a person's behalf, the kind of fetch that happens mid-conversation when someone asks an assistant to look at a page. Training crawlers collect content in bulk to build models. From 15 September, on pages that display advertising, training and agent crawlers are blocked by default, while search crawlers stay allowed. Cloudflare's reasoning is that an ad on a page signals a human audience is expected, so bots that substitute for that human are kept out.
Two details decide whether this touches you. The first: the new defaults apply to domains newly onboarding to Cloudflare, new sites added by existing customers, and free-tier accounts that have not changed their settings. Paying customers who have already configured things keep what they have. Coverage has been inconsistent on exactly which existing zones inherit the new defaults, so the only reliable answer is the one in your own dashboard.
The second detail catches people, and it applies to everybody. From the same date, mixed-purpose crawlers are judged by their most restrictive behaviour. Googlebot, Applebot and Bingbot all crawl for search and for AI features under one user agent, i.e. blocking Training also blocks Googlebot on ad-carrying pages. If someone at your company flipped that old blanket toggle in 2025 and never mentioned it, this is the sentence worth reading twice.
Alongside this, Cloudflare's pay-per-crawl experiment is becoming pay-per-use, which pays publishers when their content is used in an AI answer rather than when a bot fetches the page, launching with Ceramic.ai and You.com. And it is extending the Content Signals format in robots.txt with a parameter describing how a crawler may reuse what it takes: immediate with no storage, reference with indexing and links back, or full reproduction.
Why now
Some numbers behind this decision: by June this year, 52% of crawler requests were for AI training, up from 22% in spring 2025, and mixed-use crawlers had grown to more than a third of all crawler activity. Bot traffic has overtaken human traffic on the internet, earlier than anyone expected.
The ratios are starker still. Anthropic's crawl-to-referral ratio, per Cloudflare Radar at the end of August, sits above 20,000 pages fetched for every visitor sent back. Publishers noticed, and the same analysis shows what they are doing about it: blocking training crawlers heavily while allowing the ones that answer questions and send people onward. GPTBot now appears in more disallow rules than any other AI crawler.
What it means if you publish content and sell ads
If your revenue depends on human attention to pages carrying advertising, this is straightforwardly in your favour, and it is the first default in years that starts from your side of the table. You get three dials instead of one, and the ability to stay in AI search results while declining to be training material.
The catch is the Googlebot problem. For most ad-supported publishers, search referral traffic still pays the bills, and blocking training now costs you a search crawler as well unless you explicitly opt out of the new default. That is not an accident. Cloudflare is applying pressure to the companies that bundle search and training into one bot, and publishers are the leverage. It is a reasonable strategy and it may work, but until those companies split their crawlers, the choice on the table is uncomfortable.
What it means if you are trying to be visible in AI answers
If you run a business site with no advertising, the new default does nothing to you at all. Your product pages, your blog, your case studies carry no ads, so the ad-page rule never fires. It is worth saying plainly, because a fortnight of headlines has convinced a lot of marketing teams they are about to be cut off from AI search, and they are not.
Your exposure is in two other places. The first is that legacy toggle, which someone in your company may have switched on in 2025 and which will now take Googlebot with it on any page that carries ads. Ten minutes in the Cloudflare dashboard settles it.
The second is subtler and more interesting. The third-party pages that currently cite your category, the roundups, the comparison articles, the trade press, are exactly the ad-supported publisher pages this default targets. Your own site being open is necessary and not sufficient, because a great deal of what AI engines know about your category comes from sites that are about to make a decision about crawler access. If the review site that mentions you goes dark to training crawlers, your presence in the next generation of model knowledge quietly thins, and nothing about your own configuration will tell you that happened.
Is any of this enforced?
Here is where it is worth separating what these files say from what happens. robots.txt is essentially a request, not a lock. It has no technical enforcement and no settled legal force, and OpenAI has argued in court that it does not constitute a technological protection measure under copyright law.
The empirical picture is better than the cynical version and worse than the comfortable one. A large-scale study of crawler behaviour found that most of the major named crawlers do comply with disallow rules, with ClaudeBot, ChatGPT-User and Amazonbot all showing full compliance on that measure. Compliance with crawl-delay and endpoint-level rules was far patchier, and a handful of crawlers ignored disallow directives almost entirely. Bytespider, from ByteDance, showed compliance close to zero on that test.
Two further findings matter more than the averages. In one test, an assistant's crawler fetched a page and logged a successful response, while the assistant's own answer claimed it could not access that page. What a model says about whether it visited you is not evidence. The request line in your server log is. And AI browsers now render pages in a full browser, authenticated like a human user, which lets them reach content that blocks the same company's declared crawler. Researchers at the Columbia Journalism Review confirmed agents reaching paywalled material that had been blocked at the crawler level.
Which is why the Cloudflare change matters more than a robots.txt line ever could. The text files express intent. The network layer imposes it. That distinction is the whole story of the closing web, and it is why the enforceable question is what your CDN and your firewall do, not what your robots.txt says.
What happens to the answers?
Set the commercial argument aside for a moment. So what happens to AI answers when the open, professionally produced part of the web closes to crawlers while the volume of machine-written pages keeps climbing?
Model collapse is the well-known version of this worry: training models recursively on their own output degrades them, a result published in Nature in 2024, with the effect appearing even when synthetic data is only a small share of the mix. Its strongest claims have been challenged in the research, though, and labs curate their training data rather than hoovering up whatever is lying around, so the apocalyptic reading is not the responsible one.
A slightly different version came out of The Web Conference this year: "retrieval collapse", a two-stage failure in the retrieval layer rather than in the model itself. In the first stage, fluent, well-optimised synthetic content captures the top results and source diversity drops. The authors make a point that should worry anyone measuring this: that first stage is deceptively healthy, because the fluency of generated text keeps answers looking fine while the sourcing underneath erodes. The second stage arrives when low-quality or adversarial material gets into the retrieved set.
One analysis found roughly three quarters of newly published pages now contain some AI-generated material. Put that beside the closing web and the shape of the risk is clear enough. It is not that models forget how to speak. It is that the pool of independent, verifiable, human-produced evidence they retrieve from gets shallower, while the confident tone of the output stays exactly the same.
Both halves are the same problem from different ends. A web closing to crawlers and a web filling with synthetic text might both be a recipe that shrinks the pool of trustworthy material an engine can reach, which makes the sources that stay open and stay trusted increasingly valuable. For brands that is not a reason for despair, it is a reason that corroboration, structured facts and presence in those sources are worth more this year than last.
What to do this week
Check your Cloudflare settings before Monday. If you are on Cloudflare, open the AI crawler controls and look at all three categories, and specifically at whether the legacy block is still on. Decide deliberately whether you want to be training material, and know that saying no to training currently costs you Googlebot on ad pages.
Then look at what is reaching you. This is the part most brands still cannot answer, and it is why we built AI Traffic, which reports AI crawler activity observed at the edge of your site: which crawlers arrive, what they take, how much of your site none of them have read, and whether they honour the rules you set. In a week when defaults are changing across a quarter of the web, a configuration file is a statement of intent and your server log is the record of what happened. After Monday, the difference between those two things is worth watching closely.
We will be doing exactly that across the sites connected to us, and we will publish what changes.