For years, website owners had one blunt choice with AI crawlers: let them all in, or block them all. Cloudflare has now broken that switch into parts. On July 1, 2026, the company split AI bot traffic into three categories — Search, Agent, and Training — and gave every customer, including those on the free plan, the ability to treat each one differently. The change matters because Cloudflare sits in front of roughly 20% of the web, so its defaults shape what AI companies can take.
What the three AI bot categories mean
The old "Block AI Bots" toggle treated a search index and a model-training scraper as the same thing. They are not. Cloudflare's new classification separates them:
- Search — crawlers that index your pages so they can appear in search results and link back to you.
- Agent — bots acting in real time for a specific user, such as ChatGPT-User or an AI browser agent fetching a page on someone's behalf.
- Training — crawlers that collect content to train or fine-tune AI models.
The split matters because these three have very different value to a publisher. Search can still send readers. Training almost never does.
The September 15, 2026 default policy
The headline change is a default, not a feature. Starting September 15, 2026, domains newly onboarding to Cloudflare will have Training and Agent bots blocked by default on pages that display advertisements, while Search stays allowed. Cloudflare's reasoning is blunt: "An ad is a signal that a website owner meant for a person to land there."
One detail is worth reading carefully, because coverage differs slightly. Cloudflare's own announcement frames the new default around new domains, while TechCrunch reported that it will reach new customers, new sites, and eventually all free-tier users. Either way, existing customers are not forced into it — the settings can be changed, and opt-outs live in Cloudflare's Security settings. If you already run a site on Cloudflare, nothing flips on you overnight.
The numbers behind the change
Cloudflare's argument rests on how lopsided the exchange has become. In its Content Independence Day report, the company said AI training crawlers made up 52% of crawler requests as of June 2026, up from 22% in spring 2025. More than half of all internet traffic is now non-human. Some heavily crawled industries saw human traffic fall by about 40% in under a year.
There is also plain waste. Cloudflare says more than 50% of crawl traffic from well-behaved bots goes to re-fetching pages that have not changed — bandwidth a publisher pays for and gets nothing back from. To fix that, Cloudflare is testing a fourth, optional field called use inside Content Signals, the policy block that lives in a site's robots.txt (the plain text file that tells crawlers what they may fetch).
From Pay Per Crawl to Pay Per Use
A year ago Cloudflare launched Pay Per Crawl, which let publishers charge AI companies for fetching a page. It is now becoming Pay Per Use, which pays publishers when their content actually contributes value to an AI answer, rather than merely when a bot grabs the file. Early pilot partners include Ceramic.ai, testing a pay-per-query model, and You.com. Cloudflare has not published public rates for the new model.
Alongside it, publishers get an Attribution Business Insights dashboard and Answer Engine Optimization (AEO) analytics — AEO being the practice of tracking how your content shows up inside AI-generated answers rather than in a list of blue links. Cloudflare says participants can see top queries, snippet performance, and ranking positions. The company named beehiiv, Patreon, and Condé Nast among partners adopting the controls.
CEO Matthew Prince framed the urgency this way, as quoted by TechCrunch: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."
What this means for Filipino publishers and creators
This is one of the rare infrastructure stories that reaches Philippine site owners directly, because the tools are free-plan features, not enterprise-only ones — no local reseller, telco, or peso price is involved. Any Filipino blogger, news site, or newsletter already sitting behind Cloudflare can open the dashboard today and decide whether training bots get their work.
The local stakes were spelled out by Cloudflare's own ASEAN leadership. Kenneth Lai, the company's ASEAN vice president, told Philstar.com in September 2025 that "AI tools come and scrape the content and just go back and use the content for their own benefit. So the content creator gets absolutely nothing." He warned about the end state: "If we arrive at the day, when the internet stops having content, that's where the internet loses effectiveness."
The traffic math explains why that worries publishers here. Cloudflare Radar data covering the first eight months of 2025 found that among 14 industries tracked, news and publications were the most heavily crawled by AI bots for search queries — 21% of AI crawlers on news sites were fetching for search, while 65% were for training. Referrals barely came back: in that period, OpenAI's ChatGPT bots sent a visitor roughly once per 200 crawls, and Anthropic's Claude once per 5,400. For a Philippine site earning from views and display ads, that is the whole business model quietly draining out.
One caution: blocking is not automatically the right move for everyone. Search crawlers are what still send readers, which is exactly why Cloudflare left Search allowed in the new default rather than shutting the door completely.
What to watch next
The real test is enforcement and adoption. Blocking only works if crawlers identify themselves honestly, and Cloudflare's control over about a fifth of the web is leverage, not law. Whether Pay Per Use grows past pilot partners into something an independent Filipino publisher can actually earn from — rather than a deal reserved for large media groups — is the open question after September 15. Cloudflare's infrastructure has long been used to filter unwanted traffic at the network edge; our earlier look at how Cloudflare drops attack traffic in the kernel shows the same edge position now being pointed at AI scrapers instead of DDoS floods.
