The hosting platform wants sites to have more control over how AI companies use their content.
Cloudflare has announced plans to automatically block custom web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously gave its customers the ability to choose to prevent crawlers from scrubbing their sites with AI chatbots, but now Cloudflare’s situation is getting more defensive by default.
“Now that the majority of traffic on the Internet is non-human, we must move forward and act quickly to create a sustainable ecosystem,” said Matthew Prince, CEO and co-founder in a statement. “Cloudflare’s new tools and collaborations give website owners increased visibility and commercial opportunities and benefit AI companies with bots with a clear and transparent purpose. We hope our proposed automation changes encourage mixed-use crawlers to separate search from using an agent and training.”
Web traffic used to be used to show whether people were viewing a website’s ads or paying for its subscriptions, but the popularity of AI models that can visit sites on a user’s behalf to pull up-to-date information has upended that system. Cloudflare’s new approach is an attempt to redefine the relationship in a way that is fair to both AI companies and anyone who uses the website.
Starting September 15, 2026, new customers and new websites from Cloudflare subscribers will automatically “allow search but block training and agent usage on pages with ads.” Hybrid browsers that don’t give site owners the option to choose whether their site is powered by AI will also be banned from pages with ads by default. Users with free accounts will also switch to this default unless they opt out before the September 15 deadline, according to the company.
As part of these changes, Cloudflare is also rolling out a new version of the Pay Per Crawl feature introduced in 2025 that allowed websites to automatically block AI web crawlers without companies being paid to crawl their content. The feature is now called Pay Per Use, and instead of basic payments based on whether a web page is advertised, Cloudflare says site owners will be paid if their content appears in responses from AI chatbots. The announcement only mentions partnerships with Ceramic.AI and You.com, but Cloudflare is likely hoping that other AI companies will join as its customers opt-in.
Without trying to make the relationship between websites and AI companies more fair, as TechCrunch notes, Cloudflare also appears to be targeting Google indirectly. The company’s announcement states that “the world’s largest search engine has access to 2X more information than the leading AI companies because they make it difficult for customers to continue to be found without the use of AI.” Google’s main search engine, Googlebot, both indexes the websites of the company’s various search engines and collects information to train Gemini and power AI features such as AI Overviews and AI Mode. Google allows websites to opt-in to a different search engine called Google-Extended which is only visible to websites with normal search results, but if the publisher wanted to be included in AI Mode results, but did not want their content to train Google’s models, they have no option. Cloudflare’s new policy is an attempt to force Google and other companies with hybrid search engines to change their tactics.
