Cloudflare, a leading connectivity cloud company, has announced a significant shift in how artificial intelligence (AI) interacts with online content. The company is rolling out new classifications, enhanced analytics, and strategic partnerships designed to empower website owners and foster a more equitable 'agentic Internet'. This initiative responds to the growing dominance of automated agents and bots, which now account for over half of all web requests, fundamentally altering how information is accessed and commerce is conducted online.
The core philosophy behind Cloudflare's new approach is simple: 'Your Content, Your Rules'. While many website owners wish their content to be discoverable by AI systems, those relying on advertising or subscriptions are concerned about their intellectual property being used for AI training without compensation. Cloudflare's new tools will allow businesses to easily opt-in to AI system access if they wish, or restrict it if they do not, ensuring a level playing field that optimises crawling efficiency, protects intellectual property, and enables creators and AI companies to thrive collaboratively.
A key focus of these changes is addressing 'mixed-use crawlers' – bots that blend search, agent use, and training without clear distinctions. Cloudflare argues that these mixed crawlers disadvantage both site owners, who are forced to choose between discoverability and giving away valuable content, and transparent AI companies, whose bots are clearly labelled by intent. The company highlighted that the largest search engine currently has access to roughly twice as much information as leading AI companies because it makes it difficult for customers to remain discoverable without also being used for AI purposes.
Cloudflare is giving mixed-use crawlers a deadline to adapt. Following extensive discussions with AI companies and site owners, the company will engage with the ecosystem over the next two months to gather feedback and conduct tests. By 15 September 2026, new default classifications will be finalised. For new customers and new sites for existing customers, the defaults will permit search but block training and agent use for pages displaying advertisements. Mixed crawlers failing to provide site owners with the choice between search, agent use, and training will be blocked entirely on all ad-supported pages. Existing free customers whose settings remain unchanged by 15 September 2026 will also see these new defaults applied.
Matthew Prince, co-founder and CEO of Cloudflare, emphasised the necessity of these changes. He noted that with non-human traffic now constituting the majority of internet activity, rapid action is crucial to establish a sustainable ecosystem. Prince expressed hope that the proposed default changes would encourage mixed-use crawlers to differentiate between search, agent use, and training, ultimately providing website owners with greater visibility and commercial opportunities, while benefiting transparent AI companies.