Cloudflare is disrupting the web economy by setting a September 15, 2026, deadline to block mixed-use AI crawlers on ad-supported pages by default. The policy forces AI companies to separate search indexing from model training. To help publishers monetize their work, Cloudflare is evolving its Pay Per Crawl tool into a Pay Per Use model with partners Ceramic.ai and You.com, aiming to establish a sustainable economic framework as non-human traffic surpasses 50% of all web activity.
Robots.txt standard ineffectiveness
- ▪The robots.txt standard, or Robot Exclusion Protocol, was first proposed in 1994 and was not officially codified until September 2022
- ▪A judge ruled in December 2025 that robots.txt does not qualify as an effective control mechanism for protecting copyright under the DMCA
AI crawler blocking strategies
- ▪Cloudflare will update its default settings on September 15, 2026, to block Training and Agent crawlers on ad-supported pages
- ▪Cloudflare's September 15, 2026, default blocking rules will apply to new customers, new sites, and existing free-tier customers
- ▪Cloudflare will block mixed-use crawlers that combine search and training if a website chooses to block AI training
Alternative content acquisition methods
- ▪OpenAI's crawler requests approximately 1,700 pages per referral, while Google's traditional search crawler averages 14 pages per referral click
- ▪Anthropic's crawler requests approximately 73,000 webpages for every single visitor referral it sends back to publishers
- ▪AI companies can obtain website content through the Internet Archive, massive datasets like Books3 or Common Crawl, and republishing sites
Cloudflare licensing tools
- ▪Ceramic.ai and You.com are the initial partners working with Cloudflare to test the Pay Per Use content licensing system
- ▪Cloudflare is evolving its Pay Per Crawl program into a Pay Per Use model to pay publishers when content appears in AI answers
- ▪Cloudflare is introducing an Attribution Business Insights dashboard to show publishers how AI bots access, cite, and refer traffic to their content
Internet collaboration breakdown
- ▪AI chatbot referrals drive roughly 96 percent less traffic than traditional search, and users click cited sources only about one percent of the time
- ▪Automated bots now drive more than half of all web traffic, a milestone that arrived earlier than expected
Story comments
Loading comments…