Cloudflare CEO Matthew Prince reported that automated traffic on the company's network exceeded human traffic in May 2026. Prince projects that non-human activity could reach 1,000 times the volume of human traffic by 2031, driven by AI agents performing tasks at high frequencies.
Data indicates that Google's crawl-to-visitor ratio has shifted from approximately 6:1 to 18:1. Training traffic for large language models now represents 40 to 52% of all AI crawler activity, marking a shift from traditional search indexing.
Starting September 15, 2026, Cloudflare will implement a default block on AI training and agent crawlers for ad-supported pages on new domains and many existing sites. Publishers must opt out if they wish to permit crawler access.
This policy change shifts the burden of negotiation from publishers to AI companies. Prince suggests that the industry may move toward a pay-per-crawl or micropayment model, where AI developers compensate content owners directly for data access.
