Cloudflare CEO Matthew Prince warns that the rise of AI agents is creating new cost pressures for website operators. These agents consume server resources and bandwidth by crawling sites to generate answers for users, yet they often fail to deliver actual traffic or revenue to the content owners.
Prince highlights that this trend threatens existing internet business models. For instance, an AI agent might scrape 1,000 restaurant menus to provide a single recommendation, forcing 999 sites to bear infrastructure costs without gaining a customer. Similar issues affect news sites, where AI summaries reduce the likelihood of ad impressions or subscriptions.
To address this, Cloudflare is offering tools to identify and manage AI crawlers. The company is exploring two main solutions: improving crawling efficiency through partnerships like the one with OpenAI, which notifies bots only when content changes, and implementing a payment mechanism for content usage.
Cloudflare is actively pushing for a model where AI services compensate publishers. Site owners can set specific policies based on the agent's intent, such as charging for content access when an agent performs research. The company has officially designated per-crawl and per-usage billing as a key product direction.
