Cloudflare has introduced new tools allowing website owners to block AI training bots while keeping their content indexed by search engines. Previously, site owners faced a dilemma where blocking bots for AI training also resulted in removal from search results, as the same bots performed both tasks.
The company launched an 'Accountable' status for bot operators. To qualify, operators must allow site owners to opt out of AI training, provide mechanisms to exclude content from search summaries, and ensure transparency regarding which URLs are accessed for training purposes.
Apple, Google, and Microsoft have agreed to meet these criteria. Cloudflare now offers a 'Disallow AI Training' setting in its dashboard. When enabled, bots with 'Accountable' status are restricted to indexing tasks only and are prohibited from using site content for model training.
Cloudflare categorizes bots into search indexers, AI training bots, AI agents, and mixed-type bots. While mass-scraping bots are now manageable, the company has not yet implemented specific controls for individual AI agents due to the lack of industry standards.
The company reports that over 50% of users read AI-generated search summaries, which increases the likelihood of ending a search session by 40%. However, traffic from AI summaries shows a 3–5 times higher conversion rate compared to traditional search results.
