Cloudflare has introduced a new feature called Disallow AI Training, which enables website owners to restrict the use of their content for artificial intelligence model training. The tool specifically addresses the issue of mixed-use crawlers that previously combined search indexing and data scraping for AI.
Under the new system, mixed-use crawlers are permitted to access a site only if they comply with the owner's choice to opt out of AI training. Apple, Google, and Microsoft have already met the necessary criteria or committed to doing so, ensuring their search functionality remains unaffected.
Other companies maintain separate crawlers for search and training, allowing Cloudflare to block the latter while preserving search visibility. Bryan Becker, director of product at Cloudflare, noted that the company has been engaging with crawler operators on this issue since July.
The initiative follows industry concerns regarding bot management. A PYMNTS Intelligence report indicates that nearly 90% of organizations currently face challenges in managing bot activity, highlighting the need for better control over how digital content is accessed.
