Cloudflare updated its AI bot access rules on September 15, introducing a new 'Disallow AI Training' mode. The platform categorizes AI traffic into Search, Agent, and Training, allowing users to set specific blocking policies for each category across all service plans.
New domains now receive default settings where Search is permitted, while Training is restricted via the 'Disallow AI Training' mode. Existing sites have had their legacy 'Block AI Bots' settings migrated to the new category-based framework.
Research by Seensure indicates that following the update, blocking rates for training bots increased, while search and agent bots faced fewer restrictions. However, these findings are based on simulated requests using specific User-Agent strings rather than actual crawler traffic.
Cloudflare advises users to configure AI bot policies manually in the Security Settings dashboard. The company notes that blocking the Search category may negatively impact site visibility in AI-powered search results, a practice currently used by less than 1% of sites.
Experts recommend verifying actual bot access by reviewing server logs or the AI Crawl Control section in the Cloudflare dashboard. Relying solely on robots.txt or User-Agent strings is insufficient, as some crawlers may ignore these directives.
