Cloudflare has introduced a new setting called Disallow AI Training, which enables website administrators to permit search engine crawling while blocking content usage for artificial intelligence model training. Major technology companies, including Apple, Google, and Microsoft, have committed to respecting this configuration.
When the feature is active, Cloudflare continues to allow crawlers identified as accountable, which are used for both search and AI training purposes. The company warns that blocking certain crawlers might negatively impact a website's search engine rankings.
Existing customers who previously selected Block or Block on pages with ads in their training settings will be automatically migrated to the new option. Most users do not need to perform any manual configuration to adopt the change.
Google plans to release a URL-level transparency tool called Google-Extended in the coming weeks. Apple is currently developing its own URL-level tool, while Microsoft has committed to supporting AI training opt-outs via robots.txt by early 2027.
