OpenAI, Google, Microsoft, and Anthropic have committed to adopting a unified standard for AI web scraping. This initiative allows publishers to manage how their content is used for AI model training through a single policy token.
The proposed standard, introduced by Anthropic technical lead Kevin Kelley via the Internet Engineering Task Force, functions within a website's robots.txt file. This replaces the current requirement for publishers to configure preferences for each AI company individually.
Under the new policy, these four companies will restrict their crawlers from accessing sites that utilize the shared token. Microsoft intends to provide additional granular controls, enabling site owners to distinguish between AI training and AI grounding processes.
Despite these changes, Microsoft confirmed that websites will remain visible in Bing search results regardless of their specific scraping preferences. Anthropic has already implemented the signal, while OpenAI plans to follow in the coming months, and Google and Microsoft expect to provide support next year.
