OpenAI, Google, Microsoft and Anthropic join forces to promote AI data scraping opt-out standard

IT House reported on November 1 that OpenAI, Google, Microsoft and Anthropic recently announced that they will adopt a new standard that will allow content publishers to choose to refuse AI to crawl their own website content. With this control mechanism, content publishers can declare their content usage preferences for AI model training to multiple AI companies at once, without having to set separate rules for each company. The standard was proposed to the Internet Engineering Task Force (IETF) by Anthropic technical lead Kevin Kelley and will be integrated into a website’s robots.txt file. For a long time, robots.txt files have been used to tell web crawlers which website content is allowed to be crawled and which is prohibited from crawling. Applicable scenarios include search indexing, web page archiving, and AI model training. Currently, websites must declare their AI training-related content usage preferences to each AI company individually. Against this background, various AI crawlers continue to crawl website content and input web page materials into AI model training. Content publishers are struggling with this. IT House notes that currently, websites need to separately declare training preferences to each company, and the new standard aims to simplify this process. Anthropic has been adopted first, OpenAI is planned to be implemented in the coming months, and Google and Microsoft are expected to support it next year. Meta spokesman Andy Stone also expressed support for the establishment of such universal signals, but did not specify specific implementation plans.