Google, Anthropic, Microsoft and OpenAI back new AI crawling opt-out standard

Google, Anthropic, Microsoft, and OpenAI have submitted a draft proposal to the Internet Engineering Task Force to introduce a standardized opt-out mechanism for artificial intelligence scraping. The proposed standard uses a shared token in website robots.txt files, replacing single-provider rules with a single setting. While Anthropic has already implemented the signal, OpenAI plans adoption in coming months, and Google and Microsoft target 2027. Meta expresses support without committing to implementation.

AI crawling opt-out standard proposal

• ▪The robots.txt AI crawling setting proposed by Kevin Kelley of Anthropic will not apply to artificial intelligence models that perform exclusively non-generative tasks
• ▪Google LLC, Microsoft Corp., Anthropic PBC, and OpenAI Group PBC announced plans to support a new upcoming website crawling standard proposed by Anthropic technical lead Kevin Kelley
• ▪Google, Microsoft, Anthropic, and OpenAI submitted a draft proposal for the AI crawling standard proposed by Kevin Kelley of Anthropic to the Internet Engineering Task Force (IETF)
• ▪The AI crawling standard proposed by Kevin Kelley of Anthropic uses a shared policy token within a website's robots.txt file to block scrapers collecting generative AI training data across multiple providers simultaneously

Robots.txt file complexity

• ▪Under historical robots.txt standards, website operators must add separate entries for every single artificial intelligence provider and scraper to restrict training-related web scraping
• ▪Website operators historically used text files titled robots.txt to manage how search engine web crawlers interact with web properties

Granular publisher controls

• ▪Google previously implemented controls allowing websites to determine whether content appears in AI search features or is used for AI training without affecting search visibility
• ▪Microsoft noted that web properties will continue to appear in Bing search engine results regardless of selected artificial intelligence preferences

AI grounding prevention features

• ▪OpenAI plans to offer granular controls enabling web publishers to opt out of artificial intelligence grounding requests
• ▪Microsoft plans to provide a granular crawling setting called AI Grounding to allow website operators to block requests when AI models access webpages to answer user queries