🔍 Read the full analysis: A Practical Guide To The AI Training Opt-Out And Its Limits on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared robots.txt signal asking them not to use a site’s content to train generative AI models. The signal is voluntary, adoption dates vary, and it does not block AI answers based on live web content or remove material from models already trained.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they will honor a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The common rule could reduce the need for publishers to maintain separate training opt-outs for each company, but it is a voluntary request, not a technical block or an adopted internet standard.
The proposed rule uses the user-agent name __usage_ai-training, followed by Disallow: /, to express that a publisher does not want its site used for AI training. The rule appears in an Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Anthropic standards lead Kevin Kelley on October 30. Anthropic said it had already started implementing support, while OpenAI said it aims to implement the signal “in the coming months.”
Google said it aims to support the control in 2027; its crawler documentation specifies “early 2027.” Google also said that, if a site uses the new signal without a separate Google-Extended rule, it plans to treat the shared signal as a disallow for Google-Extended. Microsoft said its shared training preference will take effect as it crawls sites from February 1, 2027.
Microsoft is also introducing a separate page-level preference for AI Grounding—the use of web content to inform answers in Bing, Copilot and Microsoft 365. Microsoft says the new controls replace NoArchive and NoCache for training or grounding preferences after March 1, 2027. Existing company-specific crawler rules remain relevant while providers roll out the shared signal.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Simpler Signal, Not a Block
For website owners, a shared rule may make it easier to communicate a preference across participating AI providers. Instead of relying only on separate instructions for GPTBot, Google-Extended and ClaudeBot, publishers can add one common signal for training use. That may reduce maintenance, but it does not remove the need to check each company’s implementation and keep existing rules in place during the rollout.
The distinction between a request and a technical barrier matters. Robots.txt communicates a site’s preferences to crawlers; it does not itself prevent access. The source article describes the new signal as a preference, and Microsoft says its choices do not guarantee a site’s inclusion, traffic or payment. Publishers should not treat the change as a promise of compensation or as proof that every AI company will comply.
The scope also matters to readers and businesses deciding how their material can be used. The shared signal concerns training models. It does not, on its own, stop a company from crawling a page or using live web content to generate an AI answer. Microsoft’s separate grounding control addresses that use for its services, but the other announcements described in the source do not establish a shared, cross-company opt-out for AI answers.
Products Worth Considering
LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1
Samsung 27 Odyssey OLED G5 (G50SF) QHD QD-OLED Gaming Monitor
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
Company Rules Still Apply
Before this proposal, publishers commonly used separate crawler instructions for individual services. The new draft is intended to provide a shared way to state a training preference, while leaving existing controls in place. Google says its Google-Extended control does not affect Google Search; Microsoft says its preferences do not change a site’s eligibility for Bing search results. These statements distinguish AI-related controls from ordinary search inclusion.
The proposal is an individual Internet-Draft, not a finalized IETF standard. At an October 21 meeting of the relevant AI Preferences working group, one of its chairs said the group had not declared consensus on anything. The draft’s existence and company announcements confirm a proposed common mechanism and stated plans to support it; they do not establish that the internet standards process has adopted it.
Company commitments differ in strength and timing. Anthropic reported implementation work had begun, OpenAI described a coming implementation, and Google and Microsoft gave future dates. Meta spokesperson Andy Stone expressed support for publishers’ ability to state preferences and called a common signal an important step, but did not commit Meta to honoring it. The source article reports no statement on adoption from Apple, Amazon, xAI or Perplexity.
““has already started implementing support for this draft””
— Anthropic standards lead Kevin Kelley
Adoption and Scope Remain Open
It is not yet clear how broadly the signal will be honored beyond the four companies that announced support. Meta has voiced support without making an implementation commitment, and the source material reports no adoption statement from several other major technology companies. Even among supporters, the rollout schedule varies and Anthropic said it would take time to add support across its services.
The draft has no formal standing in the IETF standards process, and the working group had not declared consensus as of its October 21 meeting. The announcements also do not settle how publishers can verify compliance or what remedies, if any, would apply when a company does not follow a site’s preference. The signal does not undo use of material in models already trained, and its training scope should not be mistaken for a general ban on crawling or AI-generated answers.
Watch Rollout Dates and Standards
Publishers will need to track each provider’s implementation and retain company-specific crawler rules until the relevant services support the shared control. Microsoft’s stated date for beginning to honor its new preferences is February 1, 2027; the company says NoArchive and NoCache stop counting as training or grounding preferences after March 1, 2027. Google’s documentation points to early 2027, while OpenAI has not given a specific date in the source material.
The next developments to watch are whether the IETF working group reaches consensus, whether Meta or other companies make explicit commitments, and how providers document the signal’s practical scope. Until those details are clearer, publishers should treat the shared rule as an additional expression of preference—not as a guarantee that content will be excluded from training or AI answers.
Key Questions
What is the shared AI training opt-out?
It is a proposed robots.txt rule using User-Agent: __usage_ai-training and Disallow: / to tell participating providers that a publisher prefers its content not be used for generative AI training.
Does the signal block AI companies from crawling a website?
No. The signal communicates a preference; it is not a technical lock that prevents a crawler from accessing a site.
Does it stop websites from appearing in AI answers?
Not by itself. The shared signal described in the announcements concerns model training, not all uses of live web content in AI answers. Microsoft announced a separate AI Grounding preference for its services.
When will the companies honor the signal?
Timing differs. Anthropic said implementation had begun, OpenAI said it aims to support the signal in the coming months, Google aims for early 2027, and Microsoft plans to begin honoring it on February 1, 2027.
Is this an official internet standard?
No. It is an Internet-Draft and had not gained formal standing in the IETF standards process in the source material. A working-group chair said on October 21 that consensus had not been declared.
Source: ThorstenMeyerAI.com
