🔍 Read the full analysis: What Changes When You Set An AI Training Opt-Out? on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have announced plans to recognize a shared robots.txt signal asking that a website’s content not be used to train generative AI models. The signal is voluntary, will take effect on different schedules, and does not itself prevent crawling, affect previously trained models, or opt a site out of AI answers based on live content.
OpenAI, Google, Microsoft and Anthropic said on October 30 that they plan to recognize a shared instruction in a website’s robots.txt file asking AI providers not to use its content to train generative models. The move offers publishers a common signal instead of requiring a separate training opt-out for each participating company, but it is voluntary and does not block all AI uses of a site’s material.
The proposed instruction is a user-agent group named __usage_ai-training, followed by Disallow: /. The format appears in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published October 30 by Anthropic standards lead Kevin Kelley. It is not yet an adopted internet standard.
Companies have described different implementation timetables. Anthropic says it has already started implementing support, though it expects service-wide rollout to take time. OpenAI aims to implement the signal in the coming months. Microsoft says it will honor the training preference as it crawls sites from February 1, 2027. Google aims to support it in 2027, with its crawler documentation specifying early 2027.
The existing company-specific controls are not immediately replaced. OpenAI says its current crawler controls will continue to apply. Microsoft says its new preferences will replace NoArchive and NoCache as training or grounding preferences after March 1, 2027. Google says that, when a site uses the shared signal and has no separate Google-Extended rule, it plans to interpret the new instruction as a Google-Extended disallow.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
For publishers, the practical change is a chance to express one training preference in a format several major providers say they will recognize. That may reduce the need to maintain a different training instruction for every company, although sites should keep their existing crawler rules while providers roll out support.
The signal does not function as access control. The source account describes it as a request rather than a lock: it records a publisher’s preference but does not technically stop a crawler from retrieving pages. Its effect depends on companies choosing to honor it and on their systems applying it correctly. The announcement does not establish a shared enforcement process or specify consequences for ignoring the preference.
It also distinguishes training from other uses. A training opt-out does not, by itself, block crawling or use of a page to produce an answer based on live web content. Microsoft announced a separate page-level preference for “AI Grounding,” which it says covers content used to inform answers in Bing, Copilot and Microsoft 365. Publishers may need to make separate choices about training, AI answers and search; the shared training token does not settle all three.
The change does not provide payment, guarantee that a site will be included in a service, or remove its content from models that were already trained. Microsoft explicitly says neither preference guarantees inclusion, traffic or payment. Those limits matter for publishers weighing whether to add the rule: it expresses a choice about future training use, not a general remedy for every concern about AI systems.
Products Worth Considering
Logitech MX Keys S Wireless Keyboard, Graphite
Dell UltraSharp 27 Thunderbolt Hub Monitor (U2726DER)
Apple iMac 2024 All-in-One Desktop with M4 Chip, 24‑inch Retina Display, 16GB Unified Memory, 256GB SSD, Silver
As an affiliate, we earn on qualifying purchases.
Different Controls, Different Timetables
Before the shared proposal, site owners commonly used provider-specific directives. Examples include GPTBot for OpenAI, Google-Extended for Google and ClaudeBot for Anthropic. Microsoft has also used page-level NoArchive and NoCache tags. A shared token is intended to make the training preference easier to communicate across services, but it does not make those older controls irrelevant during the rollout.
The proposal is associated with the IETF’s AI Preferences working group, but publication as an Internet-Draft does not make it an official standard. At a working-group meeting on October 21, a chair said, “We haven’t declared consensus on anything yet.” The draft’s existence and the companies’ public plans are confirmed; formal adoption through the standards process is not.
Company commitments also vary in scope and certainty. Anthropic says implementation has begun, while OpenAI and Google describe intended future support and Microsoft has given a start date. Meta communications chief Andy Stone voiced support for publishers’ ability to express preferences and called a common signal an important step, but did not say whether Meta will honor it. The source account reports no reviewed statement from Apple, Amazon, xAI or Perplexity.
““has already started implementing support for this draft””
— Anthropic
Adoption and Coverage Remain Open
Several practical questions remain unresolved. The shared signal is still an individual Internet-Draft, and the working group has not declared consensus. The source material does not establish when or whether the draft will become an official IETF standard, or whether its wording will change during that process.
Beyond the four companies that announced plans, adoption is uncertain. Meta has expressed support for the concept but made no commitment to honor the signal. The source account found no statements from several other major technology companies, but that does not establish what those companies will decide later. Even among announced supporters, implementation dates and service coverage differ.
It is also unclear how publishers will verify that every relevant service has applied the preference, or what recourse they would have if a provider does not. The signal does not cover every form of AI use, and the distinction between model training and grounding or live-answer systems may require separate instructions. Readers should not treat the shared token as a blanket opt-out from AI processing.
Watch Rollouts Through Early 2027
Publishers can add the proposed training group if they want to state that preference, while retaining provider-specific rules until each service’s support is in place. They should check the companies’ crawler documentation and their own robots.txt behavior as dates approach; the announcements do not indicate that every service will switch on the same day.
The next dated milestones in the announcements are Microsoft’s planned start on February 1, 2027, and its stated change to the role of NoArchive and NoCache after March 1. Google’s documentation points to early 2027, while OpenAI has not given a precise date beyond “the coming months.” Further developments will depend on implementation updates and the IETF working group’s consideration of the draft.
Key Questions
What does the shared AI training opt-out do?
It adds a robots.txt instruction expressing a preference that participating companies not use a site’s content to train generative AI models. It is a voluntary signal, not a technical block on access.
Does it stop AI companies from crawling a website?
No. The signal does not itself block crawling. Website owners may need separate crawler rules if they want to restrict access, and the effect of the shared training preference depends on companies honoring it.
Does the signal opt a site out of AI-generated answers?
Not by itself. The proposed signal concerns model training, not every use of live web content in AI answers. Microsoft has announced a separate preference for AI Grounding.
When will the companies honor the signal?
Anthropic says implementation has started; OpenAI aims for the coming months; Microsoft says February 1, 2027; and Google aims for 2027, with its documentation saying early 2027. The dates and rollout scope differ.
Is this an official internet standard?
No. It is an individual Internet-Draft associated with the IETF AI Preferences work. The working group had not declared consensus as of its October 21 meeting.
Source: ThorstenMeyerAI.com
