Google, OpenAI and Microsoft Back One Shared robots.txt Opt-Out for AI Training

On October 30, 2026, Google, OpenAI and Microsoft each said they will support one shared robots.txt signal that lets a website state whether its content may be used for generative AI training. Instead of a separate rule for every AI provider, a site owner will be able to express that preference once, in a form participating providers have agreed to read the same way.

The timetable is staggered. OpenAI describes its commitment as voluntary and says it aims to implement the signal in the coming months. Google says it aims to support the new control in 2027. Microsoft has given the firmest date: its crawler will start honoring the new preferences on February 1, 2027, as it crawls. Adding the new line today is preparation, not an instant block.

Background: one file, many different rules

Until now, a website operator that wanted to keep its pages out of AI training had to identify individual crawlers and maintain separate rules for each provider. Google says the shared control will help websites express their preferences across multiple AI providers in a single step, instead of managing separate instructions for every AI developer.

Google already offered its own training control, Google-Extended, since 2023. Both OpenAI and Google say existing company-specific controls will continue alongside the new signal. The shared approach comes from the Internet Engineering Task Force (IETF) AI Preferences working group, known as AIPREF. Its technical document is an Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Kevin Kelley of Anthropic and dated October 30, 2026. It is an individual draft, not an approved IETF standard, and it expires on May 3, 2027, unless revised or adopted.

Key details: how the token works

The mechanism reuses ordinary robots.txt grammar, with no new directive names and no new grammar, according to the draft. A site adds a group naming a shared policy token, __usage_ai-training, where a crawler name would normally appear. The draft’s example opts out a whole site while leaving one folder open:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

That example expresses a preference against AI training for all paths except those under /press-kit/. A publisher can therefore limit an opt-out to part of a site.

Two details are easy to get wrong. First, a catch-all wildcard group does not count: the draft states that wildcard groups, meaning User-Agent: *, must not be interpreted as matching the policy token. If no group names the token, no preference has been expressed. Second, the token governs use, not access: the draft states that the policy token is not a directive for crawling behavior. A site that wants a crawler to stay away still needs a rule for that crawler’s own name. Training means using an asset to modify the learned parameters of a generative AI model, excluding models used for exclusively non-generative tasks.

What each company promised

OpenAI says it will honor a common signal in robots.txt for foundational generative AI training, will share implementation details later, and encourages other AI providers to honor the signal. Google says it will support the shared control in 2027. Its rule is specific: when a website uses the policy token to opt out of training and Google-Extended is not explicitly specified, Google plans to interpret that as if the publisher had disallowed Google-Extended. Websites that specifically allow Google-Extended remain eligible to appear in responses and receive clicks via grounding in Gemini Apps, according to Google.

Microsoft goes further. Alongside the shared AI Training preference, it is introducing a separate page-level AI Grounding preference, covering whether content can help shape AI answers in products such as Copilot and Microsoft 365. Its options are Allow, Disallow, and DisallowExtended. Microsoft says none of these choices changes eligibility for ordinary Bing results, and neither choice guarantees inclusion, traffic, or payment. Its transition calendar: temporary grounding preferences open in Bing Webmaster Tools on December 15, 2026, for sites using NoArchive or NoCache tags, and sites relying on those older tags must move to the new preferences by March 1, 2027.

Anthropic is tied to the effort through Kelley, listed with an Anthropic affiliation. Reporting says it has started implementing support, but it has published no separate announcement, so its timing should be treated as reported.

What it does not do, and why it matters

The shared token is a preference, not a technical barrier and not a legal right. The draft says private agreements may override preferences expressed through the mechanism. It also ties the preferences that apply to an asset to the robots.txt file in effect when that asset was retrieved, so the signal looks forward from the point each company switches it on. It does not, by itself, remove material already collected. It also does not settle AI answers: a training opt-out does not by itself control whether a page is retrieved to inform an AI-generated answer, which is why Microsoft treats grounding as a separate choice.

For publishers, the immediate value is maintenance, not enforcement. One shared token replaces a growing stack of per-company rules. The sensible step now is conservative: keep existing per-company controls in place, because they are the only ones with live effect until each company’s start date arrives, and add the shared token as an additional group. Do not merge it into a wildcard group, and do not delete existing controls in the expectation that the new token already works everywhere.

Sources

• OpenAI, “Making it easier for websites to express AI training preferences,” October 30, 2026
• Google Search Central Blog, “Supporting a shared AI training control for websites,” October 30, 2026
• IETF, draft-kelley-aipref-training-preference-signal-00, “Machine-Readable Training Preference Signal,” October 30, 2026
• PPC Land, “Google, Microsoft, OpenAI and Anthropic back one AI training opt-out,” October 2026 (Microsoft dates, grounding options, and Anthropic implementation reporting)
• TSN, “AI Training Opt-Out: Google, OpenAI, Microsoft Sign On,” October 2026 (Microsoft transition dates and grounding details)
• Unite.ai, “OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences,” October 2026
• Unite.ai, “Google Plans Support for Shared Robots.txt AI Training Control,” October 2026