OpenAI, Google, and Microsoft will honor one robots.txt signal against AI training

OpenAI, Google, and Microsoft said on October 30, 2026, that they will honor a shared signal in robots.txt, the file a site uses to tell crawlers what they may fetch. The signal lets a site state once that it does not want its content used to train generative AI models, instead of writing separate rules for each company's crawler.

How the signal works

The mechanism is described in an Internet-Draft by Kevin Kelley of Anthropic, submitted on October 30, 2026. It adds no new directive. A site writes an ordinary robots.txt group under the name __usage_ai-training, a so-called policy token, and uses the familiar Disallow and Allow lines to mark where training is refused and where it is welcome.

The draft's own example refuses training for the whole site except one directory: User-Agent: __usage_ai-training, then Disallow: /, then Allow: /press-kit/. Paths are matched as in RFC 9309, the robots.txt standard, so the longest matching rule wins.

Three details change how existing files are read. A wildcard group, User-Agent: *, must not be treated as matching the token, so a general block on all crawlers does not count as a training preference. A file without the token expresses no preference at all. And the token governs the use of content, not crawling, so each crawler's own rules still decide what it may fetch.

Training has a narrow meaning here: using content to change the learned parameters of a generative AI model. The definition comes from the IETF working group called AI Preferences, and it excludes models used only for tasks such as classification, ranking, or scoring.

What each company has committed to

OpenAI says it aims to implement the signal in the coming months and will publish details when it does. Its existing crawler controls in robots.txt continue to apply.

Google says it aims to support the control in 2027, along with other AI providers. When a site uses the token to opt out and does not name Google-Extended, Google's existing training control, Google plans to treat that as if Google-Extended had been disallowed. Rules written specifically for Google-Extended keep working alongside the shared signal.

Microsoft's plan is the most detailed. Starting February 1, 2027, Bing will honor the shared training preference as it crawls. Microsoft also adds two controls of its own: a page-level training setting that overrides robots.txt, and a separate page-level preference for grounding, the use of a page to inform an AI-generated answer. Microsoft says neither choice changes whether a site can appear in ordinary Bing search results.

The draft has no formal standing yet. The IETF Datatracker lists it as an individual draft that the IETF has not endorsed, and the current version expires on May 3, 2027.

Deadlines for sites that use NoArchive or NoCache

Microsoft is retiring the controls it introduced for this purpose in 2023. The new preferences replace NoArchive and NoCache for training and grounding. After March 1, 2027, Bing will no longer read NoArchive and NoCache as a training preference, so a site that relies on them has to publish the new preference by then, in robots.txt or on individual pages.

For a single page, Microsoft accepts a meta tag or an HTTP header, for example <meta name="Bingbot" content="ai-training:disallow"> or X-Robots-Tag: Bingbot: ai-training:disallow. Grounding can only be set page by page. A site that wants to keep refusing grounding must publish the page preference and record a temporary preference in Bing Webmaster Tools before March 1, 2027.

Microsoft warns that refusing grounding, either with Disallow or with the narrower DisallowExtended, will significantly reduce a site's visibility in AI-generated answers.