Website owners who do not want their pages feed AI models will soon be able to say that once, instead of company by company. Google, OpenAI, Microsoft and Anthropic quit October 30 that they will be a common honor robots.txt signal, __usage_ai-trainingthe participating AI developers say whether the content of a site can be used Train generative models. The mechanism is described in an IETF Internet draft written by Anthropic’s Kevin Kelley, and none of the four have yet fully activated it.
Until now, selection meant sticking to a list of vendor-specific rules. OpenAI is looking for GPTBot, Google is using Google-Extended, and Microsoft has relied on the NoArchive and NoCache tags, which they introduced in 2023. Forget one, and that company could still train on your content.
A familiar file, a new kind of name
The robots.txt Syntax does not change at all. Instead of calling a crawler on the user-agent line, a site calls the policy token and then writes normal Disallow and Allow rules below it:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example comes straight from Kelley’s draft. It opts out of the entire site from AI Training but leaves the press kit folder open. You can apply the rule to an entire domain or only to certain directories, and the usual longest match precedence decides conflicts.
The token governs how content is used, not who is allowed to search it. Blocking AI training in this way will not stop Googlebot or Bingbot from crawling and indexing your pages for normal search results.
There is a catch site owners can easily cross. The Wildcard Group, User-Agent: *never counts as a training preference under the draft. A site that blocks everything for * didn’t technically say anything about AI training, so anyone who wants to opt out has to write the new token explicitly.
Company-specific rules continue to work alongside. The draft lets each AI developer treat its own tokens as more specific than the common one, so a publisher can decide broadly and still carve out exceptions for a particular company.
robots.txt examples you can copy
These follow the syntax in Kelley’s draft. They go in the robots.txt file at the root of your domain, and nothing in that file needs to change.
To keep the entire site from AI training:
User-Agent: __usage_ai-training
Disallow: /To opt-out anywhere except a press or media folder you’re happy to share:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/To protect only one section, such as paid tutorials, and leave the rest of the site open:
User-Agent: __usage_ai-training
Disallow: /premium/Do not delete your current vendor rules when adding the new group. Until each company confirms that its support is live, the older rules are the ones that are actually read, so a belt-and-braces file looks like this:
User-Agent: __usage_ai-training
Disallow: /
User-Agent: GPTBot
Disallow: /
User-Agent: Google-Extended
Disallow: /None of this touches normal search crawls. Googlebot and Bingbot always follow the rules you wrote for them, or for *.
Microsoft page level settings work differently and live in the HTML head of the page instead of robots.txt. One tag excludes a single site from the training, the other excludes them from the floor services that Microsoft provides to third parties:
On WordPress, the site serves a virtual robots.txt, unless a real file is placed in the root folder. Yoast SEO users can edit it under Yoast SEO > Tools > File Editor, or you can upload your own robots.txt via FTP or your host’s file manager, which replaces the virtual one. Either way, open yourdomain.com/robots.txt afterwards in a browser to make sure the new group is actually there.
Four companies, four different timelines
anthropic is the furthest along. Kelley wrote in his post to the IETF mailing list that the company has already started implementing support, but he added that it will take time to cover all Anthropic services.
OpenAI says it will comply with the signal in the coming months and publish implementation details when it happens. GPTBot and its other existing robots.txt controls remain in effect.
google aiming for 2027. In a Search Central blog post, Google Search Advocate John Mueller said that if a site uses the common token without explicitly calling it Google-Extended, Google plans to read it as Google-Extended Disallow. Publishers who specifically allow Google-Extended remain eligible to appear in Gemini Apps responses, and to receive clicks from them, through soil.
Microsoft has the fixed date. It will begin honoring shared training preferences on February 1, 2027, applying them as Bingbot recrawls each site.
Microsoft separates training from the floor
Microsoft also went further than the others. In addition to the common training signal, there is a separate AI Grounding preference that decides whether a page can be pulled to inform AI-generated answers in products such as Bing, Copilot and Microsoft 365.
- Allowedthe default, keeps the page eligible for grounding anywhere.
- DisallowExtended keeps it in Microsoft’s own AI experiences but from ground services Microsoft sells to other companies, such as Web IQ and the foundation API in the foundry.
- Prohibition prevents the page from being used to inform generated responses at all.
Microsoft extends the formatting preferences to individual pages as well, so a tag-level tag can override everything robots.txt says. Krishna Madhavan, principal product manager at Bing, wrote that limiting the floor will noticeably reduce a site’s visibility in AI responses, even though regular Bing search quality won’t be affected either way.
These settings replace NoArchive and NoCache, and sites that still use the older tags have a deadline. After March 1, 2027, Microsoft will no longer consider them training or prerequisites. To keep grounding blocked, a site must publish the new tag-level tag as well as record a temporary grounding preference in Bing Webmaster Tools, an option that opens on December 15, 2026. This temporary setting will last until Bing recrawls the page or until September 30, 2027, whichever comes first. Sites that default to it skip to Allow to be left until Bing picks up the new tags.
No standard yet
Kelley presented the proposal as an individual Internet draft, which the IETF says has no formal status, and it will expire in May 2027 unless it is updated. The broader conversation is happening in the IETF’s AI Preferences (AIPREF) working group, and the token borrows that group’s working definition of AI training: content used to change the learned parameters of a generative model. Models used purely for classification or ranking fall outside.
And robots.txt was always a request, not a block. Each AI company must still implement the token and choose to respect it, and the preference will only shape how the content is used from here on out. Microsoft, for one, describes its settings as the future model training.
For now, the wisest move is for publishers to add the common token and leave existing GPTBot, Google-Extended, and Bing rules where they are until each company confirms its support is live.