Robots.txt AI: the new anti-training standard in 2026

On October 30, 2026, Anthropic, OpenAI, Google and Microsoft announced their support for a new robots.txt standard that lets any website refuse, in a single line, to have its content used to train their AI models. For an SME publishing a blog, product pages or case studies, this announcement changes how to protect content in practice, provided its limits are well understood.

In short

• On October 30, 2026, Anthropic, OpenAI, Google and Microsoft backed a shared robots.txt token, named __usage_ai-training, to express an opt-out from AI training (source: IETF draft draft-kelley-aipref-training-preference-signal, written by Kevin Kelley of Anthropic).
• The token is added to the existing robots.txt file, with no new syntax: it follows the existing Robots Exclusion Protocol rules (RFC 9309).
• Rollout is gradual: Anthropic says it has started honoring the token, OpenAI commits to applying it "in the coming months," Microsoft targets February 1, 2027, and Google targets 2027.
• At Google, opting out this way is treated the same as disabling Google-Extended, which also cuts off Gemini's grounding on your site's content.
• The token has no retroactive effect on data already collected, and does not stop crawlers that ignore robots.txt altogether: it is a declaration of intent, not a technical shield.

What happened on October 30, 2026

Until now, a site that wanted to opt out of training for one AI or another had to block each provider's specific bot one by one: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. For an SME, that means tracking a list that grows with every new model launch, and redoing that list each time.

Anthropic proposed a simpler solution: a single token, added to robots.txt, that the four major providers commit to honoring the same way. The proposal was submitted to the IETF (the body that standardizes Internet protocols) as an individual draft, draft-kelley-aipref-training-preference-signal. At this stage, the document has no formal standards status: it expires in May 2027 unless renewed, and still needs to go through broader standardization work within the IETF's AIPREF working group.

How the new token works

Technically, nothing changes in the structure of the robots.txt file. The __usage_ai-training token is used like an ordinary bot name, in a User-Agent line. Example given in the official draft:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Here, the site refuses AI training on all its content except the /press-kit/ folder. One important point: a generic User-Agent: * rule does not count as a training preference. The token must be named explicitly for the signal to be recognized.

Provider rollout timeline

Old system vs new standard

Why this matters for an SME

Most SME leaders don't publish content out of concern for AI visibility, but they do have a blog, product pages, or hand-written customer case studies that represent a real investment. Two concrete use cases:

1. Protecting editorial know-how. A consulting firm that publishes detailed sector analyses may want that content to remain a competitive edge, rather than feeding, for free, the models used by its own competitors.
2. Keeping control over published customer data. Customer testimonials or case studies containing sensitive figures deserve an explicit policy, even though most SMEs will choose to leave their marketing content open to stay visible in generative answers.

Action plan to set up the token

Summary table

FAQ

Does the __usage_ai-training token also block Google indexing?

No. It only affects AI model training. It does not change regular crawling or indexing by traditional search engines, except at Google, where it specifically cuts off Gemini's grounding on the pages concerned.

Should my SME activate it right now?

Not necessarily. If your marketing content benefits from being cited by generative AI, it is often better to leave it accessible. Reserve the token for high-value content you don't want competitors using the same AI tools to pick up for free.

Is this standard already an official norm?

No. It is an individual draft submitted to the IETF (draft-kelley-aipref-training-preference-signal), with no formal standards status to date. The four providers have committed to honoring it, but the enforcement timeline varies from one to another through 2027.

What happens to content already used by AI before this announcement?

The token has no retroactive effect. It does not erase data already incorporated into existing training sets or already-released models.

Conclusion

This new robots.txt standard simplifies a step that, until now, required tracking four separate bot lists. For an SME, the point is not to block everything by reflex, but to make an informed choice between visibility in AI answers and protection of high-value content. To go further on generative AI's place in your content strategy, check our resources or see how other SMEs have structured their use of AI in our customer stories.