Google and Microsoft back one robots.txt AI opt-out

What changed

Google and Microsoft announced support for the same idea. They said they will respect a shared signal in your robots.txt file that states whether your content may be used to train AI models.

Until now, you had to name each company's crawler and write separate rules. The Google Search Central announcement describes a single shared "policy token" that sets a broad preference across participating AI providers.

The signal comes from the AI Preferences working group at the IETF, the body that writes internet standards. It is a draft, not a finished standard.

It is not switched on yet

This is the part that is easy to miss. These are announcements of plans.

• Google: Search Engine Roundtable reports that Google plans to implement it in 2027.
• Microsoft: Bing's announcement says the new preferences take effect on February 1, 2027, as Microsoft crawls.

So if you add the line today, don't expect it to change anything this week. Think of it as getting ready.

Your old rules keep working. Google says its existing company-specific controls continue alongside the new signal.

What Bing adds: training and grounding as separate choices

Training means your content is used to build or improve an AI model. Grounding means an AI answer pulls from your pages while it is writing a response, so your site can be cited in that answer.

Bing now treats these as two decisions. According to Search Engine Roundtable's coverage, Microsoft is offering three settings:

• An AI Training setting, to allow or prevent use of your content for training.
• An AI Grounding setting, to allow or prevent use in generating AI answers.
• A Limited Grounding setting, a middle option: Microsoft's own AI products can still draw on your pages for answers, but outside services that rely on Microsoft's grounding cannot.

Microsoft says you stay eligible for Bing search results whichever you pick. My own read is that blocking grounding probably means fewer chances to be cited in AI answers, so weigh that before you switch it off.

Per Bing's post, both settings default to Allow when you publish no preference. Training preferences can also be set page by page. Temporary grounding controls are planned in Bing Webmaster Tools for eligible sites that use NoArchive or NoCache, and Search Engine Roundtable lists September 30, 2027 as the date those temporary preferences expire.

Who this affects

Every site with a robots.txt file, which is nearly everyone.

It matters most if you publish original writing, photos, product descriptions or guides that you would rather not see feeding a model. If you don't care, you can do nothing, because the default is to allow.

What to do this week

1. Open your robots.txt. Type yourdomain.com/robots.txt into a browser. If your site builder hides the file, find the setting in its SEO section or ask your developer.
2. Decide your training stance. Opt out, or allow. This is a business call. My opinion: if your content is your product, opt out. If you want every possible mention, leave it open.
3. If you use Bing, decide on grounding separately. Blocking training doesn't require blocking AI answers. Weigh being cited against control.
4. Don't delete existing rules. Google says its company-specific controls continue alongside the new signal, so there is no reason to remove what you have.
5. Wait for the exact wording. Read each provider's documentation before you paste in a directive. The standard is still being finalized at the IETF, and syntax could change. Microsoft's own announcement is a Bing post by Krishna Madhavan.
6. Put a reminder in your calendar for February 1, 2027. That is when Bing's preferences take effect, and a good time to confirm your file says what you intend.

One caution. A robots.txt line is a request that participating companies have said they will honor. It is not a lock. It only covers providers that choose to participate.

Sources