Cloudflare Introduces Setting to Prevent AI Training Without Sacrificing Search Discoverability

Cloudflare Introduces Setting to Prevent AI Training Without Sacrificing Search Discoverability Cloudflare has introduced a new Disallow AI Training control enabling website owners to stay indexed in search while preventing mixed-use crawlers from utilizing their content for AI model training. The feature relies on an 'Accountable' designation that tracks commitments and technical compliance from operators such as Apple, Google, and Microsoft. A single crawler frequently performs both search indexing and AI training, previously forcing websites to either allow training or risk complete removal from search. Cloudflare's new 'Disallow AI Training' directive operates via robots.txt preferences supported by compliant crawler operators. Apple, Google, and Microsoft are recognized under Cloudflare's Accountable designation, alongside separate crawlers from Amazon, Anthropic, Meta, and OpenAI. While Apple and Google already respect extended robots.txt rules without impacting search ranking, Microsoft is engineering domain-level robots.txt support targeted for early 2027. Cloudflare plans to roll out granular controls for AI Summaries by early next year to let site owners manage content exposure beyond blunt opt-outs. Resolving the Mixed-Use Crawler Tradeoff Website owners have historically faced a difficult compromise because major web operators use mixed-use crawlers that simultaneously crawl content for search indexes and AI training datasets. Cloudflare notes that while fewer than 1% of its sites block search bots, 17% choose to block training. To address this disparity without completely barring crawler access, Cloudflare established the Disallow AI Training setting, which communicates preferences through robots.txt rather than completely blocking traffic at the network perimeter. The Accountable Crawler Framework The mechanism relies on an Accountable designation for bot operators that provide transparency, respect opt-outs, and do not penalize search rankings. Apple, Google, and Microsoft meet these requirements, combining current tools with time-bound commitments. Amazon, Anthropic, Meta, and OpenAI also qualify because they operate distinct crawlers for search and training, allowing Cloudflare to block training traffic without jeopardizing search presence. Cloudflare tracks operator behaviors and compliance publicly via Cloudflare Radar. Platform-Specific Implementations and Timelines Implementation varies by operator. Applebot respects training opt-outs via robots.txt rules for Applebot-Extended, alongside nosnippet tags and paywalled labels, with URL inspection tools planned for next year. Googlebot honors Google-Extended directives in robots.txt alongside Search Console toggles and plans further URL-level transparency tools. Bingbot currently relies on NOARCHIVE meta tags and Webmaster Tools to exclude content from AI training without impacting search rankings; Microsoft is working on domain-level robots.txt support targeted for early 2027, meaning Cloudflare's robots.txt directive will not automatically apply to Bing until that integration launches. System Migrations and AI Summary Controls Existing Cloudflare configurations are automatically mapped into the new system, with previous Training Block settings transitioning to Disallow AI Training unless users explicitly choose to block mixed-use crawlers entirely. Starting September 15, onboarding sites receive recommended presets tailored to whether they rely on ad revenue. Cloudflare is also addressing AI Summaries, where Pew Research shows over half of consumers read summaries and become over 40% more likely to end their searches, even though referred traffic converts at significantly higher rates. The company aims to release tools early next year allowing site owners to control the exact proportion of content summarized.

Cloudflare推出新设置：允许搜索引擎爬虫索引但禁止AI训练 Cloudflare宣布推出“Disallow AI Training”功能，允许网站在保留搜索引擎索引能力的同时，拒绝混合用途爬虫将其内容用于AI模型训练。Apple、Google与微软等主要运营商均已支持或承诺支持该机制。 针对兼具搜索与训练功能的混合爬虫，Cloudflare引入“Disallow AI Training”，通过robots.txt指令实现保留搜索但拒绝AI训练。 Cloudflare设立“Accountable”认证标准，Apple、Google和微软均符合资质，且承诺禁止训练不影响搜索排名；Amazon、Anthropic、Meta和OpenAI则因分离了爬虫同样被归入合规范围。 仅不到1%的Cloudflare站点屏蔽搜索，但有17%屏蔽AI训练，针对广告变现网站，Cloudflare自9月15日起在新接入时提供更严格的预设策略。 Cloudflare计划明年初进一步支持站长精细控制AI摘要对其内容的引用程度，并推动ai-prefs等行业开放标准的建立。 解决混合爬虫的两难困境与Accountable标准 长期以来，大型互联网平台采用单一爬虫兼顾搜索与AI训练，使站长面临允许训练或失去搜索曝光的两难抉择。Cloudflare通过“Disallow AI Training”设置，结合网络层行为识别与robots.txt指令，对忽略规则的爬虫进行拦截。Cloudflare还定义了“Accountable”标准，表彰提供控制权与透明度承诺的爬虫运营商，并在Cloudflare Radar上公开追踪其合规表现。 主流科技巨头的支持与配置细节 Applebot支持通过robots.txt的“Applebot-Extended”规则退出训练；Googlebot支持“Google-Extended”指令并在站长平台提供开关；两家均明确表示禁用训练不影响搜索排名。Bingbot目前通过NOARCHIVE标签控制训练且不影响搜索权重，微软计划在2027年初全面支持robots.txt域名级“禁止训练”指令。此外，彻底选择“Block”将同时阻断Applebot、Bingbot与Googlebot的访问，包含搜索功能在内。 下一步规划：AI摘要的精细化控制 数据表明，超半数消费者会阅读搜索摘要，且阅读后结束搜索的概率增加40%以上，但AI搜索带来的访客转化率比传统搜索高出3至5倍。针对AI摘要对内容分发的深远影响，Cloudflare计划在明年初实现单点设置以控制内容在摘要中的展现比例，并与IETF等组织合作推动开放标准制定。

Without proper controls, website owners have long faced a difficult tradeoff: allow your content to be used for AI training, or risk losing discoverability in search. That tradeoff exists because some of the largest organizations on the Internet use mixed-use crawlers: a single crawler serving both search and AI training. Refuse one, and you refuse the other. Today, Cloudflare is announcing a new [Disallow AI Training](https://blog.cloudflare.com/bot-preference-sync/) setting that lets you easily stay indexed for search while refusing to let that same crawler train on your content. Apple, Google, and Microsoft honor or have committed (in a specified time frame) to honor this setting. Mixed-use crawlers were the hard part of the training question. AI Summaries are next. A site-wide yes or no is too blunt: how much of your content appears in a summary matters as much as whether it appears at all. An opt-out for AI summaries is already one of the requirements we've set for mixed-use crawler operators. By early next year, our goal is to let you control how much of your content is included — set once on Cloudflare, rather than with each operator separately. Most site owners want to be found: by humans, agents, and (good) bots. But a significant portion of the open Internet is funded by advertising, subscriptions, or direct relationships with visitors, and those models only pay when someone actually arrives. Almost every site owner considers Search beneficial: less than 1% of Cloudflare sites choose to block Search bots. Training, however, is a different story: 17% of sites choose to enable some mechanism to block training. This is exactly why we decided site owners needed more granular controls, rather than a one-size-fits-all “Block AI.” A robots.txt directive alone cannot solve this problem. Anyone can publish one, but it cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores it. A network can solve it, however: we publish the preference, identify who is crawling, classify why they are crawling, and block the ones that ignore it – then report what each operator actually does on [Radar](https://radar.cloudflare.com/ai-insights#ai-bot-transparency). But blocking removes a crawler. It doesn't change how crawlers behave. The better outcome is operators that don't make you choose at all. So since July, we've been talking to them directly. The response has been encouraging: almost all agreed that site owners should have control and transparency into how their content is used, and reassurance that their choices will be respected. To help site owners understand that, we created a designation: Accountable. The Accountable designation recognizes both capabilities available today and concrete commitments to deliver them. To qualify, a bot operator must meet or commit to meeting the following requirements: Apple, Google, and Microsoft all demonstrate that they meet the qualifications to be Accountable. Each combines capabilities available today with time-bound commitments for those still in development. The details of each of these companies’ crawlers are shared below. Cloudflare classifies bots by behavior, and a single bot can exhibit more than one behavior. Three behaviors are available as controls: A mixed-use crawler is a single crawler doing both Search and Training. Without controls, that combination creates the tradeoff described above: site owners cannot refuse one use without refusing the other. To avoid blocking Accountable mixed-use crawlers — the ones that don't force that tradeoff on website owners — we are introducing a new setting: Disallow AI Training. Disallow AI Training is named for the Disallow: directive it publishes in your robots.txt. Block and “Block on pages with ads” previously did not apply to mixed-use crawlers because blocking them could also affect search discoverability. Now that we have the new Disallow AI Training setting, Block and “Block on pages with ads” apply to *all* training crawlers, including mixed-use crawlers. Training, Search, and Agent controls are applied at the domain level. With the addition of Disallow AI Training, the available settings are: Disallow AI Training works by publishing a preference in robots.txt. An ads-only preference cannot be expressed that way: Cloudflare can detect which pages serve ads, but that list is too large and changes too frequently to enumerate in robots.txt. That's why there's no Disallow AI Training on pages with ads. Agents do not create the same search-discoverability tradeoff as mixed-use crawlers, and the Internet does not yet have a well-established directive for expressing Disallow preferences to agents. For now, we’re not including a Disallow setting for Agents. As standards such as [ai-prefs](https://datatracker.ietf.org/wg/aipref/documents/) mature, we will revisit this approach. We are making the following changes to Bot Management and AI Crawl Control: Nothing, in almost every case. Your current settings carry over on their own. If you want mixed-use crawlers gone entirely, you now have to say so. Select Block. It will stop Applebot, Bingbot, and Googlebot from reaching your site — search included. Site owners that never configured the more granular controls will be migrated to the new settings based on their legacy Block AI Bots setting: For domains that previously configured the granular controls, we will preserve the practical effect of their selections under the new definitions. Previous Training selections of Block or Block on pages with ads will migrate to Disallow AI Training. Beginning September 15, customers onboarding a new domain will be offered one of two preset configurations, depending on whether the site earns money from advertising. Ad revenue depends on a human actually seeing the page. Training replaces that visit with an answer; agents fetch the page with nobody there to see the ads. So the presets for ad-supported sites are more restrictive. You can change any of these settings during onboarding, or at any time afterward. *Recommended settings for new domains* Applebot, Bingbot, and Googlebot are Accountable. Apple, Google, and Microsoft are committed to the same principles of publisher choice and transparency. Under Disallow AI Training they can keep crawling your site for search. Selecting Block stops them entirely. We also categorize the relevant crawlers from Amazon, Anthropic, Meta, and OpenAI as Accountable. These organizations separate their Search and Training crawlers, so Cloudflare can block the Training crawler without affecting search. Applebot allows site owners to opt out of training by adding a Disallow rule to robots.txt for “Applebot-Extended”. Site owners can also currently express preferences for AI Summaries via their nosnippet [directive](https://support.apple.com/en-us/119829#:~:text=nosnippet%3A%20Applebot,products%20and%20services.) in the page HTML. Content can also be labeled as [paywalled content](https://support.apple.com/en-us/119829#:~:text=Marking%20paywalled%20content,the%20next%20section.) to exclude it from generative output. Applebot does not yet provide a tool for URL-level inspection. However, we have met with their team, and they have shared details of their in-progress solution for next year. Apple has also stated that disallowing training [does not impact search ranking](https://support.apple.com/en-us/119829#:~:text=Applebot%2DExtended%20and%20controlling%20data%20usage). Googlebot allows site owners to opt out of training by adding a Disallow rule to robots.txt for “Google-Extended”, and they provide a toggle inside their webmaster portal to exclude a site’s content from generative search results. Googlebot also provides site owners with metrics and reporting regarding search results and AI summary results. Google shared information about their existing and recently launched controls, as well as information about what they're already working on, including additional URL-level transparency tools for site-owners related to Google-Extended, which they expect to launch in the weeks to come. Google has also stated that disallowing Google-Extended [does not impact search ranking](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended:~:text=Google%2DExtended%20does%20not%20impact%20a%20site%27s%20inclusion%20in%20Google%20Search%20nor%20is%20it%20used%20as%20a%20ranking%20signal%20in%20Google%20Search.). Bingbot provides granular controls and transparency in their [Webmaster Tools](https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c). Site owners can currently express AI training preferences through Bing’s `NOARCHIVE` [meta tag](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat#:~:text=Content%20tagged%20NOARCHIVE%20will%20not%20be%20included%20in%20Bing%20Chat%20answers%2C%20not%20be%20linked%20to%20in%20the%20answers.%20Going%20forward%2C%20for%20content%20in%20our%20Bing%20Index%20that%20is%20labeled%20NOARCHIVE%2C%20we%20will%20not%20use%20the%20content%20for%20training%20Microsoft%E2%80%99s%20generative%20AI%20foundation%20models.). Microsoft is extending these capabilities and currently building the mechanism to also respect a “no training” preference in robots.txt at the domain/site level, targeted for early 2027. For Cloudflare Customers who wish to opt out of training in Bing today, in addition to using the `NOARCHIVE` tag, site owners can use the [Block URLs or Content Removal tool](https://www.bing.com/webmasters/help/block-urls-from-bing-264e560a). Microsoft has also stated that using `NOARCHIVE` [will not impact search ranking](https://blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat#:~:text=We%20also%20heard%20from%20publishers%20that%20they%20want%20to%20exercise%20these%20choices%20without%20impacting%20how%20Bing%20users%20can%20discover%20web%20content%20on%20Bing%E2%80%99s%20search%20results%20page.%20We%20can%20assure%20publishers%20that%20content%20with%20the%20NOCACHE%20tag%20or%20NOARCHIVE%20tag%20will%20still%20appear%20in%20our%20search%20results.). Until that support launches, selecting Disallow AI Training will not automatically convey a no-training preference to Bing through robots.txt. This is the same practical behavior as the previous Training Block setting, which did not apply to mixed-use crawlers such as Bingbot. We will continue to reach out and engage with all operators of AI crawlers as these capabilities evolve. Cloudflare [Radar](https://radar.cloudflare.com/ai-insights#ai-bot-transparency) publicly tracks the controls, transparency, and reporting provided by Accountable crawler operators. Making the Internet better requires both sides to have agency: crawlers need access to the open web, and the people who create that web need meaningful control over how their work is used. Today’s announcement represents concrete progress toward that balance. Progress requires infrastructure providers, content creators, technology companies, and standards bodies such as the Internet Engineering Task Force (IETF) working together to translate these principles into open, interoperable standards. Training and AI Summaries raise different questions for site owners. Training concerns whether content can be used to build AI models. Summaries affect how people discover, evaluate, and ultimately visit a business. Both matter, but they affect businesses in different ways. Controls to opt out of AI summaries are the first step. The operators identified as Accountable either provide or are completing work to provide that capability, establishing an important baseline: site owners can say no. But a site-wide choice between allowing and prohibiting summaries is still a blunt instrument. The right decision depends on the site, the content, and the business outcome. For publishers, training raises foundational questions about control, compensation, and the sustainability of original content. Summaries create a separate and often more immediate distribution question: does someone visit the publisher’s site, or consume the answer within a search or AI experience? For many other businesses, AI summaries increasingly sit between a potential customer and a website. They may answer a question, compare alternatives, recommend a product, or help someone decide whether to visit at all. The data illustrates mixed impact. [More than half](https://www.pewresearch.org/chart/a-majority-of-americans-say-they-read-ai-summaries-at-the-top-of-search-results/) of consumers read summaries in Search, and those consumers are [over 40% more likely](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/) to [end their search](https://www.bain.com/insights/goodbye-clicks-hello-ai-zero-click-search-redefines-marketing/) after reading one. This can reduce the number of visits a website receives. But consumers referred by AI Search convert at between [three times](https://aisearch.similarweb.com/blog/ai-visibility-roi/) and [over five times](https://quickseo.ai/blog/ai-search-vs-google-search-in-2026-40-stats-that-show-why-your-brand-needs-to-track-both#:~:text=AI%20search%20traffic%20converts%20at%2014.2%25%2C%20compared%20to%20Google%E2%80%99s%202.8%25) the rate of those referred by traditional search. AI may produce fewer visits while sending customers with much greater intent. That is not inherently good or bad. A publisher funded by advertising may optimize for audience volume. A retailer may prefer fewer visitors who are more likely to purchase. Cloudflare’s role is not to choose for them, but to provide the visibility and control needed to make an informed decision. Summary opt-outs are a strong start, but they are not the end state. Our next focus is helping site owners understand how summaries affect their businesses and giving them more control over how much of their content can be used. Open standards such as [ai-prefs](https://datatracker.ietf.org/wg/aipref/documents/) will be an important part of making that possible. If you would like to have a voice in this conversation, or provide feedback, please reach out to [crawlercontrols@cloudflare.com](mailto:crawlercontrols@cloudflare.com). These new controls are available to all customers, on all plans, and can be configured at the domain (zone) [Security Settings](https://dash.cloudflare.com/?to=/:account/:zone/security/settings). Not on Cloudflare yet? [Start for free](https://www.cloudflare.com/lp/pg-one-platform/) to set the traffic controls that you want today.