Cloudflare introduces Bot Preference Sync to align robots.txt with AI bot enforcement

Cloudflare introduces Bot Preference Sync to align robots.txt with AI bot enforcement Cloudflare has launched Bot Preference Sync, a feature available across all subscription tiers that automatically updates a website's robots.txt file to match edge-enforced AI bot configurations for search, agent, and training traffic. Bot Preference Sync automatically prepends category-level rules for Search, Agent, and Training crawlers to robots.txt to match edge management rules. Mixed-use crawlers performing both search indexing and model training must fulfill transparency requirements to avoid edge blocking when AI training is disallowed. The sync feature is enabled by default for new users, offering an onboarding toggle for ad-supported publishers that defaults AI training to Disallow. Synchronizing edge enforcement with robots.txt Cloudflare introduced Bot Preference Sync to eliminate discrepancies where robots.txt directives and edge-blocking rules disagree, a gap that some automated crawlers exploit to bypass restrictions. The tool automatically maps dashboard configurations for Search, Agent, and Training bots into robots.txt directives, prepending generated rules to any existing file content to ensure declared crawler preferences match edge enforcement without requiring manual file maintenance. Transparency standards for mixed-use AI crawlers To address bots that combine search indexing and training functions under a single user agent, Cloudflare requires crawler operators to provide operational transparency regarding how ingested data is used. Mixed-use crawlers that meet verification and transparency standards are permitted to access content for search while honoring training opt-outs under the Disallow setting, whereas crawlers that lack transparency face outright edge blocking when training is disallowed. Onboarding defaults and customization options Bot Preference Sync is active by default for new domains without pre-configured blocks, though onboarding publishers with ad-supported models can select an option that defaults AI training to Disallow. Because the automated sync applies broad category policies derived from Cloudflare's BotBase directory rather than custom edge logic, domain owners with specialized crawler agreements or granular rule sets can disable the sync feature to manage their robots.txt manually.

Cloudflare 推出 Bot Preference Sync：自动将 AI 爬虫策略同步至 robots.txt Cloudflare 宣布推出面向所有层级客户的新功能 Bot Preference Sync。该功能可自动将控制面板中设定的 AI 爬虫分类策略同步更新至站点的 robots.txt 文件，确保公开声明的偏好与边缘实际执行的拦截规则保持一致。 Bot Preference Sync 适用于从免费版到企业版的所有客户，会在现有 robots.txt 内容前添加规则，避免声明偏好与边缘拦截规则不一致。 针对 Search、Agent 与 Training 流量分别提供控制选项，其中 Training 支持 Disallow 设置以限制模型训练。 执行 Search 与 Training 混合用途的爬虫若想在 Disallow 状态下抓取搜索索引，必须提供额外的透明度信息并被公开追踪，否则仍将被拦截。 新客户将默认开启同步功能，依靠广告变现的发布商可在接入时默认将 Training 设为 Disallow，其他常规新客户默认不施加任何拦截。 同步声明与实际拦截规则 以往站点 robots.txt 中声明的 Disallow 指令若与实际边缘拦截策略不一致，可能导致爬虫无视偏好或绕过规则。Bot Preference Sync 会根据用户在控制面板中对 Search、Agent 和 Training 流量的配置，自动生成或在原有文件前置添加 robots.txt 规则，并基于 BotBase 定期更新爬虫列表，使公开偏好与边缘防护保持同步。 细化 Training 选项与混合爬虫透明度要求 对于 Search 与 Agent 类别，系统提供允许、仅拦截广告页或全局拦截等选项；对于 Training，用户可选择 Disallow。为应对兼具搜索与训练功能的混合用途爬虫，Cloudflare 要求爬虫所有者提供额外身份与数据使用信息。满足透明度标准并经追踪验证的爬虫才能在 Disallow 下正常抓取搜索索引，不透明的爬虫仍会被直接拦截。 针对不同业务场景的默认设定 该功能面向所有客户推出且默认开启。依靠广告变现的发布商在接入时选择对应属性后，Training 将默认设为 Disallow，从而在保留搜索可见度的同时避免内容被用于模型训练；普通新站点默认不添加任何拦截。需要通过复杂自定义规则进行精细控制的用户，可随时关闭该同步功能自行维护文件。

*Update Sept. 15, 2026:* *We've published a follow-up blog post with the latest on these controls, including a new Accountable designation for Apple, Google, and Microsoft, as well as how the Block and Disallow AI Training settings now work together. Read the [latest blog](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) [post](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/).* We’re constantly building for the different goals of our customers. Some customers want to optimize for discovery, while others want to protect their content with the strictest security policy. Among these differing policies, there are multiple ways to mitigate bot traffic. Some mechanisms simply state your preference, assuming best intent from crawlers, and other approaches actually lock down content by outright blocking with a Bot Management solution. We recognize that it's cumbersome to maintain multiple layers of protection on your website. For example, there are cases in which your robots.txt states that a crawler is Disallowed from accessing your website, while your enforcement rules actually don’t block that crawler. When your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules. A couple of years ago, Cloudflare announced an easier way to disallow AI training on your website by tackling two of these layers: a managed value of robots.txt that told a fixed list of major Training crawlers not to train on your content, along with edge-enforced blocks to Training crawlers. On July 1, 2026, we launched easier options to manage different kinds of AI traffic use cases. You can say what you want to do about Search, Agent, and Training traffic on your website. We're announcing **Bot Preference Sync**, available to *all* customers from the Free tier to Enterprise. Bot Preference Sync reflects what you've set in your AI bot configuration by updating corresponding preferences to your robots.txt, and it can be turned on or off at any time. No more static file for one use case: we'll help you **tailor your robots.txt to reflect what you’ve already configured for different AI bot categories**. For years, the most pressing question in this space was: "Is my content being used to train AI models without my permission?" It's an important question, and it isn't going away. Alongside this, the questions we increasingly hear are about discoverability and engagement. How do I show up when someone asks an AI assistant something my site can answer? How much of my traffic is coming from AI crawlers versus real people? What content is actually driving referrals, and what is it worth? The answers differ by business model. Discoverability and engagement are key, top-of-mind issues for any businesses trying to thrive on the modern web, but the funnels for these are different: an e-commerce store may want everything crawled and trained on, so its products surface when a shopper asks a chatbot for "the best sofa for a small apartment." A publisher that monetizes pages with ads may want the opposite: stay in the search index that sends readers to the page, but keep its articles out of model training and, crucially, be able to verify that its content really wasn't used without permission. There's no single right answer, which is exactly the point. Your controls should reflect your strategy, which is why we've been building tools to give you visibility and choice at every layer. Bot Preference Sync ties these together, so the preference you *set* is the preference you *publish*. On July 1, 2026, we made the case that mixed-use crawlers, or “bots that blend search, agent use, and training behind a single user agent,” put site owners at a disadvantage precisely because they make it hard to separate what you want from what you don't. That's still true, and our position on Transparency for site owners hasn't changed. But there’s more than one way to approach Transparency. We want to reward the operators who are clear about their identity and how they are using the data they crawl. For purposes of bot [Verification](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/), the owners of **bots that perform both Search and Training will need to provide additional information** in order to not be blocked when “Disallow Training” is set. Those requirements are: Bots of leading AI models and service providers that meet these criteria are tracked publicly in the [AI bot transparency](https://radar.cloudflare.com/ai-insights#ai-bot-transparency) section in Cloudflare Radar, which includes examples in which best practices are honored, as well as when they are not. Crawlers that don't provide Transparency will *not* get the benefit of the doubt — they're still blocked when you disallow training. In other words, this is a way of making Transparency the price of admission. Bot Preference Sync is a new feature that keeps your robots.txt reflecting the AI bot preferences you've already set for Search, Agent, and Training on the Cloudflare zone-level dashboard. If a site owner already has a robots.txt file, the contents added by Bot Preference Sync will be *prepended to the* existing material, so any existing Disallow directives are maintained. Instead of a site owner maintaining a separate static file, Cloudflare generates or updates your robots.txt based on your configuration, so what you say to the world and what you enforce at the edge are kept in sync. For Search and Agent, the three options we announced on July 1 remain: Allow, Block on pages that serve ads, or Block everywhere. For **Training**, we’re refining the option to stop your content being used for training models with the **Disallow** option: **Disallow**: a "no training" preference is written to your robots.txt, so that cooperating mixed-use crawlers who take the extra Transparency step can still access your content for search indexing, since they’re allowing site owners to directly verify how their data is used. Cooperating crawlers honor the preferences in robots.txt, and your Search visibility for cooperating crawlers is unaffected. Let’s take the example below, in which someone has configured their AI bot policy to say “Allow Search, Allow Agents, Disallow Training.” Since this example site has Bot Preference Sync on, their robots.txt would prepend something like the following (which has been shortened and anonymized for the sake of the example): We’ll use bots that we track in [BotBase](https://developers.cloudflare.com/bots/botbase/) to periodically update the list of bots that is added to robots.txt when you choose to Block or Disallow a given category. The Verified bots that are classified as Search, Agent, and Training can be viewed at any time in our [public bots directory](https://radar.cloudflare.com/bots/directory). For all new customers, Bot Preference Sync will be *on* by default, to make it easier to manage blocks and preferences that reflect the same policy. For existing customers who are using the legacy managed robots.txt feature, we'll prompt you to review and confirm your preferences to transition to the new Bot Preference Sync upon its upcoming launch. Some customers may want or need to be more hands-on in stating their preferences, for example, if they have a special arrangement with a given company to which they want to grant an exception. Because Bot Preference Sync is designed to tackle policy decisions made *category-wide* rather than case-by-case, it will not directly read from individual custom rules with more complex logic. Customers with a more fine-tuned security policy always have the option to turn *off* the sync that sets group policies, and tailor their file to match their custom policy. We’re also making a change that allows publishers or ad-supported sites to have a different default from other site owners. We’ve created a default to make it easier for publishing sites that rely on ads and expect them to be reserved for human visitors. At the time of onboarding, such customers can select the option, “I monetize from pages with ads on this domain", which will set Training to Disallow as the default. (Customers have the choice to change this setting at any time.) This way, you stay in search while keeping your content out of model training. For the non-publisher case, new customers will *not* have any blocks or disallows added by default when they onboard a domain: the choice is up to the customer. You can choose if you want to block Search or Agent or Training at any point, but the starting point will *not* add any blocks on your behalf. Bot Preference Sync will be available to *all* customers, on every plan, in the coming week. Keep an eye on our changelog for availability, and watch your dashboard (and inbox) for the prompt to confirm your preferences! This is one step in a longer effort. We'll keep working with the large bot operators to make sure we're not compromising on familiar challenges (like training without consent) nor emerging questions (like discoverability and engagement). Beneath it all is our effort to promote greater Transparency and control for site owners.