Cloudflare and OpenAI Share Early Results from AI Web Discovery Pilot

Cloudflare and OpenAI Share Early Results from AI Web Discovery Pilot Cloudflare and OpenAI reported preliminary results from a joint research pilot launched in July 2026, demonstrating how Cloudflare's network signals help OpenAI discover and index hundreds of millions of previously unknown web pages for ChatGPT search while reducing wasted crawler activity. A pilot using approximately 1% of Cloudflare's network enabled OpenAI to discover and index hundreds of millions of new web pages. Network signals help filter out bot-generated requests, nonexistent pages, duplicate content, and compromised individual pages before crawlers fetch them. The initiative uses web metadata strictly for search discovery and indexing rather than sharing page content for AI model training, leaving access control to domain owners. Expanded Search Discovery for ChatGPT The research pilot explored whether signals from Cloudflare's network could help OpenAI identify relevant web content more effectively. By analyzing a cohort representing about 1% of Cloudflare's total network, OpenAI discovered and indexed hundreds of millions of new pages, bringing niche and previously unindexed information into ChatGPT Search while directing users to original publishers via citations and links. Improving Crawling Efficiency and Web Safety Cloudflare's telemetry allows crawlers to bypass wasted effort caused by bot-driven traffic, empty redirect loops, and duplicate content. In addition, the partners are evaluating signals that identify compromised single pages—such as planted phishing forms—to keep harmful URLs out of search indexes without penalizing entire domains. These filters and update signals aim to reduce server load for site owners by limiting repeated crawling of unchanged pages. Publisher Control and Next Steps The pilot relies on metadata for search and discovery rather than utilizing site content for AI model training, ensuring domain owners maintain authority over crawler access through tools such as Cloudflare AI Crawl Control. Upcoming phases will focus on refining page selection, testing faster change detection, and measuring downstream business impacts for discovered sites.

Cloudflare 与 OpenAI 公布试点进展：利用网络信号提升 AI 网页发现效率 Cloudflare 与 OpenAI 公布了自 2026 年 7 月启动的研究试点初期成果。该合作利用 Cloudflare 约 1% 网络范围的信号辅助 OpenAI 爬虫，已帮助其索引数亿个新网页，同时减少无效抓取负荷并强化安全过滤。 试点仅使用 Cloudflare 约 1% 的网络规模，便助力 OpenAI 发现并索引了数亿个新网页。 OpenAI 依据相关性与实用性独立评估并排名，ChatGPT 搜索通过引用反向链接至原始信源。 双方利用网络信号识别恶意请求、重定向和重复内容，避免无效抓取，并精准标记有害页面。 项目仅使用网络元数据改善搜索与发现，不用于模型训练，站长仍可通过 AI Crawl Control 自主控制访问权限。 拓展发现渠道与搜索索引 Cloudflare 与 OpenAI 开展的研究旨在探索网络信号能否提升 ChatGPT 发现新内容的效率。试点涉及 Cloudflare 约 1% 的网络，目前已将数亿个此前未知的实用页面纳入 OpenAI 搜索索引。OpenAI 会独立评估并对这些页面进行排序，在 ChatGPT 给出回答的同时附带原始来源引用，帮助用户访问原网站。 过滤无效流量与精细化安全防护 抓取新内容面临爬虫无效消耗的挑战。双方通过识别自动化机器人造成的虚假热度、频繁重定向及重复内容，在爬取前滤除无用工作。此外，测试中的安全信号能够单独标记被攻击者植入的恶意页面（如虚假登录表单），在保障用户安全的同时，避免对合规站点的整个域名进行误惩。 站长控制权与后续演进 Cloudflare 强调，该合作仅使用网页元数据支持搜索与内容发现，并非共享内容以训练模型。站长可借助 Cloudflare 的 AI Crawl Control 工具掌控 AI 抓取行为。后续阶段双方将进一步优化页面筛选、测试更高效的变更信号，并评估内容被发现后的实际商业成效。

Optimizing your site for AI is of little use if AI can't find it. The answer to someone's question may already be on your website: the installation guide you spent a week writing, or a first-hand account of using a product, with the details that make the difference for whether they buy it. As more people turn to AI for help, you want those systems to find your work, represent it accurately, and send interested people your way. You also want control over how they access your site. Cloudflare and OpenAI [launched a research pilot](https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-research-pilot-with-openai/) in July 2026 with those interests in mind. We wanted to find out whether signals from Cloudflare’s network could help ChatGPT users discover new content more efficiently. Today, we’re sharing early results. So far, the answer is a resounding yes: the pilot has brought previously unknown pages into OpenAI’s search index, giving ChatGPT more sources to draw on to help people find relevant, timely information. Along the way, we’ve learned more about what makes a page worth discovering and how AI systems use the web. The web is full of specifics. Someone fixing a washing machine needs the manual for their exact model. Someone applying for a passport needs the current requirements and fees. Someone choosing a local restaurant needs its latest menu and opening hours. Finding the right page can be the difference between a wrong answer and one that actually helps. These pages can be valuable even if few other sites link to them. In the previous version of the web, discoverability was getting to the top of a search result page. It was then up to the human who visited your page to determine the value of the information, book that table, apply for that passport. As people increasingly rely on AI to find information and complete tasks, discoverability matters in new ways. AI systems need to find the right sources, understand what's useful, and connect people with relevant information – whether that's through an answer, a citation, or a visit to the website itself. Cloudflare's network provides signals that can help OpenAI’s search crawler discover more useful pages across the web to surface to ChatGPT users. OpenAI then independently evaluates and ranks how those pages appear based on how relevant and helpful they are. Together, this creates a feedback loop that can improve both discovery and the quality of information people find. This joint pilot, which used a participating cohort of \~1% of Cloudflare’s total network, has enabled OpenAI to discover and index hundreds of millions of new web pages, helping people discover more useful content and get more questions answered through ChatGPT. By linking back to relevant sources, ChatGPT Search also gives people a way to explore the original content and discover the websites and publishers behind it. With [GPT-6 and Intelligent UI](https://openai.com/index/gpt-6-for-everyone/), OpenAI is also making ChatGPT an even better place to shop, learn, and find answers, with more visual, interactive experiences tailored to what people need. This work complements these advances by helping ChatGPT discover more useful, up-to-date information from across the web. Finding a URL is only the start. We need to know whether it leads to a page someone can access, whether that page offers anything distinct, and what it might help someone do. We’ve learned, for example, how easily automated traffic can make a nonexistent page look interesting. Bots repeatedly ask for files that aren’t there. Some sites redirect almost any address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns gives us a way to cut wasted work before a crawler fetches the page, and improve efficiency in a way that benefits site owners. We're also testing signals that identify and flag potentially harmful pages, adding another layer of information to support safer, more reliable discovery. A legitimate site can have a single compromised page, such as a fake login form planted by an attacker. Flagging that one page, rather than the whole domain, keeps people safe without penalizing the site owner. Early results suggest these signals can help identify pages that should be kept out of an index. As more people use AI to discover information, website owners have new opportunities to reach interested audiences. For those who choose to make their content available to AI systems, these improvements make discovery easier and more efficient. Keeping up by fetching every page more often would be expensive for everyone. For a website owner, a crawler that repeatedly downloads an unchanged page adds load without bringing any benefit. The signals above are how we avoid that. Filtering keeps crawlers away from pages that don't exist or that redirect elsewhere. Change signals tell them when a page is worth another visit, and when it isn't. There's a balance to get right. Finding new pages can initially increase crawling as those pages are evaluated and indexed. The goal is to keep discovering and stay current while reducing the work that takes. For Cloudflare customers, that means less unnecessary load. For OpenAI, it means more of its search resources are spent on information that helps people. The next phase will build on what we’ve learned: improve the selection of pages, test faster and more reliable change signals, and measure what happens after content is discovered. We’re continuing to work with OpenAI to understand how better site discovery can translate into real business outcomes, and we’ll keep sharing what we learn along the way. Domain owners’ choices matter throughout. A signal about a page doesn’t grant permission to access it. This research uses metadata about the web for search and discovery, rather than sharing page content for model training. You can use [Cloudflare’s AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) to understand and manage how AI systems access your website.