CNN Deploys AI on 3.5M Epstein Files to Expose DOJ Redaction Failures

CNN利用AI审查350万份爱泼斯坦档案 揭露美司法部涂黑脱敏失误 尽管AI爬虫和计算机视觉极大地加速了文档摄入与视觉信息发掘，但记者仍必须提供行业专业认知，以判定相关发现是否确实属于全新内容。 CNN记者详细介绍了他们如何从传统的人工“阅览室”转向能够处理数百万份混乱政府记录的AI加速工作流程。面对不规则的分页、年龄验证门槛和请求频率限制，数据团队部署了自动监控机器人和AI辅助的Python爬虫来抓取这批文档，并将其导入Google Pinpoint进行实体映射和快速主张核实。调查团队还与Visual Layer合作分析了伪装成PDF的多媒体文件，最终揭露了美国司法部普遍存在的涂黑脱敏失误，导致敏感个人数据和未成年人照片遭到泄露。尽管AI在基础设施和检索方面发挥了强大的倍增作用，但相关发言人强调，在建立背景关联以及判定披露内容是否真正属于全新发现方面，记者的领域专业经验仍然不可替代。 面对美国司法部公布的 350 万份杰弗里·爱泼斯坦调查文件，传统的人工“阅览室”排查模式已无法负荷，CNN 转向引入 AI 驱动的工作流与自动化系统来处理海量数据。 (4:26) CNN 利用 Google Pinpoint 对印刷体和手写文档执行 OCR 识别与实体提取，极大加速了对突发网络爆料及线索的核实效率。 (4:26) 数据团队编写 Python 脚本每分钟轮询司法部网页变动并触发即时 Slack 告警，并借助 AI 代码生成快速调整爬虫、绕过年龄验证弹窗及通过 VPN 轮换 IP 应对反爬限制。 (13:32) 针对被司法部伪装成 PDF 的多媒体文件，CNN 与 Visual Layer 合作对 10 万多份图片和视频进行语义视觉搜索，排查未被索引的影音证据。 (22:16) 计算机视觉分析曝光了司法部严重的涂黑遮盖失误，包括外泄的护照、医疗记录以及未遮盖的未成年人照片，促使司法部在收到 CNN 质询后 24 小时内下架并替换了文件。 (23:52)

According to CNN editorial leaders Ryan Struyk, Marshall Cohen, and Matt Stiles, covering the Department of Justice's 3.5-million-document Jeffrey Epstein release required shifting from manual document reading rooms to AI-assisted data engineering and visual analysis. While AI tools accelerated scraping, entity extraction, and visual auditing, the journalists emphasized that human reporting and institutional memory remained indispensable for verification and context. Scaling Past Manual Review with Google Pinpoint (4:26) Marshall Cohen contrasted the Epstein investigation with earlier document dumps, such as the 2016 Hillary Clinton email releases and the 500-page 2019 Mueller report. While the Mueller report was handled by a 12-person team reading line by line to compile 77 documented lies, that brute-force manual model collapsed when facing the DOJ's 3.5 million Epstein files. To manage the volume, CNN deployed Google Pinpoint to unify records into a searchable workspace, running Optical Character Recognition across typed text and handwriting while automatically extracting entities such as names, locations, and organizations. Beyond indexing, Pinpoint accelerated verification when viral claims circulated on social media, allowing reporters to corroborate or debunk assertions against the source material. CNN also used AI to parse correspondence chains and map Epstein's network of associates. Cohen noted that AI served as an analytical launchpad rather than the final editorial product: journalists still performed standard reporting, contacting individuals for comment before publishing stories that prompted public fallout, resignations, and lawsuits. Automating Web Scraping and Bypassing Ingestion Obstacles (7:25) Matt Stiles explained that editorial review was preceded by an engineering challenge: the DOJ site lacked a bulk-download option, distributing millions of files across tens of thousands of paginated index pages. To avoid manual monitoring, CNN deployed automated Python scripts checking site fingerprints on one-minute intervals, triggering a Slack alert when the release dropped on Friday the 30th. The technical team then deployed dual collection scripts: a crawler mapping PDF links alongside a brute-force script querying sequential document IDs. Data acquisition encountered substantial technical friction. After CNN pulled roughly 1 million files, the DOJ introduced an age-verification gate requiring an automated click for every subsequent PDF. To bypass aggressive bot blocking and rate limiting, Stiles rotated IP addresses via VPN in bursts of 200 requests and throttled downloads to single-thread sequences. The team used generative AI coding assistants to draft utility functions and address edge cases in seconds, while logging systems maintained an audit trail of tranches, missing files, and post-publication government deletions. Computer Vision Exposes Systematic DOJ Redaction Failures (23:52) CNN encountered anomalous formatting in the DOJ dataset when a producer discovered that video files had been mislabeled and uploaded with `.pdf` extensions. To analyze the multimedia corpus without manual conversions, CNN partnered with the AI vision platform Visual Layer to index and search more than 100,000 images and videos, enabling journalists to conduct semantic visual queries and track facial features across unindexed files. This visual exploration revealed widespread DOJ redaction failures that had been publicly accessible online for nearly a month. The exposed records included sensitive personally identifiable information—such as unredacted passports, driver's licenses, and medical records displaying dates of birth and home addresses—as well as unredacted photographs of children and toddlers. Following inquiries from CNN, the Department of Justice removed the offending files and substituted properly redacted versions within 24 hours. Machine Latency and the Challenge of Historical Context (26:48) The panel outlined distinct boundaries to AI utility during live breaking news. Ingestion latency meant Google Pinpoint could not provide instantaneous answers during the opening hours of the document drop. Facing broadcast deadlines and social media speculation, CNN initially reverted to manual triage by assigning physical page ranges across a team of reporters while backend processing concluded. Furthermore, AI tools like Google Pinpoint and Gemini could not evaluate whether an excerpt was genuinely new. Because the Epstein legal history extends back to 2008, the release mixed newly disclosed internal FBI files with long-public court records from civil suits and the Ghislaine Maxwell trial. Lacking domain context, AI models could not distinguish fresh revelations from recycled exhibits. CNN relied on veteran beat reporters and outside specialists, such as Julie K. Brown of the Miami Herald, using their institutional knowledge to guide targeted semantic queries. The speakers emphasized that because the release encompasses 3.5 million files, exhaustive review was impossible within weeks of publication. As major newsrooms continue to analyze the cache alongside external innovations like Jmail—which reconstructed fragmented email PDFs into threaded inboxes—the corpus is expected to yield investigative reporting for months and years to come.

在面对美国司法部公布的350万份Jeffrey Epstein调查文件时，CNN团队利用AI辅助代码生成、Google Pinpoint以及Visual Layer计算机视觉平台，重塑了海量数据抓取、实体检索与图像审查工作流。CNN团队强调，AI主要作为技术底座和检索跳板，核心的事实核查、历史背景研判与报道把关仍高度依赖记者的专业经验。 从人工审查室到AI驱动的实体解析 (2:07) 传统调查报道在应对政府大型文件解密时，主要依靠人工“审查室”逐页翻阅。例如在2016年大选前的希拉里·克林顿国务院邮件解密，以及2019年近500页的Mueller报告中，CNN曾组织12人团队逐行核对并梳理出特朗普关联人士的77处谎言。然而，当司法部在2025年底至2026年初分批公布高达3.50万份Epstein档案时，单纯的人工肉眼排查模式彻底失效。为此，CNN引入了针对记者的AI工具Google Pinpoint，将数百万份杂乱记录统一归集到一个可检索的工作区中。 该工具对机打文本与潦草手写笔记进行了光学字符识别（OCR），并自动提取人名、机构与地点等结构化实体，替代了以往分散的协作表格。每当社交媒体或同行流出线索时，编辑部能通过AI检索瞬间定位原始文件完成交叉比对，同时利用AI解析通信链条以绘制Epstein的关系网络。资深记者Marshall Cohen指出，AI只是调查的跳板，而非终审结论；记者仍须落实外联采访与传统核实，后续报道直接促成了相关人员辞职、公众问责以及数百万美元的法律诉讼。 自动化爬虫工程与AI辅助代码开发 (7:25) 在开展采编分析前，数据团队必须攻克文件获取难题。司法部公开网站并没有提供批量下载按钮，而是将文件分散在跨越数万个索引页的复杂分页架构中。高级数据与制图编辑Matt Stiles与同事开发了轻量级Python脚本，以1分钟为间隔轮询司法部目标网址，对比页面指纹变化，在周五发布上线的瞬间向Slack发送实时预警。为获取文件，团队同时启用了遍历所有PDF链接的爬虫与按文件编号并行请求的暴力爬取脚本。 抓取过程中，司法部的服务器机制制造了多重阻碍：后续增加了针对每个PDF的强制18岁年龄确认点击、针对并行请求的封禁与速率限制。CNN通过VPN手动轮换IP，以每批200个请求的小规模脉冲式运行，并将下载节流为单线程。面对突发反爬限制，数据团队将AI作为代码编写助手，在数秒内生成边缘情况处理函数，将数周的开发周期压缩至数小时。此外，团队建立了集中式审计日志，精确记录每个批次的缺漏状态（例如通报第9批次缺失10万份，同时拉取第10批次），确保了采编端对文件完整性的全面掌握。 多媒体异常挖掘与视觉AI揭露遮盖失效 (22:16) 在完成文本抓取后，CNN团队遇到了文件格式异常：司法部在公开门户中将大量非文本格式包装在PDF扩展名下。一名制片人将文件后缀手动修改为.mp4后，意外提取出视频证物。为处理海量未经索引的图像与视频，CNN与视觉智能平台Visual Layer合作，对司法部公开门户上的10万多份图像和视频进行语义视觉搜索与人脸特征比对，排查其中涉及的人物与证据。 这一计算机视觉探索揭露了司法部严重的隐私遮盖失效，且相关敏感内容已在公网暴露近一个月。被公开的内容不仅包括未打码的护照、驾照、医疗记录、出生日期及家庭住址，还包括年轻女性的露骨照片，以及未打码的儿童和幼儿照片（包括一名小女孩亲吻Epstein面颊的画面）。CNN在《The Situation Room》节目中对模糊处理后的材料进行报道，并在周一就敏感外泄问题向司法部质询。到周二，司法部在24小时内下架了违规文件并替换为合规遮盖版本。 突发新闻的时效瓶颈与历史语境局限 (26:48) 尽管AI工具重塑了数据处理方式，但在实时突发新闻中暴露了局限。在350万份文件集中公布的最初数小时内，Google Pinpoint等系统的后台数据摄取与OCR索引存在耗时延迟，无法即刻给出结论。面对社交媒体断章取义的传播与电视直播的即时需求，CNN不得不回归“传统战术”，安排记者团队按页码区间（如1到100页）进行人工分工粗筛。 更为核心的技术瓶颈在于，Google Pinpoint和Gemini等AI模型无法解答新闻采编的核心问题——“这是否属于全新信息”。Epstein的法律纠纷跨越近20年（可追溯至2008年定罪及后续Maxwell案审判），司法部解密档案将全新的FBI/DOJ内部文件与多年来早已公开的诉讼陈述混杂在一起。AI缺乏足够的历史领域背景来辨别某份材料是惊人内幕还是2021年已公开的法庭证据。采编团队必须依赖Julie K. Brown等资深跑口记者的专业记忆与外部线索，再结合AI检索定位原始文件以完成最终研判。 媒体行业技术生态与Jmail邮件重构 (30:40) 面对Epstein文档，各媒体与独立开发者构建了不同的技术应对机制。《纽约时报》和BBC开发了专有的内部可检索数据库，而民间团队也设立了公共归档项目。由于该批文件涉及高度政治敏感性（包含针对唐纳德·特朗普等公众人物的频繁指涉），司法部在发布后曾出现文件被撤回或修改的情况。CNN借助AI编程快速搭建的监控日志与本地镜像，确保了在政府修改或删除文件时能精准追踪差异并保留底稿。 在传统新闻编辑室之外，旧金山的一组年轻软件工程师利用Cursor、Claude和Reduct等AI开发者工具构建了名为“Jmail”的独立网页界面。司法部原始解密文件将邮件往来拆散为孤立的PDF单页，导致阅读上下文极度断裂。Jmail将这些分散的PDF文件重构为具备完整回信链条、类似Gmail的收件箱界面，该平台在上线首月据称获得了约5亿次浏览量，展现了外部技术生态对非结构化政府公开记录的二次处理能力。 CNN团队总结认为，350万份文件的庞大规模意味着任何机构都无法在数周内完成穷尽式报道，各家媒体的深度挖掘仍在持续。在涉及未成年人犯罪等极高法律与道德风险的调查中，CNN将AI严格限定于抓取监控、数据索引与视觉搜索等辅助环节，新闻报道的真实性核验与价值判断依旧由专业记者主导。

## Background 该视频由人工智能行业大会 Ai4 发布于 YouTube 平台，记录了 CNN 采编与技术团队的专题对谈。节目围绕 CNN 人工智能创新总监 Ryan Struyk、资深调查记者 Marshall Cohen 与高级数据编辑 Matt Stiles 的实战复盘，详细介绍了新闻团队如何借助 AI 工具分析挖掘 350 万份爱泼斯坦（Jeffrey Epstein）调查档案，并探讨了 AI 工具的技术边界与人工事实核查的必要性。 ## TL;DR 面对美国司法部公开的 350 万份格式混乱的爱泼斯坦案调查档案，CNN 团队告别了传统的全人工翻阅模式，构建起集自动化脚本监控、并发抓取、Google Pinpoint 实体提取与 Visual Layer 多模态语义检索于一体的 AI 协同采编工作流。借助图像 AI 检索，团队敏锐发现了司法部公开文件中针对受害者隐私及未成年人的严重打码遗漏，并推动官方紧急下架整改。然而，AI 在突发事件中仍面临数据摄取延迟以及无法判断线索“是否为全新未公开事实”的认知盲区，必须依靠资深记者的领域经验与详尽的审计日志完成最终核实。整场分享生动展现了新闻机构如何将 AI 作为“战斗力倍增器”，在严守事实真实性底线的同时高效开展深度调查。 ## Atmosphere & style 对谈整体洋溢着一线新闻调查的紧张感与数据工程实战的极客氛围，三位嘉宾以务实、严谨且生动的叙事风格拆解了海量文档背后的技术攻防。分享中既有深夜监控告警、动态绕过反爬机制的生动细节，也有曝光司法部打码漏洞时的职业自豪感。整体讨论在肯定 AI 巨大提效价值的同时，始终贯穿着对新闻真实性、事实核查与职业伦理底线的敬畏与冷静反思。 ## Key takeaways - [01:46](https://www.youtube.com/watch?v=yR197k0xGyE&t=106s) 面对美国司法部公开的 350 万份爱泼斯坦案调查档案，传统的人工逐页翻阅模式已无法应对，CNN 借助 AI 与现代数据工程工具实现了采编工作流的代际革新。 - [07:25](https://www.youtube.com/watch?v=yR197k0xGyE&t=445s) CNN 数据团队部署了 Python 自动化监控机器人，通过每分钟轮询页面数字指纹，在司法部深夜释出档案的第一时间向 Slack 推送警报。 - [11:54](https://www.youtube.com/watch?v=yR197k0xGyE&t=714s) 面对司法部缺乏一键下载机制及频繁收紧的防爬虫机制（如动态年龄验证与 IP 封锁），团队采取智能爬取、并发穷举与 VPN 轮换等极限应对策略完成全量抓取。 - [17:13](https://www.youtube.com/watch?v=yR197k0xGyE&t=1033s) 团队将 AI 严格定位为辅助编写脚本和设想边界异常的“初级开发助手”，坚持全流程审计日志记录，坚决不让 AI 独立撰写报道或做最终数据分析。 - [04:26](https://www.youtube.com/watch?v=yR197k0xGyE&t=266s) 通过引入 Google Pinpoint 进行全量 OCR 与实体提取，记者能瞬间定位社交媒体爆料的真实性，并高效绘制出爱泼斯坦庞大社交网络的人际脉络图。 - [22:16](https://www.youtube.com/watch?v=yR197k0xGyE&t=1336s) 司法部将多媒体文件伪装成 PDF 格式发布，CNN 与 Visual Layer 合作利用多模态 AI 语义检索与人脸识别，成功挖掘出隐藏的视频与图像线索。 - [23:52](https://www.youtube.com/watch?v=yR197k0xGyE&t=1432s) CNN 借助视觉语义搜索曝光了司法部档案中的重大打码疏漏（包括未成年人面孔、裸照及公民隐私证件），成功迫使司法部紧急下架并重新合规打码。 - [27:08](https://www.youtube.com/watch?v=yR197k0xGyE&t=1628s) AI 在处理突发海量档案时存在数据摄取延迟，且无法判断“哪些信息此前从未公开过”，这一认知盲区必须依靠资深法庭调查记者的领域经验来弥补。 - [33:22](https://www.youtube.com/watch?v=yR197k0xGyE&t=2002s) 民间年轻开发者借助 Cursor 和 Claude 等工具搭建的 Jmail 平台，成功将打散的数万单页 PDF 重新组装为高仿真实邮件线索，有效解决了上下文断裂痛点。 - [35:33](https://www.youtube.com/watch?v=yR197k0xGyE&t=2133s) 350 万份爱泼斯坦卷宗规模庞大，各大主流媒体与调查人员目前仍借助新型 AI 与数据分析工具持续深挖并推出新的独家报道。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=yR197k0xGyE&t=4s) 会议开场与嘉宾介绍：AI 在 CNN 新闻编辑部的深度应用 CNN 人工智能创新总监 Ryan Struyk 担任本场对谈的主持人，他负责统筹 CNN 编辑部内部的新闻采编 AI 倡议、采编工作流优化，以及在符合新闻伦理规范的前提下对记者进行 AI 工具培训。Ryan 隆重介绍了两位资深采编同事：驻华盛顿资深记者 Marshall Cohen 与数字视觉新闻团队高级数据兼图表编辑 Matt Stiles。Marshall 自 2015 年加入 CNN 以来，长期深耕特朗普涉俄调查、国会山骚乱后续、特朗普与亨特·拜登的司法诉讼等重大政治与法律报道，曾荣获 1 项艾美奖及 3 项提名；Matt 曾入围普利策奖，此前供职于《洛杉矶时报》《华尔街日报》、NPR 和《德克萨斯论坛报》，目前带领跨国视觉记者团队负责前瞻与执行数据驱动的新闻报道。本次对谈的核心议题是 CNN 如何借助 AI 驱动的新工具，应对并报道涉及 350 万份文件的 Jeffrey Epstein（爱泼斯坦）调查档案这一重磅事件。 ### [01:46](https://www.youtube.com/watch?v=yR197k0xGyE&t=106s) 从“吃披萨通宵翻页”到 AI：海量泄露档案报道模式的代际演进 面对政府或司法部门突发公开的数千甚至数百万份文件（即俗称的“文档倾卸” Document Dump），新闻记者的核心挑战在于如何以极快且准确的方式筛选出具备新闻价值的核心信息。Marshall 回顾了 AI 介入采编之前的传统操作模式：在 2016 年大选前夕报道希拉里·克林顿的邮件门事件、2019 年报道近 500 页的穆勒报告（Mueller Report），以及跟进 1 月 6 日国会山骚乱调查委员会对数百位证人长达数万页的访谈实录时，编辑部只能依靠“人工阅读小组”一页一页地点阅。当时为了在封闭的会议室里啃下海量纸质或扫描件，编辑部往往通宵达旦地派发披萨作为唯一的慰藉。在穆勒报告期间，CNN 曾组织 12 人的团队逐页精读，最终手动梳理出特朗普团队陈述的 77 处谎言并制作成易于公众理解的内容。 ```mermaid flowchart TD accTitle: 新闻编辑部处理海量调查文档的模式演进 subgraph TRADITIONAL["传统人工时代 (Pre-AI)"] A["司法部 / 国会释放海量文档"] --> B["组建多人人工阅读团队"] B --> C["逐页点击翻阅 + 共享文档手动记笔记"] C --> D["耗时数天至数周产出汇总报道"] end subgraph AI_ENABLED["AI 协同时代 (爱泼斯坦档案)"] E["350 万份混乱格式文档"] --> F["自动化脚本监控与抓取"] F --> G["Google Pinpoint 统一建库与 OCR"] G --> H["实体与人物关系网自动抽取"] H --> I["记者人工核实与多方置评采访"] I --> J["即时核实竞品线索并高频发布报道"] end ``` ### [04:26](https://www.youtube.com/watch?v=yR197k0xGyE&t=266s) 工具赋能：Google Pinpoint 与爱泼斯坦人物关系网的快速构建 面对司法部释出的 350 万份爱泼斯坦调查档案，单纯依赖人工阅读已完全不可能。CNN 团队引入了 Google Pinpoint 工具。尽管该工具在资深技术人员眼中并非前沿的高深黑科技，但它对新闻实操带来了质的飞跃：它将分散混乱的文件统一收录到同一个检索空间，对所有文本及手写笔记执行 OCR 识别，并全自动提取出人名、机构与关键实体清单。此外，这一系统彻底改变了突发舆情的应对节奏——当高管在社交媒体（如 X / Twitter）上看到同行或自媒体曝出猛料并质问“我们有没有这条、内容是否属实”时，记者无需再在数千页文档中盲目翻找，而是能瞬间定位原文并进行交叉核验。 - **复杂社交网络图谱绘制**：借助 AI 提取并梳理出的电子邮件往来与短信记录，视觉新闻团队得以绘制出爱泼斯坦庞大社交圈的关系脉络图，清晰呈现谁在与谁联络。 - **严格恪守采编规范**：AI 仅作为线索挖掘的起点，记者在发布报道前依旧执行传统核实流程，逐一向涉案当事人求证并纳入其回应声明。 - **巨大的公众影响力与产出**：海量的内容输出引发了强烈的社会反响，后续报道推动了多起商界高管辞职、政客下台以及数亿美元诉讼的推进。 ### [07:25](https://www.youtube.com/watch?v=yR197k0xGyE&t=445s) 数据工程的极限挑战：混乱数据集的自动化抓取与分钟级监控系统 从数据工程角度来看，获取这 350 万份文件本身就是一项艰巨的挑战。司法部公布的并非规整清洗过的数据集，而是极其混乱、难以预测且附带极端分页系统的 PDF 集合——文件总计达 300 万份以上，单页与多页混杂，底部的翻页组件甚至延伸至上万页。Matt Stiles 指出，数据团队在新闻报道启动前必须建立严格的审计与追踪机制：必须精确记录抓取进度、确保能够随时回溯缺失文件，并监测司法部是否中途撤回了某些敏感文档以探究被撤原因。 为在第一时间捕获文档释出信号，团队摒弃了“让人工守在司法部官网不断按 F5 刷新”的笨办法，而是部署了一套基于 Python 的自动化监控与采集系统。该脚本以每分钟一次的频率轮询司法部预估可能发布文件的目标页面，通过计算并比对页面指纹（Fingerprint）和文档集数量的变化，一旦探测到更新便会立即向编辑部的 Slack 频道推送警报。该系统在上线前经过了针对其他司法部站点的严密模拟测试，确保告警机制万无一失。 ### [11:54](https://www.youtube.com/watch?v=yR197k0xGyE&t=714s) 构建自动化监控机器人与实战演练 在爱泼斯坦调查文件正式公开前的几周里，新闻编辑室充斥着各种关于发布时间的传闻与谣言。团队经常在深夜 11 点左右接到突发电话：“你醒着吗？听说今晚可能要发，你能上线吗？”为了结束这种彻夜守候的不确定状态，CNN 数据团队决定发挥专长。日常工作中，数据图形团队早已建立了大量用于抓取政府机构公开数据（如金融市场、经济指标、天气、竞选资金、选举民调等）并自动生成实时图表的数据流水线。 数据团队与同事 Coco 合作构建了一个自动化监控机器人，通过定时任务检测司法部网站页面的数字指纹（fingerprint）变化。在 30 号那个周五的深夜，文件一旦上线，监控系统立即触发并在 Slack 中推送了警报，团队第一时间收到了通知。然而，真正进入下载阶段后，新的挑战接踵而至：司法部网站根本没有“一键全部下载”（Download All）按钮，只有数以万计的分页链接。为此，团队此前利用 AI 辅助构建了一套系统，并在联邦调查局（FBI）的《信息自由法》（FOIA）公开页面上提前演练，模拟抓取其他司法部公开文件，测试可能遭遇的验证码（CAPTCHA）、反爬虫保护或年龄验证机制。 ```mermaid flowchart TD accTitle: 自动化监控与文件抓取工作流 subgraph 监控阶段 A["定时监控任务 (Scheduled Job)"] --> B["检测页面指纹 (Fingerprint)"] B --> C{"发现更新?"} C -- 是 --> D["发送 Slack 警报给团队"] end subgraph 抓取阶段 D --> E["双轨抓取策略"] E --> F["智能爬取 (Smart Crawl)"] E --> G["暴力请求 (Brute Force Range)"] F & G --> H["并发请求与重试机制"] end ``` - 团队采用了“智能爬取”（抓取页面上所有潜在的 PDF 链接）与“区间穷举”（锁定首尾文件编号后并发请求中间所有可能地址，遇到 404 直接跳过）双轨并行的抓取策略。 - 系统设计了断点续传机制，支持多名工程师并行分批（tranches）抓取数百万个请求，以求在最短时间内获取完整数据。 --- ### [15:13](https://www.youtube.com/watch?v=yR197k0xGyE&t=913s) 应对反爬机制：从模拟点击到 IP 轮换的攻防战 随着抓取工作的展开，司法部网站的防御策略发生了剧烈变动。起初，网站仅要求在访问首页时进行一次年满 18 岁的年龄验证，团队编写了机器人模拟浏览器点击该确认按钮。但在团队成功下载了约 100 万份文档后，网站机制突然收紧，变成了“每查看或下载一个单独的 PDF 文件都必须物理点击一次 18 岁年龄确认”，这彻底打乱了原本的自动化节奏。 紧接着，官方服务器启用了严格的防爬虫封锁策略。为了应对限制，团队不得不反复动态调整策略：首先被迫降低并发请求的速率，随后转为单线程逐个请求；当单线程依然触发 IP 封锁时，工程师只得利用 VPN 频繁切换 IP 地址，采取“每换一个 IP 发送 200 次请求，然后立即换下一个 IP”的极限应对方式。现场一度演变成紧张的“猫鼠游戏”。得益于预先使用 AI 辅助生成的各种模块化函数，工程师能够针对“如果对方采取某种封禁手段，我们该调用何种功能”进行即时切换。直到全美其他媒体记者对服务器的轰炸式访问逐渐降温，官方放宽了限制，下载流程才重归平稳。 - 目录与日志系统实时记录已下载文件，并随时向编辑部汇报宏观统计数据。 - AI 工具在前期被用于模拟各种意外场景（What-if scenarios），使得爬虫脚本具备极高的动态调整弹性。 --- ### [17:13](https://www.youtube.com/watch?v=yR197k0xGyE&t=1033s) 新闻真实性底线：AI 作为“初级开发助手”与全流程人工核查 在涉及指控严重犯罪（如针对未成年人的恶性侵害）的高风险调查报道中，新闻行业对准确性的要求是 100%，哪怕一个微小的错误导致更正（Correction）都是无法承受的职业灾难。因此，团队明确划定了使用 AI 的红线：AI 绝不用于独立撰写报道或替记者做最终的数据分析结论，它的角色被严格定位为一组“充满热情但需要监督的初级开发工程师（junior developers）”，主要负责辅助编写爬虫脚本、梳理架构和考虑边界异常。 为了确保整个数据管道的诚实与准确，团队投入了海量精力构建详尽的审计日志系统。每一个被下载的文件都会被实时记入日志，各分工成员（如 Casey 负责抓取第 10 批次，Matt 负责跟进第 9 批次）通过统一日志互相协同。通过编写统计脚本比对日志，团队能精确知道系统是否出现了缺漏——例如“第 9 批次中还缺少约 10 万份文档需要补抓”，从而让前方记者在宣称“我们已经查阅了全部爱泼斯坦档案”时拥有确凿的底气。 ```mermaid flowchart LR accTitle: AI 与人工核查分工模式 subgraph AI 辅助层 AI1["快速生成爬虫脚本"] AI2["设想边界异常与补丁"] AI3["多场景函数代码库"] end subgraph 人工管控与核查层 H1["系统架构与容错设计"] H2["日志记录与缺漏核对 (Logging & Audit)"] H3["数据人工清洗与新闻事实核查"] end AI 辅助层 --> H1 H1 --> H2 H2 --> H3 ``` - 以往需要数天甚至数周才能设计完成的大型数据提取与异常修复系统，在 AI 赋能下缩短至几小时甚至数分钟内解决。 - 这套高度抽象化的软件架构具有可复用性，未来可直接平移到其他政府公开文件的大规模数据抓取任务中。 - AI 成为了 CNN 小型技术团队的核心“战斗力倍增器（force multiplier）”，使处理数百万量级的数据变得可控且值得信赖。 --- ### [22:16](https://www.youtube.com/watch?v=yR197k0xGyE&t=1336s) 隐藏的多媒体线索：破解伪装的 PDF 与图像人脸识别检索 当数百万份文档下载完毕并进行 OCR 文本识别时，团队遇到了另一个匪夷所思的技术阻碍：司法部在发布时将所有内容一律打包成了 `.pdf` 后缀的链接。然而在实际排查中，团队发现部分文件根本不是文本，而是伪装成 PDF 格式的多媒体内容。一名制作人尝试将某个 PDF 文件的后缀手动修改为 `.mp4` 后，文件竟然顺利播放出视频画面，编辑部震惊地发现：“天哪，这不是文件，这是一个视频！” 这种混乱的发布格式给传统的文档 OCR 流程造成了很大盲区。虽然邮件、短信等文本线索极具新闻价值，但照片和视频往往具有更强烈的视觉冲击力和直接证据力。为了对这批非结构化的图像和视频进行深度分析，CNN 团队与专业视觉搜索公司 Visual Layer 展开合作。Visual Layer 将司法部公开的所有多媒体素材全量导入其引擎，提供了强大的多模态人脸与视觉检索能力，使得调查记者能够直接通过特定人脸搜索其在全部海量照片与视频中的出现记录，从而极大加速了图像线索的挖掘。 ### [23:52](https://www.youtube.com/watch?v=yR197k0xGyE&t=1432s) 运用多模态 AI 曝光美国司法部的打码漏洞 CNN 与视觉 AI 公司 Visual Layer 展开合作，对美国司法部（DOJ）公开上传到爱泼斯坦档案网站的超过 10 万张图片与视频展开语义搜索（Semantic Search）。这项技术让记者不必逐一打开文件并将 PDF 转换成 MP4 进行人工比对，而是能直接对图像内容进行自然语言检索。 ```mermaid flowchart TD accTitle: 司法部未打码图像曝光流程 subgraph S1["数据检索与发现"] A["司法部网站公开图像 100,000+"] --> B["Visual Layer 语义检索分析"] B --> C["发现数十张重大打码疏漏图片"] end subgraph S2["调查与问责"] C --> D["CNN 电视节目《The Situation Room》曝光"] D --> E["CNN 正式向司法部质询"] E --> F["司法部次日紧急下架并重新打码"] end ``` 通过这一系统，记者发现司法部网站上存在严重的打码失误：大量原本必须严格打码的敏感隐私在公开网站上留存了近一个月。CNN 在《The Situation Room》（情势室）节目的独家报道中对这些图像进行了模糊处理后播出，其中包括： - 年轻女性的完全未打码裸照； - 一名年轻女孩亲吻爱泼斯坦面颊的照片； - 幼儿及婴儿的未打码面部清晰照（属于应绝对保护的未成年人）； - 护照、驾照及医疗记录等个人身份信息（PII），清晰泄露了身份证号、家庭住址和出生日期。 CNN 于周一向美国司法部提出质询，到周二，司法部便将这些涉事文件全部撤下，并替换为正确打码的版本。现场记者强调，大众总以为 AI 在新闻中的作用只是自动生成文章，但真正令调查记者兴奋的，正是这种发挥舆论监督（Accountability）与深度调查能力的 AI 场景。 --- ### [27:08](https://www.youtube.com/watch?v=yR197k0xGyE&t=1628s) 突发新闻与海量档案下的 AI 技术局限性 尽管 AI 极大地提升了处理效率，但在应对突发大事件时依然暴露出诸多现实瓶颈与技术短板。面对 350 万份庞大文件，首先面临的是物理层面的数据摄取延迟（Ingestion Lag）。在文件刚公布的前几个小时内，电视直播要求主播和出镜记者立即输出分析，社交媒体上各类碎片化爆料铺天盖地，而 AI 系统仍在建立索引和向量化。在系统上线前，CNN 团队只能回归最原始的传统分工：记者们各自认领页码（如一人分摊 1 到 100 页），进行人工快速翻阅。 ```mermaid flowchart LR accTitle: 新旧信息甄别的人机协作链 subgraph INPUT["海量公开文件"] A["司法部新发档案<br/>含历史庭审记录与新发邮件"] end subgraph GAP["AI 的核心短板"] B["Google Pinpoint (Gemini)<br/>无法判断'何为全新事实'"] end subgraph HUMAN["领域专家核验"] C["资深法庭记者<br/>参与过麦克斯韦尔审判"] D["一线深度调查记者<br/>如《迈阿密先锋报》Julie Brown"] end subgraph VERIFY["最终线索检索"] E["AI 精确语义检索与交叉核对"] end A --> B A --> C A --> D C --> E D --> E ``` 另一个更深层次的缺陷在于：**现有 AI 工具无法回答调查记者的终极问题——“什么是新的？”（What's new?）**。爱泼斯坦早在 2008 年就已首次定罪，在过去近 20 年间累积了海量民事诉讼、刑事案件以及其同伙吉丝莲·麦克斯韦尔（Ghislaine Maxwell）的审判记录，已有数十万页文件公之于众。司法部本次上传的不仅有 FBI 从未披露的内部邮件，也混杂着大量已公开多年的庭审卷宗。 当记者向内置 Gemini 的 Google Pinpoint 提问“哪些信息此前从未公开”时，AI 完全无法甄别。这一认知盲区只能依赖长期跟进该案的专家来弥补： - 长期旁听法庭听证会、熟记证人证言的法庭记者能够立即指出：“两年前这件证据就已经公开，不算新闻”； - 长期调查爱泼斯坦案的《迈阿密先锋报》记者朱莉·K·布朗（Julie K. Brown）在社交媒体上指出某份文件是业界苦等多年的关键证据，CNN 团队再利用 AI 语义搜索在海量数据库中迅速定位该文件，完成核实与深度报道。 --- ### [30:40](https://www.youtube.com/watch?v=yR197k0xGyE&t=1840s) 行业应用对比、删帖追踪与 AI 辅助代码工程 在全行业覆盖爱泼斯坦档案的过程中，多家主流媒体与机构展示了各自的技术架构。除 CNN 外，《纽约时报》开发了专有的结构化检索数据库系统，BBC 以及多家社会倡议团体也构建了专属检索系统，以应对可能发生的数据被删风险。 在本次报道中，由于部分档案涉及前总统特朗普（相关文件由特朗普政府依据国会法案公开发布），司法部网站出现了部分文档被撤下、修改或重新上线的争议情况。CNN 数据团队借助 AI 辅助搭建的代码系统，完整保留了历史版本日志（Log）： - 只要外界反馈某个文件的 URL 失效，数据团队便可直接在本地数据库调取快照，查证该文件此前是否存在、具体在何时被删； - 记者回忆，在紧迫的截稿线（Deadline）压力下写代码，工程师通常不得不放弃编写单元测试和详细日志；而这次借助 AI 编程助手，团队在极短时间内迅速补齐了日志监控与测试逻辑，确保了新闻报道的速度与绝对准确性。 --- ### [33:22](https://www.youtube.com/watch?v=yR197k0xGyE&t=2002s) 民间创新案例：Z 世代开发者打造的 Jmail 邮件重构平台 除了主流媒体的系统，民间开发者也贡献了现象级的技术应用。一组常驻旧金山的 Z 世代（Gen Z）年轻程序员利用 Cursor、Claude 和 Reduct 等 AI 开发工具，搭建了一个名为 **Jmail** 的交互式网站，将爱泼斯坦档案中零散分布的电子邮件重新组装成高仿真的 Gmail 邮箱界面。 | 司法部原始文件发布形态 | Jmail 重构后的交互形态 | 记者报道面临的实际痛点与突破 | |:--- |:--- |:--- | | **单页独立 PDF**：一封长达 20 轮的往复邮件被拆解为 20 个独立的单页 PDF 文件。 | **完整对话线索（Threads）**：将相关邮件聚合为完整的时间线与邮件对话列表。 | 若记者只翻到第 12 封 PDF，无法判断后续是否有回复或反转。单片断章取义会导致报道失实，Jmail 彻底解决了邮件上下文断裂的问题。 | | **碎片化无序编号**：后续邮件并不一定会按顺序排列在下一个编号的文件中。 | **可视化收件箱**：支持按发件人、收件人及日期完整呈现真实邮箱体验。 | 记者与公众能够以第一视角完整审视爱泼斯坦与政商名流之间的完整沟通脉络。 | 开发团队声称 Jmail 在上线首月就获得了超过 5 亿次浏览量。CNN 团队对此进行了专门报道并发布了社交媒体短视频，对这种利用现代 AI 编程栈实现的数据重组创新给予了高度评价。 ### [35:33](https://www.youtube.com/watch?v=yR197k0xGyE&t=2133s) 350万份档案的持续挖掘与分享总结 梳理这批爱泼斯坦案卷宗对于众多报道团队而言堪称一场“数据盛宴”（bonanza）。这是一个极其重大的新闻事件，本质上也是一个由海量数据和文件驱动的调查报道。由于涉及多达 350 万份庞大文件，直到今天仍有新的独家报道在不断涌现——就在几天前，《纽约时报》（The New York Times）依然从中挖掘出了全新线索并刊发报道。正如讲者所言，“没有人能一本正经地说，自己能在短短几天或几周内把所有文件都看个底朝天”，各类调查人员与记者目前仍在持续深挖这批档案。 在分享的最后，主持人向 CNN 的资深记者马歇尔·科恩（Marshall Cohen）和资深数据编辑马特·斯泰尔斯（Matt Stiles）致谢，感谢他们无私分享利用 AI 工具处理超大规模新闻数据集的实战心得与宝贵经验。随着针对这批档案的调查与技术挖掘不断深入，未来仍会有更多内幕与深度分析陆续揭晓。讲者最后感谢了主办方的邀请与现场观众的聆听，并祝愿大家在本次大会（Ai4）接下来的议程中有所收获。 - **数据集规模**：爱泼斯坦案调查共计公开了约 350 万份文件，其规模决定了报道与挖掘工作是一个长期的动态过程。 - **行业持续跟进**：各大主流媒体（包括《纽约时报》等）在文件公开数周乃至更久之后，依然在借助新型数据分析工具持续产出独家调查报道。 ## Notable quotes > [01:46](https://www.youtube.com/watch?v=yR197k0xGyE&t=106s) 吃披萨通宵翻页 — Marshall Cohen > [04:26](https://www.youtube.com/watch?v=yR197k0xGyE&t=266s) 我们有没有这条、内容是否属实 > [11:54](https://www.youtube.com/watch?v=yR197k0xGyE&t=714s) 你醒着吗？听说今晚可能要发，你能上线吗？ > [15:13](https://www.youtube.com/watch?v=yR197k0xGyE&t=913s) 每换一个 IP 发送 200 次请求，然后立即换下一个 IP — Matt Stiles > [17:13](https://www.youtube.com/watch?v=yR197k0xGyE&t=1033s) 充满热情但需要监督的初级开发工程师（junior developers） — Matt Stiles > [17:13](https://www.youtube.com/watch?v=yR197k0xGyE&t=1033s) 我们已经查阅了全部爱泼斯坦档案 — Marshall Cohen > [22:16](https://www.youtube.com/watch?v=yR197k0xGyE&t=1336s) 天哪，这不是文件，这是一个视频！ > [27:08](https://www.youtube.com/watch?v=yR197k0xGyE&t=1628s) 什么是新的？（What's new?） — Marshall Cohen > [27:08](https://www.youtube.com/watch?v=yR197k0xGyE&t=1628s) 两年前这件证据就已经公开，不算新闻 > [35:33](https://www.youtube.com/watch?v=yR197k0xGyE&t=2133s) 没有人能一本正经地说，自己能在短短几天或几周内把所有文件都看个底朝天 — Ryan Struyk ## Mentioned CNN, Ai4, Jeffrey Epstein（爱泼斯坦）, Ryan Struyk, Marshall Cohen（马歇尔·科恩）, Matt Stiles（马特·斯泰尔斯）, Google Pinpoint, Gemini, Python, Slack, FBI, 美国司法部（DOJ）, Visual Layer, The Situation Room, Ghislaine Maxwell（吉丝莲·麦克斯韦尔）, Julie K. Brown（朱莉·K·布朗）, 《迈阿密先锋报》, 《纽约时报》, BBC, Cursor, Claude, Reduct, Jmail ## Video description Marshall Cohen - Senior Reporter, CNN Matt Stiles - Senior Data Editor, CNN Ryan Struyk - Director, AI Innovation, CNN You've heard about the Epstein Files, but do you know how reporters used AI to cover the story? Three CNN journalists will explain how they used AI to analyze 3.5 million documents from the Jeffrey Epstein investigation. The talk will explore how new AI-powered tools allowed journalists to smartly sift through a massive dataset in ways that weren't possible just a few years ago, to uncover new stories. We'll also explore the limits of these tools, and the continued importance of human verification. The discussion will be of interest for journalists, lawyers, investigators of all sorts, and anyone who followed one of the most explosive news stories of the year. [Source](https://www.youtube.com/watch?v=yR197k0xGyE)

[1:46] Marshall Cohen: 吃披萨通宵翻页 [4:26] 我们有没有这条、内容是否属实 [11:54] 你醒着吗？听说今晚可能要发，你能上线吗？ [15:13] Matt Stiles: 每换一个 IP 发送 200 次请求，然后立即换下一个 IP [17:13] Matt Stiles: 充满热情但需要监督的初级开发工程师（junior developers） [17:13] Marshall Cohen: 我们已经查阅了全部爱泼斯坦档案 [22:16] 天哪，这不是文件，这是一个视频！ [27:08] Marshall Cohen: 什么是新的？（What's new?） [27:08] 两年前这件证据就已经公开，不算新闻 [35:33] Ryan Struyk: 没有人能一本正经地说，自己能在短短几天或几周内把所有文件都看个底朝天

[2:07] Marshall Cohen: a lot of too much pizza [4:26] Marshall Cohen: Hey, do you have this? Do we have this? Is this accurate? [11:54] Matt Stiles: Yeah, your bot worked. [13:32] Matt Stiles: What if they do this? What if they do that? [17:13] Matt Stiles: eager junior developers [26:48] Marshall Cohen: Is this new? [35:33] bonanza for so many different groups [35:33] nobody can say with a straight face that they went through everything in a matter of days or weeks.

[0:04] Uh, all right. Thank you so much for that  introduction. Uh, we are really excited to [0:08] be here again. Uh, my name is Ryan Struck.  I'm our director of AI innovation at CNN, [0:13] which means that I lead our editorial AI efforts  in the newsroom. Uh so I focus on how we use AI in [0:19] our news gathering and reporting initiatives,  how we use it in our editorial workflows [0:24] and then training journalists on the AI tools  that we have uh how to use them and how to use [0:29] them uh responsibly in line with our standards.  Uh I'm very excited to welcome two of my favorite [0:36] colleagues uh here from our newsroom um Marshall  Cohen and Matt Styles. Uh Marshall is a senior [0:43] reporter based in Washington. Since joining CNN in  2015, he's covered the Trump Russia investigation, [0:50] the fallout from the January 6th insurrection at  the United States capital, uh President Donald [0:55] Trump's legal cases, Hunter Biden's legal cases,  and much more. He won an Emmy award in 2020 and [1:01] has received three additional nominations. Matt  Styles is a senior data and graphics editor for [1:08] CNN in our digital visual news team. He helps  lead a team of visual journalists around the world [1:14] and is responsible for anticipating and executing  datadriven visual coverage. He is a Pulitzer Prize [1:20] finalist and prior to joining CNN worked at  the Los Angeles Times, the Wall Street Journal, [1:26] NPR, and the Texas Tribune. Uh thank you guys  so much for being here. uh very excited to talk [1:32] about this uh really uh important major uh story  and um how AI really helped us in the process of [1:41] reporting and and news gathering. Um so Marshall,  I want to start with you and and ask you kind of [1:46] a little bit about kind of in your experience over  the last several years at CNN. You are no stranger [1:52] to document dumps in the past. um what made these  documents particularly difficult to work with [2:01] and what role did AI play in our ability to  report on them? Yeah, it's great to be here and [2:07] thanks to all of you for uh coming out to watch  us talk. It's not a very uh appealing term of art, [2:13] a document dump, but that is what we call it. Uh  when the government or Congress or some entity [2:19] releases thousands or millions of documents on  our laps and uh as journalists, it's our job to [2:27] figure out what's new, what's important,  what's newsworthy, uh to do it quickly, [2:34] and to do it accurately as well. And the  Epstein Files release in December and January [2:41] and February of this year attracted international  attention. Uh, and we did it differently at CNN [2:48] because of AI. It wasn't our first document dump.  Um, my career started at CBS News before the 2016 [2:56] election. I covered Hillary Clinton's emails  when they were released by the State Department. [3:01] The Mueller report came out in 2019. that was  almost 500 pages long. And after January 6th, the [3:10] congressional committee that investigated January  6th interviewed hundreds of witnesses and some of [3:15] those transcripts were hundreds of pages long. So  in the olden days, uh we used to go manually page [3:24] by page and literally we would have a team. We've  been in some of those rooms. I remember because I [3:29] I was at ABC at the time with dealing with some  of these uh chains and and Yeah. You would have [3:33] a reading team that would sit there and just click  through page after page. Usually the best part of [3:37] it was that they would bring you lunch or dinner.  It was the pizza. Uh a lot of too much pizza. But [3:43] um manually page by page. And look, if it's, you  know, look, if it's a 500page Muller report and [3:50] you have 12 people that have been covering that  for years, okay, you can get through that. Um [3:56] to go back in history, I think I have here  on my next slide, let's see, what did we do? [4:02] We went page by page through the Mueller  report manually. This is the pre-ai era for us. [4:08] And we had a whole team and we documented 77 lies  that Mueller identified that was told by various [4:15] members of the Trump team. Um, we thought this  was a great way to take a very long report that [4:21] very few members of the public are going to are  going to read and do it in a digestible fashion. [4:26] Fast forward now we're working to this year.  Hundreds of thousands of pages of the Epstein [4:33] files were released. Uh eventually it totaled 3.5  million documents from the Justice Department and [4:42] thankfully the technology has advanced. We used  Google Pinpoint for most of our work. It's a tool [4:51] that Google makes available to journalists, which  it's probably not incredibly sophisticated. If [4:57] you're at other panels and stuff, you're probably  hearing about far more sophisticated technology [5:00] than this. But it's very important for us because  it helped us put all these documents into one [5:06] universe. It uh did an OCR o over everything to  make everything searchable, including handwriting, [5:13] and created a space where we could browse  these files in a much more organized fashion. [5:23] Uh, it also created lists of names and entities.  Previously, you know, if you saw something that [5:31] was interesting, maybe you'd write it down.  Maybe we would have a shared document and [5:34] start keeping tabs of various names that come up.  The [snorts] AI did that for us. It doesn't sound [5:40] earthshattering, but it was a big step forward for  us. Uh and uh one other benefit of this tried and [5:48] trueue practice in journalism is that your bosses  see stuff on Twitter that your competitors are [5:54] doing and they're like, "Hey, do you have this?  Do we have this? Is this accurate?" And previously [6:01] we would have to manually sip through hundreds  of pages and we it would be very difficult to [6:05] find things just that surfaced from the internet  from other journalists or citizens or whomever. [6:11] Now with these tools, we were able to very  quickly, no pun intended, pinpoint where that is, [6:18] find the material, and then we could ver verify  it or corroborate it for our own purposes. [6:24] The AI helped us do stories like these. Our  colleagues worked on mapping all the relationships [6:30] between various members of Jeffrey Epstein's  larger web. That was something that was [6:36] incredibly helpful for us to start uh who  was emailing who, who was texting who, [6:42] the AI could really do a good job of building  those relationships. And then we brought it to [6:46] life editorially. And of course, as journalists,  we don't just throw this on the internet. We [6:50] reached out to all these people. We get their  side of the story as well. We include their [6:54] response and their comment. But the AI helped us  really start that process. And ultimately it was [7:00] very effective for us because we just turned out  tons of content and the audience really really [7:06] was there for it. This was a huge story. We've  seen resignations in the business community. [7:11] Politicians have had to step down uh in hundreds  of millions of dollars of lawsuits. Uh so much [7:19] fallout from this and we were able to turn out  so much material in large part thanks to the AI. [7:25] Um, Matt, I want to ask you because in some  ways this this is kind of the the final product, [7:30] the result of of all this reporting  and and and using using the AI to [7:36] uh to to extract kind of editorial insights and  and perform human review on those. But just just [7:43] acquiring and ingesting the documents was its  own herculean effort and AI also played a role [7:49] in that. How right? And um and thank you all  for coming to this. Good to see so many people. [7:55] Um, so AI played an important role for us in  trying to figure out when DOJ was going to [8:00] release the files, when they did, and then  also to be able to acquire them all and to [8:05] analyze them so we can go through those those  steps. And um, this was a a really like um, [8:13] this wasn't a clean data set. It it was a chaotic  dump, for lack of a better word. Um and we it [8:22] was a really challenging effort and so we'll go  through kind of like what we did and how we were [8:27] able to get these documents. Um the fundamental  thing we faced is we had this data engineering [8:34] you know challenge before us. How do we go  out this sort of like chaotic unpredictable [8:40] uh list of files with this crazy pageionation  system that DOJ dreamed up and acquire [8:47] and download millions of files and then make sense  of them keep track of them, analyze them and then [8:54] present all that to the newsroom so that they  could turn this into stories. Right? This was it. [9:00] I mean what you're seeing here imagine these are  individual PDFs times 3 million times 3 million [9:08] some of them are multi-page some of them are  single page and what you don't see on this slide [9:12] is at the bottom there's pageionation you know  like one page one page two page three and it would [9:17] just go on like 10,000 20,000 30,000 uh so we had  to figure all that out um so the first before we [9:26] before we get to how we did it I wanted to also  emphasize is that you know some of the tools we're [9:31] going to talk about today are probably relatively  normal in your world or the engineers that work [9:36] with you. Um but for us it was rel relatively  novel to be able to dive in with some of these [9:40] more detailed uh AI tools and we wouldn't have  been able to get as far uh to be as thorough to [9:46] be as ambitious and frankly quick uh as we were  without these AI tools. So that that's a key [9:53] takeaway. But we also had to be very careful about  our journalistic, you know, principles and that [10:00] when we were downloading these files, we had  to keep track of what we were downloading [10:04] and how much we had so that the newsroom could  confidently say there were this many documents. [10:10] We're giving you information according to what  we've downloaded so far because this didn't happen [10:15] right away. We needed to be able to audit what  we downloaded uh and go back and download things [10:21] that we might have missed. We needed to be able  to trace if documents were removed and and we had [10:26] already downloaded them so we could go back and  see what was objectionable on those files. So all [10:31] that had to be thought of uh at the beginning. But  before we could get to the downloading the files, [10:37] we had to figure out when they were going to  release them. So uh we showed a couple slides [10:41] from earlier releases in uh in late in 2025 where  they had released tranches of data and that gave [10:48] us some clues about where these files might be  and the kind of format that they would come in. [10:53] We didn't expect the crazy pageionation and all  the roadblocks that they would throw in our way. [10:59] But we needed to know when they were going to  put the files up. We couldn't have an editor or [11:04] reporter just sit on DOJ's website hitting refresh  all day, right? So we set up uh an automation [11:11] system and we this was part of two main systems  that we used uh to track the files and then [11:18] ultimately harvest them. Um first monitoring  what we did was set up a simple job that had [11:25] a Python script that would go to DOJ's website.  we had a a sense of where the documents might be [11:33] and it would check every minute and if the  fingerprint of the page changed, if the number of [11:39] sets of documents changed, we would get uh a Slack  message. And we tested this on other DOJ sites, on [11:47] other sites that change occasionally to make sure  that um that we were getting those messages and [11:54] alerts. It would be funny if I set up this whole  system and then we didn't get the message, right? [11:58] So, uh, we didn't know. There were rumors about  when the release might come, but in the weeks, [12:03] refresh my memory, a lot of late nights, a lot  of you got a call at like, you know, 11:00, [12:08] hey, are you are you awake? We hear  it might be tonight. Can you uh [12:13] can you log on? And that's when my  colleague Koko, who I work with closely, [12:18] went to her and said, "Look, let's set up a  monitoring system so we don't have to just, [12:22] you know, constantly get these messages in the  middle of the night." Um, and I I work on this [12:26] data graphics team where we have all kinds of data  pipelines for everything you could imagine from [12:32] markets to economic indicators, weather, campaign  finance, elections, polling, all that stuff. We [12:38] have these data pipelines that'll go out,  grab this data from governmental agencies, [12:42] transform it, and then serve it up to a lot of  live charts that reporters use on the website. So, [12:47] it was a natural thing for us to do  to set up a bot to go out and check. [12:52] Um, so that's what we did. It was a scheduled  job. Check the fingerprint. Get the Slack message. [12:59] And you can see this is the actual message that we  got uh on the 30th. In the weeks leading up to the [13:06] release, we kept hearing, "Oh, it's going to be  tomorrow." Oh, I think it's going to be Friday. [13:10] You know, and then we finally figured out, "Oh,  it's going to be it's going to be this Friday." [13:14] I think that was a Friday. Yeah, it was. Um, and  sure enough, the bot, as soon as they put it up, [13:19] the bot found it, sent a message, and I sent a  DM to Koko saying, "Yeah, your bot worked." Um, [13:25] shout out, Coco. Uh, so that worked. So, we  knew we had the the documents, right? And then [13:32] we went and looked at what they released and that  was a whole other challenge. Um, we there was no [13:39] download all button, right? So we ran into that  pageionation. We ran into pages like we showed [13:45] earlier. Um and we fortunately had used AI to  set up a pretty robust system that would uh [13:54] go back and practice grabbing PDF files from  other DOJ dumps. we insticted on the FBI's FOYA [14:02] document page so that we could practice  grabbing seeing what hurdles we ran into, [14:08] whether there was capture or whether there was bot  protection that had been set up on the front end, [14:13] whether there were age um requirements that  you had to check to be able to download. And [14:19] the system also um let me show you this uh would  allow us to uh we had like sort of two methods. [14:28] We could do just um a smart kind of uh crawl  of the entire website once the release happened [14:37] and just make a big old list of every potential  PDF URL and go through them all any PDF that we [14:43] found. And then we had more of a brute force kind  of approach where we just found the first one and [14:49] the last one and just did a request for every  number in between and it if if it hit a PDF [14:55] that didn't exist it just skipped it and so it's  millions of requests and we ran them in parallel [15:02] and we set up systems so that if it stopped  we could start it again and we had multiple [15:07] people doing different trunches of this so we  could try to get it as quickly as possible. [15:13] Let's see. Um, we did hit some hurdles uh once we  were there. Um, initially we knew that they would [15:21] have an age verification check. So, we had a  we had uh the bot would mimic a browser and [15:29] physically click the age verification button every  single page, every single first the first time, [15:35] right? But eventually every single PDF to to  see a single every single time every single PDF [15:42] you had to verify you were over 18 which obviously  was not part of the initial plan. Yeah. I think we [15:48] had gotten about a million documents before  they did that. So that slowed us down. Um [15:53] they implemented some bot blocking. Uh, so we had  to like slow down our parallel requests and then [16:00] we had to slow down any parallel requests and  just do them one at a time for a while and then [16:05] we had to slow those down because they started  blocking us and then I'm like switching IPs, [16:10] you know, using VPN and sending 200 requests and  then sending another 200. It turned into this [16:16] uh crazy scene. Um, but fortunately that, you  know, the AI system or the the AI tools we used [16:23] allowed us to just pose all these questions. What  if they do this? What if they do that? And there's [16:28] a different function for that. Um, and we were  able to call on those when needed and quickly [16:32] pivot when we noticed that their strategy had  changed. Eventually, they loosened things up once [16:39] not every single journalist in America wasn't  there, you know, hammering their servers. Um, [16:44] and it and it went a little more smoothly.  But, um, it was not super easy at first. Um [16:51] we we'll get into some other things here but  just you can see from our directory structure [16:55] we also had systems for counting what we had  so far looking for gaps in what we had acquired [17:03] uh doing some like sort of topline uh numbers that  we could give to the newsroom as well in addition [17:08] to just physically grabbing the documents. Yeah.  Yeah. Well, Matt, I want to ask you about I want [17:13] to go back to what you said about kind of the  journalistic standards at the beginning of [17:17] this because I'll tell you in my job what I lose  sleep over is the fact that like AI makes mistakes [17:24] and if you're in the AI world like you understand  that it's not bulletproof and when you work in [17:29] an industry where your accuracy rate needs to be  100%. um you know I in a in a former life I was a [17:38] TV producer and like in a in a a data reporter and  had to like make corrections and there's no worse [17:46] feeling than like the gut-wrenching like uh I I I  have to issue a correction like I made a mistake. [17:53] So what and there can be real consequences to  that. Exactly. Especially there's nothing more [17:58] higher stakes than a story where people are being  accused of heinous crimes against children. So [18:05] it is, you know, it does need to be bulletproof.  So what kind of safeguards, you know, [18:10] did you build into this process and what role  did human review play in in this process? [18:18] Yeah, it was a significant  amount of human review. Um, [18:23] from my perspective, we we use AI tools a lot, but  generally it's to like make our systems better. [18:29] um we don't use it to write stories or to do data  analysis for stories that humans do, right? But we [18:37] do everything up to that point to make it easier,  right? Um, and in this case though, you know, [18:44] I looked at the AI tools we were using  as kind of like a group of uh junior [18:50] like eager junior developers who could really  go out and help us, you know, build scrapers [18:56] um think through the structure uh think about all  the contingencies, right? But all that was human [19:02] designed the software we designed knowing  you know after doing these dumps before [19:09] both at CNN and other places understanding  the pitfalls and what might happen if this [19:16] the process stops and we're not able to tell  people in the newsroom what we collected and [19:20] what we didn't. [clears throat] Um so all that  involved human review. We had a plan for scale how [19:26] to handle errors as I mentioned. Uh we did a whole  bunch of testing on other stuff. Fortunately, [19:31] we had um you know a couple weeks into the runup  of the big three million page dump. Um and then we [19:39] we also I spent almost as much time on this as I  did on everything else was every when a file got [19:46] downloaded it was logged and every anyone who  used the system was using the logs and we were [19:52] communicating for you know who was downloading  what batch or whatever. Um, but in the end we had [19:57] a log of everything we had. And that way we could  use our scripts to count that and we could go back [20:04] to the newsroom and say, "Well, we're missing  about 100,000 from that number nine trunch, [20:10] which I need to go get." And meanwhile, Casey, one  of my colleagues, is getting the rest of number [20:16] 10 trunch, right? Uh, so that was a huge deal  for us in the newsroom because we're wondering, [20:21] okay, is it done? Is it done? Is everything in  there? you know, can we say that we've looked [20:25] through the entire Epstein files, you know,  and and having people on the on your side of it [20:32] have an actual accounting of what's in our system  and what's missing and what's on the way was what [20:39] helped us keep this honest. Yeah. And there's, you  know, you have to do human review when the system [20:44] starts to run. You have to make sure it's actually  downloading and what you're asking it to download, [20:49] too. I mean, there's a lot of oversight that  goes in. Um but you know to to just wrap up the [20:56] what we in using these AI tools for this story  it was a big a big shift or eye openener for a [21:02] lot of people in our newsroom um in that you  know we could build something this ambitious [21:09] build really like a a system software we could  use for other dumps later because it's all um [21:15] abstracted we just we could stick it on some  other PDF dump later right um We did that in [21:22] in you know hours not days. When when problems  arose we fix them in seconds or minutes not days. [21:30] This could have taken weeks even planning up  to this you know thinking about how we ingest [21:35] election data. How much work goes into that on  the front end. Um so it changes what's possible [21:41] in a newsroom. CNN's a big newsroom but we still  have a limited number of nerds like me that can [21:47] go hack away to to build stories, right? Um, and  so AI really is a force multiplier for us and it [21:54] it really changed um the game here. It it allows  us to um take something kind of crazy uh and that [22:02] seemed too big to get our arms around um and make  it scalable and make it work uh and make it so [22:09] that the newsroom could trust it. Um that's great.  So um all right. So when when this whole process [22:16] finishes um then Marshall one of the other  things that we ran into at that point was that [22:22] uh there were there were text documents in  there but there were also images and video [22:27] and the OCR process that we were running on the  text documents uh did not translate to the images. [22:35] So uh how did you go about analyzing the images  and the videos that were part of that was a real [22:41] challenge. That was a real challenge for us. And  imagine you're looking on the DOJ website and you [22:47] saw all those links of the PDFs. Uh everything  in there was PDF. Even if it wasn't a PDF, [22:55] um there were videos in there. And one of our  producers changed PDF tompp4. And then all of [23:03] a sudden, holy cow, this is a video. It's it's  not a document. It's a video. which speaks to [23:10] the kind of insane way that the Justice Department  put this out. Uh, and that really tied our hands. [23:17] The documents were incredible. I'm sure many of  you read about the emails and the text messages, [23:22] but obviously photos and videos can tell a very  compelling story. So, we actually partnered with [23:29] a company called Visual Layer uh that does really  interesting search for photos and videos. uh they [23:40] had ingested all the material from the DOJ and  just had a very strong search that was very easy [23:47] to use for us. You could search certain faces and  see what other pictures they came up. You could [23:52] search for items or things, a semantic search of  of the all of the videos and all of the photos. [23:59] I used that tool to do this story. Uh  those searches helped us uncover the fact [24:06] that there were dozens of photos that had been  uploaded to the Justice Department website that [24:13] absolutely should have been redacted but were not.  And they included naked photos of young women, [24:20] uh personally identifiable information  like passports and medical records, [24:24] um non-explicit but still problematic pictures of  babies and children that absolutely should have [24:31] been redacted. you don't want to be posting the  face of some random toddler on the internet. Um, [24:37] and this was a very compelling story. Uh,  and it was only made possible because of [24:42] what we did with, uh, that company, Visual Air.  I've got a short clip here, uh, from our show, [24:48] The Situation Room, that I want to show you how we  used, uh, AI to bring this story to life. You have [24:56] new reporting about botched redactions. What more  are you learning? Hey Pam, we've known for a while [25:02] about redaction problems in the Epstein files,  but it's much more widespread than we thought. [25:08] We worked with an AI company called Visual Layer  to examine more than a 100,000 images that the DOJ [25:14] posted to its Epstein Files website, and we found  more than a dozen photos that definitely should [25:19] have been redacted, but weren't and remained on  the public DOJ site for nearly a month. Now, I'm [25:26] about to show you some of these pictures. Keep in  mind, CNN blurred the versions that you're about [25:32] to see, but they were available, fully unredacted,  on the DOJ site for weeks. There were pictures of [25:38] a young girl kissing Jeffrey Epstein on the cheek.  There were also nonsexual but unredacted images [25:46] of other young children and toddlers who are  obviously minors and should have been protected. [25:51] And also there were several pictures of passports  and driver's licenses revealing private data like [25:57] ID numbers, addresses, and dates of birth.  CNN asked DOJ about these images on Monday, [26:04] and by Tuesday, they were all taken down and  replaced with properly redacted versions. [26:12] Couldn't have done that without AI. We would have  had to go file by file and change PDF to MP4. [26:17] It's just ridiculous. Wouldn't have been possible.  And I think a lot of people think when they think [26:21] about AI and news their minds immediately go to  generating articles and what do not do. Correct. [26:28] And what excites me about as a journalist about  the role of AI in journalism is accountability [26:34] in investigative work like this that enables  us to write a story or or perform an analysis [26:40] that we wouldn't have been able to do without it  100%. Um so uh you know we are really kind of in [26:48] the first uh uh few years of kind of widespread  AI adoption especially in news and we are all [26:55] figuring out what kind of the limitations are the  current limitations are too. Um were there any [27:01] things that you tried to do that AI could not  do or or what limitations did you run into [27:08] when you were working on this story? There were  definitely limits. This is not something that is [27:13] perfect and always going to have the answers  that we wanted. Um, physical limitations. [27:19] If you're dealing with 3.5 million documents,  it takes time to do what Matt did to get it [27:25] into the AI system that so then we can start  gleaning insights from it. It takes time, [27:31] but you know, when the documents come out,  we're live on TV, you know, and and so our [27:38] anchors and correspondents on air, they got to  talk about something. And they can't just say, [27:43] "All right, uh, please everybody be patient.  We'll let you know what it says." You know, [27:47] this is live TV. The internet obviously was  exploding. People on Twitter were highlighting [27:53] this and this email and this text and this photo.  Oh my god, all these things going crazy. Um, [27:59] and we're in those first minutes and couple hours,  we're waiting we're waiting for the stuff to get [28:06] ingested to the AI. So, we started honestly with  some old school manual searches in those first [28:11] few hours. You know, we literally just had a team  that was divvied up, you take pages one through [28:18] 100, you take page 100 through whatever, and  just see how far you can get before the AI is [28:24] up and running. That was a real challenge. And  also once it was up and running honestly so this [28:30] Google Pinpoint system has um Gemini built in and  you're supposed to be able to ask it questions and [28:36] stuff like that and some of those sometimes it was  very good but one thing that I haven't seen any AI [28:42] tools yet they can do is answer the question  that we always ask as journalists which is [28:49] is this new what's new Jeffrey Epstein has he  was first convicted uh of sex crimes in 2008. [29:00] That is almost 20 years ago. There have been  20 years of civil lawsuits, criminal cases, [29:07] the Gileain Maxwell trial for his co-conspirator.  Hundreds of thousands of documents have already [29:13] been public related to Jeffrey Epstein. Uh  this released millions of additional files, [29:20] but sprinkled on the DOJ website, they were very  transparent about this. Part of it was just the [29:26] case record from the court cases that have been  public in addition to never-beforeseen emails from [29:32] the DOJ and the FBI. Um, when we went to Gemini,  you know, I mean, it would be a joke to ask, hey, [29:39] what's new? What hasn't been seen before? It can't  answer that. At this point, the only people that [29:44] really can't answer that were our colleagues that  that covered the Gain Maxwell trial. And they were [29:49] there during the evidentiary hearings. They were  there when the witnesses testified and they could [29:54] say, "No, no, no. We saw that 2 years ago. That  already came out. Yes, it's a big revelation, [29:59] but that already came out. That's not new." Or  other reporters like Julie Brown from the Miami [30:04] Herald, who has been covering Epstein from the  very beginning. She was tweeting very powerful [30:10] insights and saying, "Hey, we've been waiting  for this document. We finally got to see it." And [30:15] then we would take what she tweeted, use our new  newly powered AI semantic searches in Pinpoint, [30:22] find it, match it, and then we could do that.  So there were limits, but I mean the ball has [30:26] moved forward significantly, but it there are  definitely limits to this and also just getting [30:31] the documents into Pinpoint. So, we downloaded  them all from DOJ, but we had to another [30:40] uh we've only got a couple minutes left, but  we obviously were not the only ones covering [30:44] this story. Uh there were a lot of other  news organizations doing incredible work. [30:49] Um Matt, what did you see kind of around  the industry that that jumped out at you or [30:54] impressed you uh with with other organizations  and other folks using AI to cover? Yeah, there I [31:00] mean there were quite a few groups that sort of  built things that were similar to what we did. [31:05] I was impressed by the New York Times uh  that built our sort of their own proprietary [31:10] uh searchable database system uh that they've you  know started to talk about and uh hopefully other [31:15] journals can learn from that. The BBC also did  something similar uh and there have been uh you [31:20] know advocacy groups and stuff that have put  together systems like similar to ours uh that [31:25] made these documents available or sought to  make sure that if they were deleted later [31:31] that there was some sort of historic record of  them and then there was Jmail you want to talk [31:36] about I'll get into that but also on the deletion  your systems were able to identify when things [31:41] were deleted. This was something we worried  about tremendously and we did see deletions. [31:45] There were there was a back and forth controversy  because some of these documents uh mentioned [31:51] President Trump which obviously these documents  were released by the Trump administration pursuant [31:56] to a congressional action and things were deleted  and things were put up. So you you were able [32:02] to keep track of stuff that got deleted as  well which I think fueled some stories. Yeah, [32:07] it did. And we, you know, we were able  to go back and if someone said, you know, [32:10] this URL to this document no longer exists,  we could query our log and see if it was there [32:15] and we should we we we would have a copy. Some of  them made no sense. Some of them were obvious why [32:20] they had been taken down, right? It was confusing,  but we had the the AI, you know, system that um [32:28] the AI assistant that had helped. And when you're  coding on deadline, what are the first things you [32:35] don't do? You don't do tests, you don't log,  you just get the job done as fast as possible [32:41] so that you can write your story or you can  help somebody else do their story on air, right? [32:46] Uh in this case, we had the luxury of saying,  "Oh, wait. You know, we should log everything. [32:50] I don't, you know, I don't have anybody to spend a  couple days building that, but I have an AI system [32:55] that can help do that and we can test it." Uh I,  you know, I don't have some tests, but we can use [33:00] our AI system to to build that. So, it gave us a  luxury of doing this, right? um and doing it fast. [33:07] And that's that's always what we aim for,  accurate and fast. Our our bosses tell us a lot [33:15] that they would prefer to be accurate over  being first. Um obviously there was a lot [33:22] of uh innovation that occurred outside of the  journalism spaces as well. Maybe some of you [33:28] have seen this before. or on the left side here  is something that is called Jmail. And a group [33:34] of Gen Z programmers who live in San Francisco  took it on for themselves on their own accord to [33:42] go through the Epstein files and create a Gmail  suite that was reflective of Jeffrey Epstein's [33:50] real Gmail account. This is what we should have  done, by the way. Yeah, I wish it was incredibly [33:56] valuable because imagine what you would see on  the Justice Department website, individual PDFs of [34:04] individual emails. And if it's a 20 email chain,  that would be 20 PDFs. And if you are a reporter [34:13] and you stumble upon email number 12 of 20 and  you're like, "Oh, this is good. He's This is He's [34:20] He's emailing somebody. You know, we didn't know  about that. That's good stuff. You have no clue [34:27] if there are replies. You have no clue what emails  came next. All you saw is that one PDF. And you [34:34] can hope and pray that the next sequential PDF  would be the correct one, but it wasn't always. [34:40] And so that would make our jobs very hard because  as journalists, we need to tell the full story. If [34:47] we're going to report that person X was emailing  with Jeffrey Epstein and we only read half of the [34:51] email chain, then we've failed. Um, this solved  that problem. It was really interesting. And um, [34:59] these Gen Z folks said that they used Cursor,  Claude, and Reduct to make their site. [35:04] I'm not a coder, so I mean I know what Claude is,  but um, and they said that they had 500 million [35:10] views on their Jail site in the first month. uh  we haven't verified that but it's there's a lot of [35:17] interest out there and we spoke to the creators of  that and on the right hand side here is a vertical [35:23] video that we did for social media where we talked  about how this group um did their own innovations [35:33] to go through the files and you know it was just  a bonanza for so many different groups because [35:39] it was such an important story and it was a  data document story. There are still stories [35:44] coming out. Literally, I think I saw one in  the New York Times just a few days ago where [35:49] they are still people are still digging  through these files. 3.5 million files. [35:54] Nobody can say with a straight face that they went  through everything in a matter of days or weeks. [35:58] Stuff's still coming out. Uh well, thank  you uh Marshall and Matt for sharing your [36:05] your takeaways and your expertise with us  and we will continue to stay tuned as more [36:10] more developments and more uh insights  come out of uh those investigations. Uh [36:15] thank you so much for having us here and uh  enjoy the rest of the conference. Thank you.