USA TODAY Co. Deploys AI Evaluator Agents to Replace Manual Reviews

AI Fatigue Is Real - Your Governance Model Needs a Makeover With USA TODAY Co., Inc — Ai4

Ai4

Manual human-in-the-loop verification triggers judgment fatigue and fails against non-deterministic models, requiring automated supervising evaluations to enforce accuracy standards.

AI-assisted summary of AI Fatigue Is Real - Your Governance Model Needs a Makeover With USA TODAY Co., Inc — original source. Check the original for full context.

Source published: 2026-08-20

Detailed notes

## Background Published on YouTube by Ai4 on August 20, 2026, this conference presentation features Jessica Davis, Vice President of AI Product at USA TODAY Co., Inc. The session addresses the growing issue of AI fatigue caused by manual review bottlenecks and outlines how organizations can modernize governance through automated AI evaluations.

## TL;DR Jessica Davis argues that enterprise AI fatigue stems from judgment exhaustion when employees are expected to manually review every non-deterministic output. To resolve this, USA TODAY Co. moved governance from 'in-the-loop' manual checks to 'on-the-loop' automated AI evaluations supported by an empirical Risk-Value Matrix. A newsroom case study demonstrates how deploying a supervising evaluator agent boosted public records generation accuracy from a stalled 75% to over 99% while shortening release cycles from months to days. Davis concludes that enterprise leaders must clearly define the distinct role of human judgment rather than forcing workers to babysit AI systems.

## Atmosphere & style The presentation carries an empathetic, candid, and highly pragmatic tone, opening with a frank acknowledgment of widespread workplace burnout and the gap between AI hype and real operational strain. Davis speaks with steady authority, grounding complex governance theory in tangible newsroom realities. The session feels like an empowering, actionable masterclass that bridges the gap between high-stakes editorial trust and modern AI product management.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) Enterprise AI fatigue stems from a fundamental disconnect where AI has transformed daily workflows, but organizational expectations and workloads have remained static. - [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) The breakdown in manual human-in-the-loop review is driven by judgment fatigue rather than process fatigue, as humans cannot maintain critical scrutiny across every single generation. - [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) Traditional enterprise governance fails with generative AI because static frameworks designed for deterministic code cannot handle non-deterministic model outputs. - [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) Modern governance requires moving from human-in-the-loop to automated AI evaluations that supervise models 'on-the-loop' from above. - [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) A major industry gap exists in non-tech sectors like news media, where leaders often lack awareness of automated evaluation frameworks despite operating in high-trust environments. - [06:58](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=418s) Aligning product delivery speed with governance risk mitigation requires establishing a shared empirical core through a formal Risk-Value Matrix. - [08:23](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=503s) Relying strictly on prompt engineering and manual debugging stalled public records generation accuracy at roughly 75% due to persistent hallucinations. - [08:23](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=503s) Deploying a dedicated supervising AI evaluator trained on vetted samples raised FOIA generator accuracy to over 99% across the majority of US states. - [08:23](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=503s) Evaluation-driven development compressed feature release cycles and compliance deliberation from months down to weeks and days. - [11:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=664s) Future enterprise governance depends on leadership clarity to define where automated systems can be trusted so human expertise is reserved for high-impact community needs.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) AI Fatigue and the Flaws of Manual "Human-in-the-Loop" Review

Opening the session toward the end of the conference day, Jessica Davis introduces herself as the Vice President of AI Product at USA TODAY Co., which encompasses USA TODAY alongside 200 local newsrooms across the United States, making it America's largest publisher. Having founded the company's AI governance program four years earlier to safeguard responsible AI adoption across newsrooms and sales teams, Davis frequently encounters widespread burnout across the enterprise. Common complaints like "I can't keep up," "I'm tired," and "When does the time savings come?" reflect a growing sense of AI fatigue. When polling the room on whether AI delivers on its promised return on investment and time savings, the audience is evenly split between those who save time and those for whom AI has merely introduced additional work. While AI tools have reshaped workflows, organizational expectations have not adjusted, leaving workers buried under more tasks than ever.

In high-trust environments like journalism where accuracy is paramount, workers are plagued by the persistent question: "Is this AI content right?" Reviewing every single generation creates an unsustainable burden. Davis clarifies that when governance models collapse under this pressure, the failure is not "process fatigue" but "judgment fatigue." While human-in-the-loop validation is vital because models can make mistakes, relying on individuals to maintain constant vigilance and bring critical judgment to every single output is simply not humanly possible. This dynamic quietly transformed daily job responsibilities without adjusting operational expectations. Addressing this is fundamentally a leadership challenge rather than a technical one, requiring organizations to pause and evaluate whether they are working for the AI or if the AI is working for them.

- **Scale of Operation:** USA TODAY Co. operates USA TODAY and 200 local newsrooms across the US. - **Core Dilemma:** The gap between shifted workflows and static expectations is the root cause where employee fatigue lives. - **Human Burden:** Treating human review as a mandatory filter for every individual output leads directly to judgment exhaustion.

[04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) Why Static Governance Fails and the Missing Layer of AI Evaluations

Reviewing AI outputs is difficult because teams must simultaneously evaluate multiple criteria: accuracy as table stakes, usefulness, user adoption, safety, and fairness. Traditional enterprise governance frameworks were designed for a static software world, where code behaves deterministically. Generative AI, however, is non-static and non-deterministic; submitting the exact same prompt multiple times can yield entirely different results. Consequently, static governance practices inevitably buckle under the variability of large language models. Davis argues that the solution is not to insert human judgment inside the execution loop for every output, but to establish AI evaluations as the missing layer that operates "on the loop above at the top." This approach requires technical and non-technical leaders to align upfront on what success looks like—slowing down briefly before speeding up.

While AI evaluations are standard in specialized tech environments, non-tech industries remain largely unaware of them. Davis highlights her research from the City University of New York (CUNY) Executive Program, where she spent nine months interviewing global newsroom leaders about their AI product workflows. Strikingly, very few organizations were actively experimenting with AI evaluations, and most had never even heard of the practice. This finding revealed a critical industry-wide gap: non-tech organizations that rely heavily on institutional trust need concrete frameworks to determine precisely when human judgment must intervene versus when automated evaluations can reliably supervise output quality.

```mermaid flowchart TD accTitle: Evolution of AI Governance Architecture

subgraph Legacy["Legacy Static Governance (In-the-Loop)"] A["LLM Output"] --> B["Human Reviewer"] B -->|"Exhausting Manual Check"| C["Approved Content"] end

subgraph Modern["Modern Automated Governance (On-the-Loop)"] D["Pre-defined Standards"] --> E["Supervising Evaluator Agent"] F["LLM Outputs"] --> E E -->|"Automated Eval (>99% Acc.)"| G["Validated Content"] E -.->|"Flagged Exceptions"| H["Human Escalation"] end ```

[06:58](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=418s) Aligning Product, Governance, and the Risk-Value Matrix

Within an enterprise, functional tension often emerges between product teams and governance teams. Product leaders are tasked with driving business outcomes and rapid delivery, whereas governance teams focus on mitigating risk, monitoring compliance, and ensuring safety. To resolve this friction, USA TODAY Co. placed empirical data at the center of both disciplines, supported directly by AI evaluation benchmarks.

By establishing a structured Risk-Value Matrix, the organization created a formal mechanism to navigate enterprise risk thresholds. When product initiatives demonstrated exceptionally high business value paired with specific evaluation metrics, teams established a clear escalation path to executive leadership to deliberately raise the acceptable risk tolerance. Grounding these governance debates in objective evaluation data transformed the operational pace, accelerating release cycles from months of deliberation down to weeks and days.

[08:23](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=503s) Case Study: Automating FOIA Request Generation with a Supervising Agent

Davis illustrates the transformative impact of automated evaluations through a real-world newsroom project involving the Freedom of Information Act (FOIA) and state-level public records laws. Journalists routinely rely on public records requests to hold government officials accountable, but navigating the divergent legal requirements and submission procedures across all 50 states is complex and time-consuming. Newsrooms asked the AI product team to build a system capable of batch-generating accurate, legally sound records requests that government agencies would accept.

For over nine months, the team labored through traditional prompt engineering and iterative testing, yet the model repeatedly hallucinated government email addresses and legal specifics. Whenever one bug was resolved, another emerged, leaving the tool stuck at an unacceptable 75% accuracy rate—a fatal flaw since a single inaccuracy leads agencies to outright reject requests. The breakthrough occurred when the team moved away from manual prompt tweaking and deployed a dedicated supervising agent loaded with over 100 sample public records requests to evaluate generations programmatically. This evaluation-driven debugging boosted accuracy to over 99% across the majority of US states. Furthermore, when journalists requested new features, such as formatting requests for specialized agency web forms, the team was able to ship fully evaluated updates within days rather than months.

- **Initial Roadblock:** Over 9 months of manual tuning stalled at approximately 75% accuracy due to hallucinated contact details and complex multi-state legal requirements. - **Evaluation Mechanism:** Built a supervising AI agent trained on 100+ vetted sample public records requests to systematically evaluate and score outputs. - **Outcome:** Pushed accuracy to over 99% across most states and reduced feature development cycles from months to days.

[11:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=664s) The Leadership Mandate: Continuous Monitoring and Defining the Human Role

Looking ahead, Davis outlines the ongoing modernization of enterprise governance, emphasizing the transition from single-tool implementations to continuous, cross-enterprise monitoring. While USA TODAY Co. successfully proved the evaluation model on individual tools like the FOIA generator, scaling automated monitoring across the entire enterprise software suite remains an active priority.

Davis concludes by emphasizing that the future of governance relies on high-level leadership clarity rather than technical tooling alone. Executives must clearly define what distinct role only humans can play, allowing organizations to stop "babysitting" every output and instead deploy human judgment where readers, customers, and communities need it most.

- **Current Enterprise Goal:** Expanding automated evaluation and monitoring capabilities across all enterprise AI tools. - **Core Leadership Question:** Identifying where systems can be trusted independently so human experts can focus exclusively on high-impact editorial and community needs.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) I can't keep up — Jessica Davis

> [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) I'm tired — Jessica Davis

> [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) When does the time savings come? — Jessica Davis

> [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) Is this AI content right? — Jessica Davis

> [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) process fatigue — Jessica Davis

> [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) judgment fatigue — Jessica Davis

> [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) on the loop above at the top — Jessica Davis

> [11:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=664s) babysitting — Jessica Davis

## Mentioned Jessica Davis, USA TODAY Co., Inc, USA TODAY, City University of New York (CUNY) Executive Program, Freedom of Information Act (FOIA)

## Video description Jessica Davis - Vice President, AI Product, USA TODAY Co., Inc.

AI fatigue is real and your governance model may be making it worse. As AI scales, manual review creates bottlenecks, not quality. I'll share how USA TODAY Co. is modernizing governance with automated AI evaluations: structured standards that scale without the burden. Leave with a practical framework for stronger accountability and less manual overhead. If the old model isn't holding, this talk is for you.

[Source](https://www.youtube.com/watch?v=bWbMQo1G_L8)

中文简报

USA TODAY Co. 部署 AI 评估智能体取代人工审查人工在环验证会引发判断疲劳且无法应对非确定性模型，需要自动化监督评估来执行准确性标准。 Jessica Davis 认为，企业对 AI 的疲劳感源于员工被要求人工审查每一个非确定性输出时所产生的判断力耗竭。为解决这一问题，USA TODAY Co. 在实证风险-价值矩阵（Risk-Value Matrix）的支持下，将治理模式从“在环”（in-the-loop）人工检查转变为“环上”（on-the-loop）自动化 AI 评估。一个新闻采编室案例表明，部署监督评估智能体后，公共记录生成的准确率从停滞不前的 75% 提升至 99% 以上，同时发布周期从数月缩短至数天。Davis 总结指出，企业领导者必须明确界定人类判断的独特作用，而不是强迫员工去“看管”AI 系统。 USA TODAY Co. 的 AI 产品副总裁 Jessica Davis 指出，企业 AI 倦怠的根源在于工作流已重塑但组织预期未变；依靠人工逐条审查生成内容会导致不可持续的“判断疲劳”。 (0:04) 传统的静态治理框架无法应对生成式 AI 的非确定性输出，治理模式需从人工在环（in-the-loop）转向由自动化 AI 评估在上层监督（on-the-loop）。 (4:14) 通过建立“风险-价值矩阵”并以自动化评估数据为核心，团队消除了产品交付速度与合规风控间的冲突，将治理决策与发布周期从数月压缩至数天。 (6:58) 在全美各州公开发布记录（FOIA）请求生成案例中，传统提示词工程准确率长期停滞在约 75%，而在引入基于 100 多份样本训练的专属监督评估 Agent 后，生成准确率提升至 99% 以上。 (8:23) 企业 AI 治理的现代化重点在于实现跨工具的持续自动化监控，领导层必须明确定义人类不可替代的核心职责，摆脱对日常输出的“保姆式”审查。 (11:04)

Reading summary

According to Jessica Davis, Vice President of AI Product at USA TODAY Co., enterprise AI fatigue stems from judgment exhaustion caused by treating manual human review as a universal filter. To sustainably scale adoption, organizations must replace static in-the-loop checks with automated 'on-the-loop' AI evaluations that programmatically benchmark non-deterministic outputs. Manual Review Pitfalls and Judgment Fatigue (0:04) Across USA TODAY Co.'s footprint of USA TODAY and 200 local newsrooms, Davis observed widespread employee exhaustion characterized by concerns over when promised time savings would materialize. An audience poll reflected this split evenly between workers saving time and those burdened by additional tasks. Davis argued that this issue is rooted in an operational mismatch: while AI tools have transformed daily workflows, enterprise expectations remain static. In high-trust fields like journalism, forcing employees to verify whether every generation is correct imposes an unsustainable demand on human attention. Davis clarified that governance breakdowns in these environments represent judgment fatigue rather than process fatigue. While human validation remains necessary because models hallucinate, expecting individuals to maintain constant critical vigilance over every single output is unfeasible. Solving this operational friction is fundamentally a leadership challenge rather than a technical problem, requiring executives to reassess whether staff are serving the AI rather than the AI serving staff. The Need for Automated AI Evaluations (4:14) Traditional enterprise governance frameworks were designed for deterministic software, where code operates predictably. Large language models, however, are non-deterministic and can produce divergent outputs from identical prompts. Reviewing these outputs requires simultaneously tracking accuracy, usefulness, user adoption, safety, and fairness—criteria that quickly overwhelm manual, static governance workflows. Davis proposed moving human oversight from 'inside the loop' to 'on the loop' by introducing automated AI evaluations that benchmark generations against predefined standards before escalating exceptions to human reviewers. Despite their adoption in specialized tech companies, automated evaluations remain largely absent in non-tech sectors. During a nine-month research initiative in the City University of New York (CUNY) Executive Program, Davis interviewed global newsroom leaders regarding their AI product workflows. She found that very few organizations were testing AI evaluations, and most were unfamiliar with the concept. Davis identified this lack of awareness as a critical vulnerability for trust-dependent institutions that need structured thresholds to determine when human intervention is truly necessary. Resolving Enterprise Friction via Risk-Value Alignment (6:58) Enterprises frequently experience organizational tension between product teams, which are incentivized to drive business velocity and transformation, and governance teams, which focus on mitigating risk and ensuring compliance. To bridge this divide, USA TODAY Co. integrated AI evaluation benchmarks as a shared data layer between both functions. By implementing a formal Risk-Value Matrix, the organization created an objective mechanism to negotiate risk thresholds. When product teams used evaluation data to prove that an initiative delivered substantial business value while maintaining measurable safety benchmarks, they could escalate decisions to executive leadership to intentionally expand risk tolerance. Grounding these deliberations in empirical metrics streamlined decision-making, reducing deployment cycles from months of governance debate down to weeks and days. Automating Public Records Requests at Scale (8:23) Davis demonstrated the utility of automated evaluations through a newsroom project automating public records requests under the Freedom of Information Act (FOIA) and varied state laws across all 50 US states. Because government agencies reject filings containing even minor inaccuracies, the system required near-flawless precision. For over nine months, traditional prompt engineering and iterative testing remained stalled at roughly 75% accuracy, consistently hallucinating legal specifics and agency email addresses. The development team broke this impasse by deploying a dedicated supervising AI agent trained on more than 100 verified sample requests to programmatically evaluate and score generations. This evaluation-driven approach elevated system accuracy to over 99% across most states. When reporters subsequently requested adjustments, such as reformatting outputs for distinct agency web portals, the team deployed fully validated updates within days instead of months. Davis stated that modernizing enterprise governance requires expanding from isolated tool implementations to continuous, cross-enterprise monitoring. The decisive leadership mandate is to clearly define where automated systems can be trusted independently, freeing human professionals from babysitting routine outputs so their critical judgment can focus on high-impact editorial and community responsibilities.

精炼阅读摘要

USA TODAY Co. 人工智能产品副总裁 Jessica Davis 指出，在生成式 AI 部署中强行要求人工逐条审核会导致“判断疲劳”并引发普遍倦怠；企业必须突破传统的静态治理，引入自动化“回路之上”的 AI 评估层，以客观数据平衡业务敏捷度与风险控制。人工全量复核引发的“判断疲劳”陷阱 (0:04) 掌管《今日美国》（USA TODAY）及全美200家地方编辑部的 USA TODAY Co. 观察到，AI 工具虽然重塑了业务流程，但未能在全企业层面普遍兑现节省时间的承诺；现场民调显示，认为 AI 提升效率与认为其增加额外负担的参会者各占一半。Davis 指出，在对准确性要求极高的新闻采编等信任密集型环境中，要求员工对每一条非确定性的生成结果持续保持高度警惕与批判性审视，导致治理模型在“判断疲劳”（judgment fatigue）下崩溃。这种将人类审查作为所有输出强制过滤器的做法，在未调整运营期望的情况下悄然加重了日常职责，本质上属于管理层领导力调整的缺位，而非单纯的技术流程缺陷。传统静态治理失效与评估监督层的引入 (4:14) 传统的企业治理框架专为行为可预测的静态、确定性软件构建，然而大语言模型具有非确定性，相同的提示词可能生成截然不同的内容，导致静态审查标准无法应对生成结果的多变性。Davis 主张治理架构应从让审查人员置身于执行回路内部（in-the-loop），升级为在系统顶层运行自动化的 AI 评估层（on-the-loop），通过预设标准和监督代理来承担常规审查，仅将异常情况升级至人工复核。然而，Davis 在纽约市立大学（CUNY）高管项目针对全球编辑部负责人开展的长达9个月的调研显示，非技术行业对 AI 评估技术普遍缺乏认知且鲜有实验，在高度依赖机构公信力的行业中形成了关键的治理工具断层。利用风险-价值矩阵化解跨部门目标张力 (6:58) 在企业内部，追求业务成果与快速交付的产品团队，与专注于规避合规漏洞、保障安全与公平性的治理团队之间经常存在功能性摩擦。USA TODAY Co. 通过将客观的 AI 评估基准数据作为两者的共同支撑，构建了结构化的“风险-价值矩阵”（Risk-Value Matrix）。当特定产品方案展现出极高的业务转化价值，并附有严谨的量化评估数据作为风险佐证时，团队便拥有规范的升级通道报请高管层主动调高风险承受阈值。这一机制将治理分歧从主观争论转变为基于数据的决策，成功将产品功能的研讨与发布周期从数月压缩至数周乃至数天。监督代理将公共记录申请准确率提至99% (8:23) 在全美各州法律要求各异的《信息自由法》（FOIA）公共记录申请工具开发中，准确性是决定政府机构是否受理的硬指标。该项目最初历经9个多月的传统提示词工程调试与人工排错，但由于模型持续幻觉生成错误的政府邮箱与法律细节，系统准确率长期徘徊在不可接受的75%基准线上。团队随后放弃单纯的手工调优，转而构建了一个载入100多个已核验样本的专属监督 AI 代理，利用自动化程序对生成文本实施系统化评估。这一评估驱动的纠错模式推动系统在全美绝大多数州的准确率突破99%，并使后续针对特定机构网页表单格式的新功能交付周期从数月锐减至数天。 Davis 强调，治理现代化的下一步是从单一工具的评估验证迈向全企业软件生态的持续自动化监控。高层领导者的核心任务在于明确界定只有人类才能胜任的关键角色，使员工彻底摆脱对基础 AI 内容的机械式看护，将宝贵的专业判断力集中投入到高影响力的采编工作与社区连接中。

Detailed notes — zh

## Background 该内容由频道 Ai4 于 2026 年 8 月 20 日发布在 YouTube 平台上。视频记录了 USA TODAY Co., Inc. AI 产品副总裁 Jessica Davis 的主题演讲，核心探讨了随着 AI 规模化应用带来的“AI 疲劳感”，以及如何通过引入自动化 AI 评估体系来重构治理模式、减少人工微观复核的负担。 ## TL;DR USA TODAY Co. 的 Jessica Davis 指出，企业面临的 AI 疲劳感并非源于技术本身，而是因为传统的“人在环路”逐条人工复核模式导致人类判断力严重过载。为了破解这一困境，组织需要转向“人在环上”的治理架构，引入涵盖准确性、实用性、安全性与公平性的 AI 评估（Evaluations）中间层。通过构建“风险-价值矩阵”和监督 Agent，USA TODAY Co. 成功将 FOIA 信息公开申请等高幻觉场景的准确率提升至近 100%，并将决策发布周期大幅缩短。演讲强调，领导层必须停止让员工为 AI 做“保姆”，通过建立对自动化评估的信任，将人类生产力释放至更高价值的判断与服务中。 ## Atmosphere & style 整场演讲呈现出专业、坦诚且极具务实洞察的氛围。Jessica Davis 语言干练，开场便通过现场举手互动切中业界痛点，打破了盲目乐观的 AI 叙事，流露出对一线员工真实疲劳感的情感共鸣。随着对传统复核模式局限性的剖析和实际案例数据的引入，演讲节奏从问题反思逐步转向清晰有力的解法输出，展现出高管层在技术转型潮中的战略定力与实践智慧。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) 全美最大出版集团 USA TODAY Co., Inc. 的 AI 产品副总裁 Jessica Davis 指出，企业界普遍存在严重的“AI 疲劳感”，AI 带来的“节省时间”承诺在实际应用中往往未能实现。 - [02:05](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=125s) AI 治理与应用瓶颈的根本原因在于人类判断力的过载，而非流程疲劳；领导层未能及时调整工作模式与产出预期，导致人类员工沦为 AI 输出的“保姆”。 - [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) 针对 AI 动态输出的特性，USA TODAY Co. 提出了评估 AI 质量的四大维度：准确性（Accuracy）、实用性（Utility）、安全性（Safety）以及公平性（Fairness）。 - [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) AI 治理模式需从逐条逐句人工复核的“人在环路”（Human-in-the-loop）升级为宏观把控与标准定义的“人在环上”（Human-on-the-loop）。 - [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) 调研显示，多数非科技类新闻媒体在面对 AI 转型时，普遍缺乏针对 AI 产品开发与自动化评估（AI evaluations）的标准框架与认知。 - [06:58](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=418s) 数据是连接产品部门（追求业务效益）与治理部门（追求风险合规）的核心枢纽与桥梁。 - [06:58](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=418s) 通过引入“风险-价值矩阵”与数据驱动的沟通机制，高价值项目可争取更高的风险容忍度，从而将产品上线周期从几个月大幅缩短至几天。 - [08:23](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=503s) 在批量提交 FOIA 申请的实际案例中，通过构建监督 Agent 并结合标杆样本进行 AI 评估，将系统准确率从 75% 提升至近 100%。 - [11:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=664s) 未来的 AI 治理挑战在于将自动化评估与实时监控能力，从单一试点工具扩展到全企业范围内的多个工具与业务线上。 - [11:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=664s) 现代化 AI 治理需要领导力的变革：建立对自动化评估系统的信任，停止微观的“保姆式”复核，将人类生产力重新释放到高价值的判断与服务中。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) AI 疲劳感与治理模式重构在研讨会接近尾声的演讲中，USA TODAY Co., Inc. 的 AI 产品副总裁 Jessica Davis 探讨了一个在当下企业界普遍存在的现象——“AI 疲劳感”（AI fatigue）。作为全美最大的出版集团，USA TODAY Co. 旗下拥有遍布全美各地 200 家地方新闻机构以及旗舰报纸《USA Today》。Jessica 提到，她在四年前就在公司内部创立了 AI 治理体系，致力于确保 AI 技术在新闻和商业层面的责任化应用。然而，在她与一线记者、销售领导层以及其他企业高管的交流中，常常听到诸如“我实在跟不上了”、“我很累”、“到底什么时候才能真正节省时间”等抱怨。为了现场验证这一痛点，Jessica 邀请在场听众举手示意：有人认为 AI 确实节省了时间，也有不少人觉得 AI 反而增加了工作负担，现场观众的反馈几乎呈现一分为二的对立态势。这种期望与现实的落差表明，尽管应用 AI 工具后工作流程发生了重大改变，但员工依然觉得工作量不减反增。 - **组织背景与规模**：USA TODAY Co., Inc. 拥有 200 家地方新闻机构及《USA Today》，是美国最大的出版集团。 - **治理体系建构**：Jessica Davis 在四年前创立了公司的 AI 治理框架，旨在全面保障 AI 的负责任应用。 - **普遍面临的困境**：企业普遍反映 AI 带来的“时间节省”承诺未达预期，反而引发了显著的“AI 疲劳感”。 --- ### [02:05](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=125s) 人工复核的困境与“人在环路”的局限对于新闻出版业而言，产品的核心竞争力建立在公众信任之上，因此“确保 AI 生成的新闻内容准确无误”成为重中之重。Jessica 强调，当 AI 治理或应用出现瓶颈时，根本原因往往不在于流程疲劳（process fatigue），而在于人类判断力（judgment）的过载。在 USA TODAY Co.，“人在环路”（Human-in-the-loop）模式曾被视为质量把关的铁律，员工自觉需要去检查和复核 AI 生成的每一条输出（output）。然而，这种无休止的监督要求员工时刻保持警惕，不断判断“这是否达到了良好标准”。人类不可能对庞大的生成内容逐一进行高质量的判断，这种微观层面的把关让工作性质悄然改变，但组织对产出的预期却并未调整。她指出：“这是一个领导力问题，而非技术问题。”如果领导层不及时停下脚步审视“究竟是人类在为 AI 打工，还是 AI 在为人类服务”，这种期望与实际工作模式之间的鸿沟就会持续演变成深度的疲劳感。 - **核心矛盾**：评估 AI 输出需要持续投入人类判断力，而逐条人工复核在规模化应用中难以持久。 - **领导力反思**：治理困境并非技术缺陷，而是领导层未能及时调整工作模式与产出预期。 - **痛点根源**：人类员工被逼成为 AI 输出的“保姆”，导致工作负担不降反升。 --- ### [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) 引入 AI 评估（Evaluations）作为关键中间层在传统的静态软件时代，治理模式针对的是确定性系统；但 AI 并非静态软件，相同的提示词（prompt）在不同时间可能产生完全不同的回答。 Jessica 提出了评估 AI 质量的四大维度：准确性（Accuracy）、实用性（Utility）、安全性（Safety）以及公平性（Fairness）。对新闻业而言，准确性是最低门槛（table stakes），但产品能否获得用户吸纳（adoption）以及是否安全公平同样至关重要。为了打破“逐条复核”的死循环，Jessica 提出将“人在环路”（Human-in-the-loop）升级为“人在环上”（Human-on-the-loop）。这意味着在正式部署前，管理者与技术团队需先停下脚步，共同定义“什么是成功”与“优秀的标准是什么”。Jessica 分享了她在纽约市立大学（CUNY）高管项目期间为期九个月的研究成果：在采访全球多家新闻媒体高管时，她发现极少数人在 AI 产品开发中尝试过 AI 评估（AI evaluations），大部分人甚至从未听说过这一概念。这表明非科技行业在面对 AI 转型时，普遍缺乏一套能够平衡风险与效率的标准框架。 - **评估四大维度**：准确性（Accuracy）、实用性（Utility）、安全性（Safety）、公平性（Fairness）。 - **模式变革**：从针对每条输出的“人在环路”（Human-in-loop）转向宏观把控与标准定义的“人在环上”（Human-on-loop）。 - **行业现状**：据 CUNY 研究调研，多数非科技类新闻媒体对 AI 自动化评估（AI evaluations）仍缺乏认知与实践。 --- ### [06:58](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=418s) 风险-价值矩阵与数据驱动决策在组织架构与职责划分上，产品部门（Product）通常致力于推动业务成果与绩效考核，而治理部门（Governance）则侧重于风险控制与合规监管。这种职能差异容易在组织内部引发权责混淆与摩擦。USA TODAY Co. 发现，连接这两者的核心枢纽在于“数据”（Data）。为了加快这家传统媒体巨头的数字化转型速度，团队开发了一套“风险-价值矩阵”（Risk-Value Matrix）。当某一 AI 应用场景能带来极高业务价值时，团队可以通过数据向高管及决策层（Executive Team）发起提级倒逼（escalation path），从而合理提升风险容忍度门槛（risk threshold）。这种基于数据的沟通机制，成功将过去需要数月才能完成的决策与产品上线周期，压缩到了数周甚至数天。 - **权责对立与统一**：产品部门关注业务产出，治理部门关注风险合规，而“数据”是统一二者的核心桥梁。 - **风险-价值矩阵**：利用该矩阵评估应用场景，高价值项目可争取更高的风险容忍度。 - **决策效率提升**：基于数据的评估与沟通机制将产品发布周期从几个月缩短至几天。 --- ### [08:23](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=503s) 实践案例：利用 AI 监督 Agent 破解 FOIA 申请难题为了说明评估机制的效果，Jessica 分享了一个具体的新闻采编案例：根据《信息自由法》（FOIA）及各州公共记录法，记者需要向政府部门提交信息公开申请以监督公权力。然而，全美 50 个州及联邦政府拥有不同的法律条款和提交渠道，批量提交 FOIA 申请极易引发 AI 幻觉（hallucination）。在长达九个月的时间里，AI 不断幻觉出假的电子邮箱或格式错误，导致申请准确率卡在 75% 左右——对于严谨的新闻机构来说，这一准确率完全不可接受，因为错误的申请会被政府机构直接拒绝。后来，团队改变思路，构建了一个“监督 Agent”（Supervising Agent），并导入了 100 多个样本记录申请作为标准参照，利用 AI 自身来进行规模化评估（evaluations）。通过将人类的专业判断转化为评估标准并定位调试，系统准确率一举提升到了 99% 以上（接近 100%）。当需要新增支持表单提交等功能时，团队只需几天即可完成迭代部署，极大解放了记者的生产力。 - **应用场景**：自动化批量提交《信息自由法》（FOIA）及各州公共记录公开申请。 - **技术瓶颈**：初期因 50 州法律差异与提交渠道复杂，AI 频繁产生幻觉，准确率长期停滞在 75%。 - **解决方案与突破**：构建监督 Agent 并载入 100+ 标杆样本，将准确率一举提升至近 100%，新功能开发周期缩短至几天。 --- ### [11:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=664s) 未来展望与人机协作的终极定位在演讲最后，Jessica 展示了现代化 AI 治理的整体框架，并明确了 USA TODAY Co. 下一步需要解决的课题。虽然团队已经在单个工具上成功实施了自动化评估，但未来的挑战在于如何将这种实时监控（monitoring）能力拓展到全企业范围内的多个工具与业务线上。这不仅是一个技术监控问题，更是一个关乎领导力（Leadership）的深刻议题。领导层必须明确人类在 AI 时代不可替代的独特角色，彻底摆脱对系统输出进行微观“监护”（babysitting）的旧模式。只有建立起对自动化评估系统的信任，员工才能将宝贵的精力重新投放到真正需要人类判断的地方——去服务读者、社区与广大受众。 - **未来演进方向**：从单一工具的评估试点，扩展到全企业级多工具的实时监控与治理。 - **领导层核心任务**：明确人类员工不可替代的独特价值，停止对 AI 输出的微观“保姆式”复核。 - **终极目标**：建立系统信任，将人类生产力释放到高价值的社区服务与内容创作中。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) 我实在跟不上了 — 一线员工 / 记者 > [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) 我很累 — 一线员工 / 记者 > [00:04](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=4s) 到底什么时候才能真正节省时间 — 企业高管 / 员工 > [02:05](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=125s) 这是一个领导力问题，而非技术问题。 — Jessica Davis > [02:05](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=125s) 究竟是人类在为 AI 打工，还是 AI 在为人类服务 — Jessica Davis > [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) 什么是成功 — Jessica Davis > [04:14](https://www.youtube.com/watch?v=bWbMQo1G_L8&t=254s) 优秀的标准是什么 — Jessica Davis ## Mentioned Jessica Davis, USA TODAY Co., Inc., USA Today, Ai4, 纽约市立大学 (CUNY), 信息自由法 (FOIA), AI, Human-in-the-loop, Human-on-the-loop, Supervising Agent ## Video description Jessica Davis - Vice President, AI Product, USA TODAY Co., Inc. AI fatigue is real and your governance model may be making it worse. As AI scales, manual review creates bottlenecks, not quality. I'll share how USA TODAY Co. is modernizing governance with automated AI evaluations: structured standards that scale without the burden. Leave with a practical framework for stronger accountability and less manual overhead. If the old model isn't holding, this talk is for you. [Source](https://www.youtube.com/watch?v=bWbMQo1G_L8)

Source quotes

[0:04] 一线员工 / 记者: 我实在跟不上了 [0:04] 一线员工 / 记者: 我很累 [0:04] 企业高管 / 员工: 到底什么时候才能真正节省时间 [2:05] Jessica Davis: 这是一个领导力问题，而非技术问题。 [2:05] Jessica Davis: 究竟是人类在为 AI 打工，还是 AI 在为人类服务 [4:14] Jessica Davis: 什么是成功 [4:14] Jessica Davis: 优秀的标准是什么

Source quotes

[0:04] Jessica Davis: I can't keep up [0:04] Jessica Davis: I'm tired [0:04] Jessica Davis: When does the time savings come? [0:04] Jessica Davis: Is this AI content right? [0:04] Jessica Davis: process fatigue [0:04] Jessica Davis: judgment fatigue [4:14] Jessica Davis: on the loop above at the top [11:04] Jessica Davis: babysitting

Original transcript

[0:04] Thank you so much. We're toward the end of the day and we're going to be talking about fatigue. I'm [0:10] Jessica Davis, vice president of AI product at USA Today Co. We include um 200 local newsrooms across [0:18] the United States and USA Today. Uh so we're America's largest publisher. So I work with [0:25] journalists, sales leaders, I work across the enterprise because I founded our AI governance [0:31] four years ago um to ensure responsible use of AI across our organization. Things that I've heard in [0:39] that work and at this conference um probably sound pretty familiar to you, right? I can't keep up. [0:49] I'm tired. When does the time savings come? So, we'll be talking about all of that today. And if [0:56] that's you right now, right, there is another way to use AI. You've probably heard this in [1:00] other rooms today. Um, and we'll hopefully have a couple minutes for questions. So, really, if [1:07] you've been here, you've heard the question, the AI ROI, right? Or another way, did AI live up to [1:14] its promise of saved time? Well, it just depends, right? Um, does AI save you time? Raise your hand, [1:21] please. Do you feel that that's true for you? Great. Does anyone feel like it's not like it's [1:27] added work for you? So, the room is split almost to half. Um, and what we've really experienced is, [1:36] um, you know, what organizations expected hasn't really been what's been delivered. Um, and so, [1:46] right, the workflows have shifted with these tools. Um, but people feel like, hey, when is this [1:57] going to catch up? Um, there's more work to be done than ever before. And the main question that [2:05] these systems produce for people in day-to-day work is, especially for a company like mine where [2:11] trust is crucial to our product, right? Getting the news right, is this AI content right? and the [2:19] time it spends that people spend reviewing that can be exhausting. So without clear standards, [2:27] right, that could never be fully answered. So we'll be talking about that some more. And when [2:34] we think about governance and ensuring that people, you know, are doing responsible AI, [2:41] when that breaks down, that's not process fatigue. It's the judgment, right? Reviewing [2:49] all the outputs. Human the loop is really important to our work using AI at USA Today [2:57] um because it can be wrong and our audiences trust us to get it right. Um, so we have people who feel [3:05] it's their responsibility to check every output. And I hear in other industries, in the rooms I've [3:11] been in, they feel the same way too, depending on your industry and depending on what does your [3:17] customer need from you, what do they expect from you, how do they use your product. Um, and so this [3:24] requires constant vigilance of is this what good looks like? And that's just not humanly possible [3:31] to bring that judgment to every output. Um, and it has to be done over and over again. So this job [3:41] quietly changed but the expectations never did. Right? Unless, and this is a leadership issue, [3:48] not an AI issue, not a technology issue. Has your organization taken a pause and say what [3:55] is it? Do are we working for the AI or the AI working for us? And so that's what changed, [4:03] right? The work change, the expectations didn't and the gap that's where that fatigue lives. [4:14] Why AI reviewing feel why why reviewing AI feels hard? We've put a couple examples here um just [4:20] to right for us especially is it accurate is table stakes like the for news organization I had shared [4:28] um but also is it useful right do you have adoption um of the products you create using AI [4:34] is it safe is it fair um and all these things have to be evaluated um individually and governance was [4:43] built for a static world a world that right AI is not static software it's changing you prompt an [4:53] AI and it may give you a different answer if you prompt it multiple times you know this you're not [4:58] beginners to AI maybe some of you are um and so governance for a static system that doesn't work [5:07] for this so that explains some of that fatigue we really believe that evaluations are the missing [5:15] layer what does good look like defined not in the loop for every output on the loop above at [5:23] the top. Taking that pause before to slow down before we speed up just take a breath. What [5:32] does success look like is really an important step that technical and non-technical leaders [5:40] alike can unite around. AI evaluations have been around for a while and so that's not new. I just [5:48] graduated from um City University New York's executive uh program and I did nine months of [5:54] research and interviewed news leaders across the globe and I asked are you using AI valuations in [6:00] your AI product work? What time savings are you seeing? What value are you getting out of it? [6:05] And very few said they were experimenting with AI evaluations. Some were aware of the practice um [6:12] but most hadn't ever heard of it. We're not a tech we're not a tech industry. We're a news industry. [6:18] Um and so that was really telling for me that we needed to set a standard for how when trust is [6:25] really important. How do we define evaluations for the media industry? As we've been going though, um [6:33] and I'll talk about an example of one that we did in just a few minutes, right? I'm learning that, [6:40] you know, there are many non- tech organizations that are on the same journey. When do we know [6:46] when to get involved and use our judgment, right? Versus that exhausting checking every time. So, [6:58] who owns what? Um I sit in product and I founded our AI governance with other partners across the [7:06] organization four years ago like I shared in the introduction if you join late and what we found [7:12] is it confused people because product's job is to hold people accountable to business outcomes [7:18] whereas governance right um is measuring risk and monitoring right are we meeting compliance needs [7:26] um what we found at the middle and the AI evaluation findings um support this, right? [7:32] Data is at the center and that's what unites both sides. We developed a risk um value matrix [7:40] where we were able to say, hey, let's work as an organization so that we can move faster in our [7:46] transformation as a legacy media organization. And we did find that did move us forward, right? When [7:54] we're specific and when the value was greater, we're able to have an escalation path to our [8:00] leadership and executive team saying, "We need to raise um our risk threshold." These conversations [8:10] didn't happen overnight, but it's when we started applying the data, that's when they accelerated [8:15] from months to weeks to days in shipping, find getting more value out of what we built. [8:23] So I wanted to give an example. Um you may have heard of the freedom of information act. This [8:28] is a really crucial tool and a law that allows journalists to get information from government [8:34] officials and hold uh power accountable in our country and each state has its own public [8:42] records laws. Our use case and the problem our journalists asked us to solve. How do we submit [8:48] these requests in a way that's accurate that government officials respond to and multiple [8:56] at the same time in batch? We labored over this for I it like over nine months and it [9:05] kept hallucinating. We kept following best practices and we kept stumping it with the [9:10] complexity of 50 states, different laws, federal laws, different ways to submit foyer requests. [9:16] The AI kept hallucinating email addresses. There were just so many things that were going wrong and [9:22] we would solve one and another issue would pop up. Our journalists worked with us step by step and we [9:28] um we just kept feeding it and kept finding the errors. We were about 75% accuracy rate and that [9:35] just wasn't good enough for us. It wasn't until we built a supervising agent. We loaded up over [9:43] a hundred um sample record requests and we were able to use AI to scale that up. We were able [9:50] to get to over 90 um 9% like like just a fraction of like almost 100% accuracy. Our journalists are [10:01] using it and we've been able to get to that rate across multiple states like the majority of them. [10:08] It wasn't until we embraced AI, putting our judgment in it as the evaluator, identifying where [10:16] we needed to put more attention in debugging. That is when things really sped up. Um, also we [10:25] started getting more requests from our users. Hey, sometimes the state or federal agency we have to [10:33] submit via form. we were able to add that feature within days. Whereas before we cycled through [10:41] months at a level that met our criteria for what was excellent, acceptable, and actually useful to [10:48] our journalists because if you submit this wrong, the government agency declines your request. And [10:57] so that is what's at stake here is actually helping our journalists do their work better. [11:04] So this framework right here's the single framework that you can carry out of this [11:09] room you know yes you can take photo of it and really this is then now and where we're heading [11:18] next what we're still solving for and I think it's important to share right yes we have been really [11:25] working hard at this we want to transform as an organization and as an industry we know a lot [11:32] is at stake And we think AI can help us. What we need to solve for is the monitoring. We did this [11:41] with one of our tools. We want to do it though across more of the organization across multiple [11:47] enterprise tools. We still are working on turning that capability on um and really being clear [11:58] about the role of humans. These are leadership conversations though also monitoring maybe a [12:03] technology one but the leadership conversation about what do we expect of our people what is [12:10] the role only they can play and where can we lean on systems to where we're not babysitting them on [12:16] every output when can we trust those systems and put ourselves where our customers and our readers [12:22] and our audience needs us right the communities that we serve. So that's what's at stake and [12:27] thanks for for being here. You can connect with me on LinkedIn and happy to be in touch.