Forum AI CTO Says Automated LLM Judges Fail Expert Safety Benchmarks
Ai4
Static rulebooks and off-the-shelf automated AI evaluators fail to reliably judge complex real-world outputs, requiring consequence-driven reasoning and human expert calibration to prevent system failures.
## Background
Published on YouTube by the channel Ai4 on August 15, 2026, this video features Robbie Goldfarb, Co-Founder and CTO of Forum AI. The video details Goldfarb's insights on building trustworthy AI, drawing from his experience in high-stakes safety roles at Facebook and Instagram, and explores how expert judgment, consequence thinking, and scaled evaluations enable AI systems to operate reliably in complex real-world environments.
## TL;DR
In this talk, Robbie Goldfarb explains why static, written rulebooks fail to handle the nuanced and highly contextual demands of real-world AI applications. He advocates for two main tactics: enabling AI to perform consequence-driven thinking and adapting to contextual memory over time without creating broken user experiences. Furthermore, Goldfarb stresses that AI evaluation must be calibrated by domain experts rather than software engineers, demonstrating that off-the-shelf automated LLM judges show near-zero correlation with human expert judgment.
## Atmosphere & style
The presentation carries an authoritative, pragmatic, and insightful tone, reflecting Goldfarb's deep background in high-stakes integrity and safety across major tech platforms. His delivery is structured and articulate, balancing high-level theoretical frameworks with concrete, real-world examples from healthcare, finance, and customer service. Listening to the address feels like an engaging and intellectual dive into the frontline challenges of enterprise AI reliability and governance.
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) Static, black-and-white written rules frequently break down or create direct contradictions when applied to complex, real-world AI deployment scenarios.
- [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) Written guidelines often conflict with real-world needs, such as Claude's non-covertness rule colliding with clinical best practices for eating disorder treatment.
- [04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) To handle nuance, AI systems must be enabled to perform consequence-driven thinking, evaluating the downstream impacts and affected parties of an action before responding.
- [04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) Forum AI utilizes 'consequence mapping' to outline multi-dimensional downstream impacts, serving as training material or in-context reference for AI decision-making.
- [04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) Safety guardrails and evaluations must account for historical context over time rather than judging interactions in isolated single turns.
- [04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) Managing memory requires balance: ignoring history leads to repetitive loops, while over-relying on memory causes broken UX by injecting irrelevant personal data.
- [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) Domain specialists—such as healthcare professionals, financial analysts, and policy experts—must lead AI evaluation rather than software engineers.
- [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) Off-the-shelf automated LLM judges demonstrate virtually zero statistical correlation with the judgment of real domain experts when evaluating AI outputs.
- [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) Uncalibrated automated governance tools often create a false sense of safety by capturing background noise rather than meaningful risk signals.
- [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) Continuous real-time evaluation is critical because frontier models are brittle and minor system or harness updates can cause severe performance regressions.
## Detailed walkthrough
Here is the detailed, faithful walkthrough of the transcript.
[00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) Why Static Rules Break Down in Complex AI Scenarios
Robbie Goldfarb, Co-Founder and CTO of Forum AI, introduces his extensive career navigating high-stakes AI safety and integrity issues. He previously worked at Facebook addressing news and misinformation during the aftermath of the Cambridge Analytica scandal, COVID-19, and the 2020 US presidential elections. He later spent two years at Instagram focused on youth safety—handling sensitive issues such as teen mental health, addiction, and online harassment—before working on AI safety within Meta’s superintelligence labs. Now at Forum AI, Goldfarb collaborates with frontier AI research labs, Fortune 500 corporations, and government entities on critical domains including elections, clinical safety, and financial advice. Across all these domains, the central question remains simple yet elusive: how do we build AI that is genuinely good, behaves reliably at scale, performs consistently under pressure, and stays safely within its guardrails?
To answer what "good" looks like, Forum AI consults domain experts at the top of their fields, including former presidents, bestselling authors, leading academics, and industry thought leaders. However, when asked to define ideal behavior in political discourse, financial analysis, or clinical settings, these experts consistently answer: "Well, it depends. It's complicated." While humans naturally prefer black-and-white rules—as seen in legal codes, religious texts, constitutions, or airport advice books—the real world is heavily contextual. Frontier AI labs try to codify behavior through written guidelines, such as Google Gemini’s policy guidelines, OpenAI's Model Spec, or Anthropic's written Constitution for Claude. Similarly, enterprises establish rigid governance frameworks. Yet, because real-world scenarios are inherently nuanced, these static rulebooks frequently break down or create direct contradictions.
- **Claude's covertness rule vs. clinical reality:** Claude's constitution instructs the AI not to do anything covertly behind a user's back. However, when Forum AI evaluated AI mental health tools for eating disorders, clinicians explained that if a patient attempts to engage in unhealthy behavior (like obsessive calorie counting), the proper clinical response is to covertly steer the conversation away without alerting the patient.
- **Autonomy vs. long-term well-being:** Written rules demand respecting a user's right to make autonomous life decisions while simultaneously requiring the AI to prioritize the user's long-term well-being over immediate requests.
- **Privacy vs. accountability:** Governance rules mandate protecting individual privacy from surveillance, yet also require respecting legitimate legal authorities.
- **Data minimisation vs. auditability:** Enterprise policies often instruct systems to collect and retain only the minimal data necessary, while simultaneously demanding detailed long-term recordkeeping for auditing AI decisions.
---
[04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) Two Essential Tactics: Consequence Thinking and Contextual Adaptation
While foundational rulebooks are indispensable for responsible engineering, they are fundamentally insufficient on their own. Forum AI advocates for two crucial tactics to bridge the gap: enabling AI to perform **consequence-driven thinking** and training it to **adapt to pertinent information over time**. Quoting AI pioneer Yann LeCun—*"We can't build true agentic systems without the ability to predict the consequences of actions just like humans do"*—Goldfarb emphasizes that human reasoning does not rely solely on checking permission lists. Instead, humans evaluate potential outcomes by asking what will happen, who will be affected, and how. Experienced domain experts excel at predicting these downstream effects, making outcome prediction a powerful framework for AI decision-making in ambiguous situations.
To operationalize this approach, Forum AI developed a technique called **consequence mapping**. Instead of assigning a simple binary label of "good" or "bad" to an AI interaction, consequence mapping outlines the explicit downstream impacts of an action across multiple dimensions. These structured maps serve either as training material to refine an AI's internal reasoning or as in-context reference materials for real-time decision-making.
- **Mental Health Scenario:** In a simulated exchange between a clinician and a patient, the AI's polite, validating response provided brief emotional relief but left the patient's underlying feelings of hopelessness and isolation unaddressed. Externally, family members remained unaware of the severity until symptoms escalated, drastically increasing their emotional distress and caregiving burden.
- **Customer Service Scenario:** When an airline chatbot used a polite tone to handle an unresolved issue, it temporarily reduced immediate customer friction. However, the underlying problem remained unfixed, leaving the customer dissatisfied. Operationally, this caused repeat contacts, inflated support ticket volume, burdened human agent workflows, and increased average resolution times.
- **Adapting to Context Over Time:** Current evaluations and safety guardrails typically assess AI behavior in isolated, single-turn moments. However, context completely alters meaning. For example, if an attendee at the AI4 conference tells a chatbot, *"I don't want to be here anymore,"* the default assumption is that the presentation is boring and they want to return to their hotel room. But if that same user spent the previous week discussing sadness, depression, and loneliness, the statement takes on a critical mental health implication.
- **Managing Memory and Historical Traps:** Failing to use historical context leads to frustrating loops—such as an airline chatbot repeatedly asking a customer to fill out the same baggage form an hour later because it forgot the previous interaction. Conversely, over-relying on memory causes broken UX; Goldfarb notes that ChatGPT frequently injects irrelevant company information or personal details into completely unrelated prompts, such as a request for a cooking recipe.
---
[10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) Expert-Calibrated Evals and the Pitfalls of Automated Grading
To verify whether these safety and reasoning tactics are actually working, robust evaluation is essential. Goldfarb strongly maintains that evaluating domain performance should **not** fall to software engineers, even if having developers write the tests is the easier default. Instead, evaluation must be led by domain specialists: financial analysts assessing financial models, public policy experts evaluating election advice, healthcare professionals judging clinical interactions, and workflow experts evaluating corporate operations. To achieve this, Forum AI built a platform split into two core offerings: a **Standardized Evaluation Platform** and a **Tailored Evaluation Platform**.
The Standardized Evaluation Platform offers pre-built, expert-trained evaluators designed to provide defensible guardrails for high-stakes enterprise applications. For example, former financial regulators trained evaluators to catch unauthorized financial advice, while Wall Street professionals created benchmarks to measure the accuracy of financial analysis. This external validation is critical; as Forum AI Co-Founder Campbell Brown wrote in a *Wall Street Journal* op-ed titled *"AI Needs Public Quality Testing,"* companies face massive operational and reputational risks when they "grade their own homework." Furthermore, as Microsoft CEO Satya Nadella recently noted, *"Private evals may be the biggest IP a company owns,"* highlighting how custom evaluation datasets allow organizations to embed their proprietary institutional knowledge into AI systems.
- **Evaluating with Industry Leaders:** Forum AI spent hundreds of hours with domain authorities—including CNN host Fareed Zakaria—to translate human editorial judgment into algorithmic evaluations for factual accuracy in political and geopolitical discourse.
- **The Failure of Off-the-Shelf LLM Judges:** Forum AI ran an experiment comparing popular open-source automated LLM judges against real domain experts reviewing the exact same AI outputs. The automated judges scored outcomes as "good" or "bad" with virtually **zero statistical correlation** to the judgments of human experts.
- **Capturing Signal over Noise:** Uncalibrated automated governance and monitoring tools often create the illusion of safety while merely capturing background noise. Calibrating evaluators directly against expert judgment significantly improves chance-adjusted accuracy and reliably catches high-risk edge cases.
- **Model Brittleness and Ongoing Monitoring:** Frontier models are notoriously fragile. Goldfarb highlights a recent case where minor harness-level adjustments to GPT-56 drastically altered its benchmark performance. Because small systemic tweaks or underlying model updates can lead to severe performance regressions, evaluations cannot be a one-time setup—they require continuous real-time oversight.
- **Availability:** Forum AI's Standardized Evaluation Platform is generally available to the public upon request via their website, while additional spots for the Tailored Evaluation Platform were opened specifically for AI4 conference attendees.
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) how do we build AI that is genuinely good, behaves reliably at scale, performs consistently under pressure, and stays safely within its guardrails? — Robbie Goldfarb
> [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) Well, it depends. It's complicated. — Domain experts
> [04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) We can't build true agentic systems without the ability to predict the consequences of actions just like humans do — Yann LeCun
> [04:37](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=277s) I don't want to be here anymore
> [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) AI Needs Public Quality Testing — Campbell Brown
> [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) grade their own homework
> [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) Private evals may be the biggest IP a company owns — Satya Nadella
## Mentioned
Robbie Goldfarb, Forum AI, Facebook, Cambridge Analytica, Instagram, Meta, Google Gemini, OpenAI, OpenAI Model Spec, Anthropic, Claude, Yann LeCun, AI4, ChatGPT, Campbell Brown, The Wall Street Journal, Satya Nadella, Microsoft, CNN, Fareed Zakaria, GPT-56
## Video description
Robbie Goldfarb - Co-Founder & CTO, Forum AI
I’ve worked on misinformation at Facebook, youth safety at Instagram, and high-stakes reliability with most foundation model labs. That experience has made clear that more capable AI will not automatically become more trustworthy. In this conversation, I’ll explore what it takes for AI to earn trust, including how systems can exercise judgment, account for consequences, and operate within a defensible range of reasonable action. I’ll also examine how humans can be effectively involved and how expert judgment can be translated into evaluations and safeguards at scale.
Forum AI CTO称自动化LLM评判器未达专家安全基准
静态规则手册和现成的自动化AI评估器无法可靠评判复杂的真实场景输出，需引入结果驱动型推理与人类专家校准以防止系统失效。
在本次演讲中，Robbie Goldfarb阐释了静态书面规则手册为何无法应对真实世界AI应用中微妙且高度依赖上下文的需求。他主张采取两大主要策略：使AI具备结果驱动型思考能力，以及在不破坏用户体验的前提下随时间推移适应上下文记忆。此外，Goldfarb强调AI评估必须由领域专家而非软件工程师进行校准，并指出开箱即用的自动化LLM评判器与人类专家的判断几乎呈现零相关。
静态书面规则在复杂的真实 AI 部署场景中往往会失效或产生直接矛盾，例如 Claude 的不隐瞒原则与进食障碍治疗的临床实践发生冲突。 (0:04)
为应对细微情境，Forum AI 提倡后果驱动思维（consequence-driven thinking）与后果映射（consequence mapping），在响应前评估潜在多维影响和波及群体。 (4:37)
安全护栏与评估必须结合长期历史上下文，而非依赖单轮判断；同时需平衡记忆机制，避免陷入死循环或错误注入无关个人隐私。 (4:37)
AI 评估必须由医疗、金融或政策等行业专家主导而非软件工程师；开箱即用的自动化 LLM 评估器在判定结果时与领域专家的真实判断几乎呈零统计相关性。 (10:24)
Reading summary
Robbie Goldfarb, Co-Founder and CTO of Forum AI, argues that static rulebooks are insufficient for governing AI in high-stakes environments, requiring systems to incorporate consequence-driven reasoning, multi-turn context, and continuous evaluations calibrated by human domain experts.
The Breakdown of Static AI Rulebooks (0:04)
Frontier AI developers and enterprises typically rely on codified rules, such as OpenAI's Model Spec, Anthropic's Claude Constitution, or Google Gemini's guidelines. However, Forum AI's consultations with subject matter experts across politics, finance, and medicine show that real-world deployment is dominated by contextual nuances where rigid rules contradict each other. For example, Claude's constitution prohibits taking covert actions behind a user's back, yet clinical experts on eating disorders advise covertly redirecting patients who engage in obsessive calorie counting rather than confronting them directly.
Consequence Mapping and Multi-Turn Context (4:37)
To navigate ambiguity, Goldfarb points to Yann LeCun's argument that agentic systems require the ability to predict the outcomes of their actions. Forum AI implements consequence mapping, which replaces simple binary ratings of 'good' or 'bad' with multidimensional forecasts of downstream impact. In a simulated healthcare scenario, a chatbot's polite, validating response provided brief emotional comfort to a patient but failed to address underlying hopelessness, ultimately escalating the caregiving burden on the family. Similarly, an airline bot resolving an issue with polite tone temporarily lowered friction, but failing to fix the root problem increased repeat contact volume and prolonged human agent resolution times.
Effective reasoning also demands adapting to information across time rather than evaluating single interactions in isolation. A phrase like 'I don't want to be here anymore' may represent mere boredom at a conference, but the same statement indicates acute mental health risk if preceded by a week of discussions about depression. Goldfarb notes that system designers must balance historical context carefully: failing to track history leads to repetitive tasks like re-submitting lost baggage forms, while over-relying on memory causes systems like ChatGPT to inject irrelevant personal or company data into unrelated prompts.
Pitfalls of Automated LLM Evaluation (10:24)
Goldfarb asserts that software engineers should not serve as the primary evaluators of specialized AI systems; assessments must instead be designed by domain professionals such as financial analysts, healthcare providers, and policy experts. Addressing risks highlighted in a Wall Street Journal op-ed by Forum AI Co-Founder Campbell Brown, Goldfarb warns against companies 'grading their own homework.' In an internal experiment, Forum AI found that popular off-the-shelf, open-source automated LLM judges showed virtually zero statistical correlation with actual domain experts when scoring identical outputs as 'good' or 'bad.' Uncalibrated automated governance risks generating a false sense of security by measuring background noise instead of actionable safety signals.
Continuous Monitoring Against Model Brittleness (10:24)
Evaluation must operate as an ongoing process rather than a static pre-deployment check because frontier models exhibit significant brittleness. Goldfarb notes that minor harness-level adjustments to GPT-56 caused severe fluctuations in benchmark scores. To maintain defensible guardrails, Forum AI deployed a Standardized Evaluation Platform built with former regulators and analysts for common compliance needs, alongside a Tailored Evaluation Platform designed to capture proprietary enterprise workflows.
While consequence mapping and expert-grounded benchmarks provide structured paths beyond rigid rulebooks, the inherent fragility of frontier models means organizations must maintain continuous, real-time oversight rather than relying on one-time testing setups.
精炼阅读摘要
Forum AI 联合创始人兼首席技术官 Robbie Goldfarb 指出，传统的静态规则与未校准的自动化大模型评分无法应对真实世界的复杂性；唯有引入预测下游影响的“后果映射”、动态上下文感知以及由行业专家校准的持续评估，才能确保高风险场景下的 AI 具备可靠与安全性。
静态规则的局限与多维价值冲突 (0:04)
人类社会偏好非黑即白的明文条文，前沿实验室与企业也普遍尝试通过书面准则来规范模型行为，例如 Google Gemini 的政策规范、OpenAI 的 Model Spec 以及 Anthropic 为 Claude 制定的宪法。然而，顶尖政治、金融和医疗专家普遍认为真实场景“高度复杂且因境而异”，静态规则在现实中极易失效或产生逻辑冲突。例如，Claude 的宪法规定模型不得暗中操作，但在 Forum AI 针对进食障碍的心理健康评估中，临床医生明确指出，当患者试图进行强迫性计算卡路里等不健康行为时，专业做法应当是暗中将对话引开，而非直接惊动患者。
此外，静态框架还面临深层的根本性矛盾：一方面要求尊重用户自主决策权，另一方面要求将用户的长期福祉置于即时要求之上；既要保护个人隐私免受监控，又须配合合法监管与司法调取；在遵守数据最小化原则的同时，企业又被要求留存详尽的长期审计日志。Goldfarb 指出，仅靠基础规则手册无法为高风险系统提供可靠的安全保障。
后果映射：从二元判断转向影响推演 (4:37)
针对规则失效的问题，Forum AI 主张引入后果驱动思维（Consequence-driven thinking）。正如 AI 学者 Yann LeCun 所言，若无法像人类一样预测行动后果，就无法构建真正的智能体系统。人类专业人士在面对模糊情境时，不会机械地对照权限清单，而是权衡不同行为的波及范围与受影响对象。为此，Forum AI 提出了“后果映射”（Consequence Mapping）技术，放弃简单的“好与坏”二元标注，系统化梳理模型输出在多个维度上的下游衍生后果，以此作为模型训练数据或实时决策的上下文参考。
在心理健康模拟场景中，AI 礼貌且具共情色彩的回复虽然暂时缓解了患者的表面情绪，却掩盖了深层的绝望感与孤立状态，导致家属因未能及时察觉病情升级而承受更大的心理创伤与照护负担。同样在客户服务场景中，航司聊天机器人采用礼貌措辞应对未解决的问题，短期内降低了沟通摩擦，却导致用户后续反复联系，进而造成客服工单激增、人工坐席负担加重以及平均处理时长显著攀升。
时间维度上的上下文感知与记忆平衡 (4:37)
现有的评估与安全护栏大多基于单轮孤立交互，但时间上下文往往会彻底改变语句的语义。例如，在行业峰会上如果一位参会者对聊天机器人说“我不想再待在这里了”，单轮评估通常会判定其意图是演讲乏味、想要回酒店休息；但如果该用户在过去一周内持续向模型倾诉悲伤、抑郁与孤独，同一句话便具备了极高风险的心理健康危机含义。
利用历史上下文同样需要精细的工程平衡。完全忽略历史记忆会导致交互体验割裂，例如航司机器人遗忘一小时前的沟通，要求用户重复填写行李表格；然而过度依赖记忆则会破坏用户体验，例如 ChatGPT 常在无关的日常烹饪食谱中错误插入用户此前输入的商业背景或公司信息。系统必须具备准确提取关键信息并随时间自适应调控上下文的能力。
专家校准评估与自动化裁判的失效 (10:24)
Goldfarb 强调，高风险领域的 AI 表现绝不能由软件工程师独立评判，必须交由特定领域的资深专家主导，如金融分析师把关金融模型、公共政策专家评估选举内容、医护人员审核临床对话。Forum AI 联合创始人 Campbell Brown 在《华尔街日报》专栏文章《AI Needs Public Quality Testing》中指出，企业若“自己给自己的作业打分”，将面临巨大的运营与声誉风险；同时微软 CEO Satya Nadella 也曾表示，私有评估体系或许是企业所拥有的最大知识产权。
针对行业广泛采用的开源 LLM 作为自动化裁判（LLM judges）的做法，Forum AI 进行了对比实验。结果显示，未经校准的开源大模型裁判给出的“好/坏”评分，与真实行业专家的专业判断在统计学上几乎呈现“零相关性”，这表明廉价的自动化工具往往只捕获了背景噪声并造成虚假的安全感。此外，前沿基础模型表现出高度的脆弱性，例如对 GPT-56 测试环境的微小调整便会导致基准性能剧烈波动，这说明评估绝非一次性流程，必须依赖与专家判断高度对齐的持续监控体系。
Goldfarb 总结认为，随着前沿模型架构日益脆弱且充满不确定性，依赖内部工程测试或黑盒自动化裁判已无法满足真实商业环境的合规与风控诉求。企业构建差异化竞争壁垒的核心，正逐渐转向如何将顶级专家的隐性专业判断沉淀为动态、可验证的私有化评估基础设施。
Detailed notes — zh
## Background
该视频由频道 Ai4 于 2026 年 8 月 15 日在 YouTube 发布，主讲人为 Forum AI 联合创始人兼 CTO Robbie Goldfarb。视频简介指出，Robbie 结合其在 Meta 及前沿实验室应对虚假信息和高危安全场景的经验，深入探讨了仅凭模型能力的提升无法自动换来信任，并分享了如何让 AI 具备后果推理能力、引入专家判断以及在大规模部署中建立可靠评估与安全机制。
## TL;DR
Robbie Goldfarb 探讨了在现实复杂环境中单纯依赖固定规则治理 AI 所面临的崩溃困境，指出抽象原则在真实决策场景中往往会相互冲突。为解决这一问题，Forum AI 提出了两大核心策略：通过“后果映射”培养系统的后果驱动思维，以及利用长上下文推理适应时间维度的动态信息。此外，他强调评测职责必须交由垂直领域的专业人士而非纯技术人员，并揭示了通用大模型裁判（LLM Judges）与专家判断缺乏统计相关性且模型自身极具脆弱性的行业痛点，主张构建由专家校准的持续评估体系。
## Atmosphere & style
演讲整体呈现出一种沉稳务实且充满一线实战洞见的基调。主讲人以严谨从容的工程师兼创业者姿态娓娓道来，从亲历的重大公共事件切入，语气客观而冷静，直击当前大模型工业界盲目迷信死板规则与自动化评测的浮躁乱象。整场分享节奏紧凑、逻辑严密，既有对系统脆弱性的审慎警示，又给出了具实操性的工程框架，给听众带来强烈而深刻的认知震撼与启发。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) AI 能力的提升并不等同于可信度的自动增长，在医疗、金融和选举等复杂高风险场景中，界定并实现 AI 行为的“优秀与向善”极其依赖动态上下文。
- [03:07](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=187s) 人类倾向于用死板的规则与政策指南（如 Model Spec、宪法 AI）约束系统，但在真实业务场景中，各项合理原则之间经常发生直接冲突（如用户自主权与长远健康的矛盾）。
- [03:55](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=235s) 仅靠静态规则不足以指导高难度决策，弥补规则缺陷的两大核心策略为“后果驱动思维”与“随时间适应关键信息”。
- [05:06](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=306s) 后果映射（Consequence Mapping）技术通过梳理行为在情绪、运营等多维度的连锁反应与潜在影响，帮助模型摆脱死板标签并进行类似人类专家的假设推理。
- [07:35](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=455s) 当前 AI 评估常将交互割裂为单一瞬间，而真实决策高度依赖历史背景，必须引入长上下文推理评估（Long Context Reasoning Evals）来测定系统的时序适应能力。
- [08:50](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=530s) 历史上下文并非越多越好，过度依赖记忆或强行代入不相关背景同样会导致系统输出质量下降，需平衡实时过程学习与持续学习能力。
- [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) AI 评估职责不应由软件工程师独揽，必须由金融、法律、医疗等垂直领域的专家来评判与定夺具体输出的质量。
- [12:00](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=720s) 企业“自己给自己批改作业”存在合规与业务风险，将企业内部专属经验转化为定制化评估基准（Private Evals）已成为企业核心的知识产权与防御性壁垒。
- [14:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=844s) 开箱即用的通重大模型裁判（LLM Judges）与人类专家的判断往往缺乏统计学相关性，未加校准的自动化监控可能只捕捉到了无效噪音。
- [15:10](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=910s) 大语言模型具有高度脆弱性，测试框架或提示词的微小变动都可能引起性能剧烈波动，因此评估必须作为端到端持续监控机制贯穿系统全生命周期。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) 复杂的现实世界与“AI 如何向善”的困境
主讲人 Robbie Goldfarb 回顾了自己的职业生涯，他长期处于 AI 与各类高度复杂、敏感议题的交汇点。他曾在 Facebook 经历剑桥分析（Cambridge Analytica）丑闻发酵期、COVID-19 疫情以及 2020 年美国大选期间负责新闻与虚假信息治理；随后在 Instagram 从事了两年青少年安全工作，应对青少年心理健康、网络成瘾和霸凌骚扰等问题；之后又在 Meta 的超级智能实验室专注于 AI 安全。如今作为 Forum AI 的联合创始人兼 CTO，他与大多数顶尖的前沿模型实验室、财富 500 强企业及政府机构合作，解决涉及选举、临床医疗安全和财务建议等复杂高危场景。
尽管面对的领域各不相同且极其繁杂，但核心问题却非常纯粹：“我们如何构建出真正‘向善且优秀’（Good）的 AI？”对当下的企业和前沿实验室而言，最焦虑的问题无非是：大规模部署时如何确保 AI 行为可靠？在高风险场景下如何保证表现一致？以及如何确保系统不会挣脱护栏导致灾难性后果？
为了寻找答案，Forum AI 团队与前国家元首、畅销书作家、思想领袖以及顶级学者等各领域的顶尖专家展开了深度探讨，试图理清“优秀”在各领域的具体定义（如什么是优秀的财务分析、健康的政治辩论或合格的临床处置）。然而，专家们给出的回答几乎如出一辙：“这取决于具体情境，现实太复杂了。”
- **真实世界的特质**：极其依赖上下文和细微差别，正确决策受大量动态变量影响。
- **人类对规则的天然偏好**：人类本能地渴望非黑即白的清晰边界，因此创造了法律、宗教典籍、宪法乃至机场书店里的行为指南；AI 领域同样沿袭了这种“立规矩”的思路（如 Gemini 的政策指南、OpenAI 的 Model Spec、Anthropic 为 Claude 制定的宪法，以及各企业的治理框架）。
---
### [03:07](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=187s) 当死板规则走向崩溃：真实场景中的伦理冲突
这些精心制定的规则虽然初衷良好且构成了负责任技术的基石，但在复杂的真实世界中极易失效甚至相互冲突。
```mermaid
flowchart TD
accTitle: 规则冲突示例图
subgraph Conflict1["案例一：隐蔽干预与用户自主"]
A1["原则：严禁在用户背后隐蔽行事"] <--> B1["临床实践：引导进食障碍患者时需隐蔽转移卡路里话题"]
end
subgraph Conflict2["案例二：即时意愿与长远福祉"]
A2["原则：尊重用户自主决定权"] <--> B2["原则：优先保障用户长远健康"]
end
subgraph Conflict3["案例三：数据治理冲突"]
A3["原则：仅收集保留最低限度数据"] <--> B3["原则：保留详尽记录以供审计"]
end
```
这些规则单独看都非常合理，但在具体业务执行中却经常“打架”，仅凭抽象规则根本不足以指导高难度决策。为了弥补这一短板，团队总结出了两条核心策略：
1. **后果驱动思维（Consequence-Driven Thinking）**
2. **随时间适应关键信息（Adapting to Pertinent Information Over Time）**
系统若能稳定践行这两项策略，其输出和行为就会更贴近真实领域专家的判断。
---
### [05:06](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=306s) 策略一：后果驱动思维与“后果映射”
正如 Yann LeCun 所言：“如果不具备像人类一样预测行动后果的能力，我们就无法构建出真正的智能体系统。”人类在做决策时，首要考虑的往往不是“这是否符合某条死板规定”，而是“如果我这么做，会发生什么？谁会受到影响？会产生怎样的连锁反应？”具备丰富行业经验的专家极其擅长预判各种假设情境下的潜在结局。
为了将这种推理能力产品化，Forum AI 提出了**后果映射（Consequence Mapping）**技术。该方法不预先给行为贴上“好”或“坏”的标签，而是细致梳理真实场景中某种行为所带来的多维度后果。
| 场景类型 | 采取的行为 | 产生的具体后果维度 | 真实影响展开 |
|:--- |:--- |:--- |:--- |
| **心理健康临床干预** | 临床医生对患者的具体沟通方式 | 患者情绪反馈 | 带来了短暂的心理释怀与被认同感，但长期的绝望感和孤独感仍未得到根本解决。 |
| | | 家属反应与负担 | 家属在症状恶化前完全不知情，导致后续照料负担与心理压力骤增。 |
| **客服聊天机器人** | 客服机器人对用户投诉的回复话术 | 客户体验层面 | 礼貌的措辞缓解了即时冲突，但由于核心问题未解决，导致客户更加不满且感到不便。 |
| | | 运营指标层面 | 导致用户重复进线，直接推高了工单总量、人工客服工作量及平均解决时长。 |
后果映射图谱既可以用作训练模型推理能力的语料，也可以作为上下文工具供模型在处理复杂模糊案例时实时检索参考。
---
### [07:35](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=455s) 策略二：时序上下文适应与长上下文推理评估
当前业界对 AI 的评估和治理框架普遍存在一个致命缺陷：将交互割裂为**孤立真空中的单一瞬间**。然而，真实世界中某个时刻的最佳行动，往往取决于此前发生的一连串背景事件。
- **多义词句的语境依赖**：如果在 AI 4 大会现场有人对机器人说“我不想待在这里了”，最直观的理解可能是演讲太无聊想回酒店休息；但如果该用户在过去一周里持续倾诉绝望、悲伤与孤独，这句话的含义就必须被重新审视。
- **客服记忆缺失的反面案例**：向航空公司客服机器人询问行李丢失进度，机器人发来一张表格；一小时后用户再次催问，系统由于缺乏历史记忆，依然机械地回复“请填写这张表格”，造成极差的体验。
针对这一问题，Forum AI 开发了**长上下文推理评估（Long Context Reasoning Evals）**，专门测定 AI 系统识别和调用历史背景信息的能力。
值得注意的是，历史上下文并非越多越好，“过度依赖记忆”同样会导致糟糕输出。例如 ChatGPT 等对话模型有时会出现过度记忆的问题——用户只是询问一份菜谱，它却突然强行代入用户公司的背景信息，造成无关干扰。此外，业界在实时过程学习（Real-time Procedural Learning）与持续学习（Continual Learning）等方向上的演进，也为长时序动态适应提供了更多技术支撑。
---
### [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) 评估范式转变：专家评测与企业专属 IP
要验证上述策略是否真正生效，必须依赖科学的评估体系。Robbie 强调，**评估 AI 表现的职责绝不应由技术人员/工程师独揽**。虽然由开发者直接测试最省事，但真正的评判权必须交给垂直领域的专家：
- 财务分析应由金融专家评估；
- 选举与政策建议应由公共政策学者评估；
- 临床行为应由医疗从业者定夺；
- 企业内部工作流应由熟悉该业务流程的业务专家审核。
```mermaid
flowchart LR
accTitle: Forum AI 评估体系双轨架构
subgraph Platform["Forum AI 评估体系架构"]
direction TB
subgraph Std["标准化评估平台 (Standardized)"]
S1["前监管人员训练：防范未授权财务建议"]
S2["华尔街专家构建：财务分析与预测评估"]
S3["资深媒体人构建：政治与地缘事实准确性 (如 Fareed Zakaria)"]
end
subgraph Tailored["定制化评估平台 (Tailored)"]
T1["企业内部专家经验资产化"]
T2["将私有知识转化为专属评估基准"]
T3["构建防御性壁垒与业务差异化"]
end
end
```
针对行业痛点，Forum AI 联合创始人 Campbell 曾在《华尔街日报》发表专栏文章《AI 需要公开的质量检验》（*AI needs public quality testing*），指出企业“自己给自己批改作业”存在巨大的合规与业务风险，必须引入具备外部可信度的评测。与此同时，微软 CEO 萨提亚·纳德拉（Satya Nadella）曾指出：“**私有评估基准（Private Evals）可能是企业拥有的最大知识产权（IP）。**”将企业特有的知识经验转化为定制化评估基准，是打造差异化竞争力的关键。
---
### [14:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=844s) 破除“大模型裁判”幻觉与应对模型脆弱性
在自研评估系统的过程中，团队发现了当前自动化评估工具的一个严峻事实：**开箱即用的大模型裁判（LLM Judges）往往与人类专家的判断缺乏统计学相关性**。
Forum AI 曾进行过对照实验：使用开源及市面上通用的 LLM 自动化评测工具对同一批 AI 输出打分（判断 Good/Bad），同时邀请资深领域专家进行独立评分。结果显示，自动化裁判给出的结论与专家意见几乎没有相关性。这意味着许多看似神奇的自动化监控与治理平台，实际捕捉到的可能只是无效噪音。而经过领域专家校准的裁判系统，能大幅提高几率校正准确率（Chance-Adjusted Accuracy），从而精准拦截高危故障。
```mermaid
flowchart TD
accTitle: 大模型脆弱性与持续评估必要性
A["微小 Harness 层改动 / 模型权重迭代"] --> B["基准性能剧烈波动 (暴涨或严重退化)"]
B --> C["必须摆脱单次评估，实施端到端持续监控"]
```
此外，大语言模型具有高度的**脆弱性（Brittleness）**。例如近期业内对 GPT-56（或前沿模型）的测试中，仅在 Harness 测试框架层面对上下文或提示词做细微调整，就能导致其在前沿基准上的跑分产生巨大波动，甚至出现性能大幅倒退。因此，系统评估绝不能是一次性的出厂测试，而必须是在模型更迭与上下文变化全周期中持续运行的监控机制。
Robbie 在总结中重申，构建可靠 AI 不仅需要打破死板规则、引入后果推理与长时序适应，更需要依赖由领域专家校准的高质量评估体系。目前 Forum AI 的标准化评估平台已全面开放申请，定制化评估平台也面向参会者开放了试用名额。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) 我们如何构建出真正‘向善且优秀’（Good）的 AI？ — Robbie Goldfarb
> [00:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=4s) 这取决于具体情境，现实太复杂了。 — 各领域顶尖专家
> [05:06](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=306s) 如果不具备像人类一样预测行动后果的能力，我们就无法构建出真正的智能体系统。 — Yann LeCun
> [05:06](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=306s) 如果我这么做，会发生什么？谁会受到影响？会产生怎样的连锁反应？ — Robbie Goldfarb
> [07:35](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=455s) 我不想待在这里了 — 案例用户
> [07:35](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=455s) 请填写这张表格 — 航空公司客服机器人
> [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) 评估 AI 表现的职责绝不应由技术人员/工程师独揽 — Robbie Goldfarb
> [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) AI 需要公开的质量检验 — Campbell
> [10:24](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=624s) 私有评估基准（Private Evals）可能是企业拥有的最大知识产权（IP）。 — 萨提亚·纳德拉 (Satya Nadella)
> [14:04](https://www.youtube.com/watch?v=sNWVNVeJwLc&t=844s) 开箱即用的大模型裁判（LLM Judges）往往与人类专家的判断缺乏统计学相关性 — Robbie Goldfarb
## Mentioned
Robbie Goldfarb, Forum AI, Facebook, Instagram, Meta, Gemini, OpenAI, Model Spec, Anthropic, Claude, Yann LeCun, ChatGPT, 萨提亚·纳德拉 (Satya Nadella), Campbell, Fareed Zakaria, 华尔街日报 (The Wall Street Journal), AI 4 大会, GPT-56
## Video description
Robbie Goldfarb - Co-Founder & CTO, Forum AI
I’ve worked on misinformation at Facebook, youth safety at Instagram, and high-stakes reliability with most foundation model labs. That experience has made clear that more capable AI will not automatically become more trustworthy. In this conversation, I’ll explore what it takes for AI to earn trust, including how systems can exercise judgment, account for consequences, and operate within a defensible range of reasonable action. I’ll also examine how humans can be effectively involved and how expert judgment can be translated into evaluations and safeguards at scale.
[Source](https://www.youtube.com/watch?v=sNWVNVeJwLc)
[0:04] Robbie Goldfarb: how do we build AI that is genuinely good, behaves reliably at scale, performs consistently under pressure, and stays safely within its guardrails?
[0:04] Domain experts: Well, it depends. It's complicated.
[4:37] Yann LeCun: We can't build true agentic systems without the ability to predict the consequences of actions just like humans do
[4:37] I don't want to be here anymore
[10:24] Campbell Brown: AI Needs Public Quality Testing
[10:24] grade their own homework
[10:24] Satya Nadella: Private evals may be the biggest IP a company owns
Original transcript
[0:04] I've spent most of my career at the intersection
of AI some pretty complicated issues.
[0:11] I worked at Facebook on news
and misinformation during the
[0:14] fallout of the Cambridge Analytica scandal.
During COVID, during the US 2020 elections.
[0:20] I worked at Instagram on youth safety for 2
years covering everything from teen mental health
[0:25] to addiction and harassment. I worked on AI
safety at Meta's super intelligence labs and
[0:30] in my current capacity as co-founder of Forum
AI, I've worked with most major frontier labs,
[0:35] Fortune 500 companies, and government on complex
use cases including elections, clinical safety,
[0:43] and financial advice. Now, across all these
pretty complicated areas, the question we're
[0:49] trying to answer is actually usually pretty
simple. How do we make AI that is good?
[0:58] When you think about the questions that are
plaguing enterprise, that are plaguing the labs,
[1:01] that are plaguing the industry as a whole,
it often comes down to this idea, right?
[1:07] How can we be confident that our AI will behave
reliably when deployed at scale? How can we be
[1:12] confident that our AI will perform consistently
in high-stakes use cases? How can we be confident
[1:18] our AI won't break free of guardrails and kill
us all? These are the questions we're asking.
[1:25] Now, in my work at Forum, I've had the privilege
of working with a pretty remarkable group of
[1:30] domain experts at the top of their field. We work
with former presidents, best-selling authors,
[1:35] thought leaders, top academics, and we
press really hard to understand what is
[1:39] good look like in their respective domains.
What does good financial analysis look like?
[1:46] What does good political discourse look
like? What does good behavior look like
[1:50] in a clinical setting? What does good look like in
the context of a particular type of organization?
[1:57] Unfortunately, the answer is always some version
of well, it depends. It's complicated. The real
[2:04] world is really complicated. It's incredibly
contextual and nuanced and the right thing to do
[2:09] often depends on a large number of different
factors. Now for us as humans, that sucks.
[2:15] Because we like it when things are black
and white. We like it when there's rules.
[2:19] It's why we have legal texts, we have religious
texts, we have constitutions. You can have books
[2:26] that you can buy at the airport and say this is
how to be good. And now with AI, we're we're doing
[2:33] a similar thing. On the one hand, you have the
labs themselves defining their own rule books,
[2:38] right? You have Gemini's policy guidelines,
opening eyes model spec, Anthropic as they do is
[2:45] take a little bit of a different approach, but
still similar with Claude's written constitution.
[2:51] In an enterprise setting, companies are
defining their own policies and governance
[2:55] frameworks. All ways to define what is good.
What do we How do we want these systems to
[2:59] behave? To guide how we train, implement, and
instruct these systems. Now the problem is,
[3:07] like we were saying before, the real world's
complicated. And these rules tend to break down.
[3:12] I can give you some examples. There's a
a line in Claude's constitution that says
[3:17] you shouldn't do anything covertly behind a
user's back. It's a pretty reasonable rule.
[3:24] Now we were doing some work with a customer
building evaluations for AI providing mental
[3:30] health advice. Focus particularly on eating
disorders. And one of the things we learned
[3:35] from clinicians is that when a user is
trying to engage in unhealthy discussions,
[3:38] so for example, calorie counting, the
proper indicated behavior actually is
[3:43] to covertly divert the conversation to a
healthier one without tipping them off.
[3:48] That's one example. This This is also
from Claude's constitution. It says to
[3:52] respect the user's right to make
decisions about their own life, but also
[3:56] consider the user's long-term well-being rather
than merely satisfying their immediate request.
[4:03] Protect individual privacy and freedom from
surveillance, but also respect legitimate legal
[4:08] authority. And this one we was from a enterprise
policy framework we were looking at. And it says
[4:16] on the one hand collect and retain only the
minimum data necessary, but also maintain
[4:20] detailed records so AI decisions can be audited.
So you can see how these might not always jive.
[4:27] Same time though, all these rules are good.
They all make sense and they're incredibly
[4:31] incredibly important foundation to building
technology responsibly, but they're not enough.
[4:37] So what do we do? We've spent a lot of time
thinking about this across a wide variety
[4:41] of domains and continue to come back to these
two tactics that seem to be the missing pieces.
[4:46] The AI needs to be able to address cons-
sorry, the AI should be able to ex- to
[4:51] um execute consequence-driven thinking. And the AI
should be able to adapt to pertinent information
[4:56] over time. What we found is if a system is able to
consistently adhere to these tactics, then it will
[5:02] more consistently behave in line with what real
domain experts would define as good performance
[5:06] or good behavior. Let's talk about them. The
AI should exercise consequence-driven thinking.
[5:13] This is a quote from Yann LeCun. He says, cannot
we can't build the we can't build true agentic
[5:17] systems without the ability to predict the
consequences of actions just like humans do.
[5:21] And this is true. We've seen it in our research,
but it's also intuitive. When we as humans
[5:24] think about what to do, we don't think is this
permitted according to the rules, we think what
[5:29] will happen if I do this. What will be affected?
Who will be affected? How will they be affected?
[5:37] And experts, people who have been around the
block in a given area are actually really good
[5:41] at predicting the likely outcomes of those
theoretical scenarios. And so going back
[5:46] to the examples of the conflicting rules in
the rule books, you can see how being able to
[5:49] identify different possible outcomes and predict
what those look like can be a really powerful
[5:54] framework for making decisions in complex or
ambiguous cases. Now one technique we've developed
[6:02] for putting this into practice is something
called consequence mapping. The idea here is
[6:06] pretty simple. We take a real-world scenario
that an AI could conceivably find itself in.
[6:12] And then we outline what were what were the we
don't say is it good or bad. We say what were
[6:16] the outcomes of the behavior that you that that
that took place. And so in this case, this is
[6:22] um an engagement between a clinician and
a patient in a mental health setting.
[6:27] And what we've done is we said, "Okay, based on
what that clinician said and the way they behaved,
[6:31] what happened?" So what were the resulting
emotions for the patient? Well, they had
[6:35] brief relief and validation, but still there was
ongoing hopelessness and isolation that remained
[6:39] largely unaddressed. What were, for example, the
the reactions of the family? The family members
[6:44] remained unaware until the symptoms worsened and
increased their distress and caregiving burden.
[6:49] This is uh an example with uh a customer service
chatbot. So we took same idea, an engagement
[6:57] between a customer service chatbot and a user,
and said, "Okay, based on what that chatbot
[6:59] decided to say, what were the outcomes?" Well,
on the customer service, the polite tone reduced
[7:04] the immediate friction, but the unresolved problem
left the customer dissatisfied and inconvenienced.
[7:10] What was the operational impact? Well, the the
repeat contacts increased the ticket volume,
[7:14] the human agent workflow, and the
average resolution time. And so
[7:19] consequence maps like this can be a powerful
tool for training the reasoning of an AI, but
[7:24] also more simply just as a tool that's available
in context for it to reference. In either case,
[7:29] can be a powerful tool for making decisions
in those sorts of complex and ambiguous cases.
[7:35] Now the second tactic the AI should be able
to adapt to pertinent information over time.
[7:41] So much of how we think about AI is in the context
of these isolated and moments in a vacuum. Even if
[7:47] you go back to the rule books from earlier and you
read the policies and the examples that are given,
[7:52] they're mostly in the context of a single
moment. Even most evals out there today
[7:57] in governance frameworks define success in the
context of an individual moment. But of course,
[8:03] in the real world, that's not the way things
happen. The right thing to do in a given moment
[8:06] often depends largely on all the things that
happened leading up to that moment. And I'll give
[8:12] you a couple examples here. If we take this room
right here at AI 4, let's say someone pulls out
[8:16] their phone and they go to a chatbot and they say,
you know what, I don't want to be here anymore.
[8:20] The obvious interpretation of that is maybe
my talk is boring and you want to leave and
[8:26] go back to your hotel room. [clears throat] But if
that same person spent the week until now talking
[8:31] about hopelessness and sadness and depression and
loneliness, the way you might interpret I don't
[8:35] want to be here anymore could be different.
Right? Maybe to give a more benign example,
[8:42] a customer service chatbot, um I think we can all
relate to this sort of situation. Let's say you're
[8:47] talking to an airline's customer service chatbot,
your bags still haven't arrived, you say, where
[8:51] are my bags? It says here, why don't you fill
out this form? So you go, you fill out the form,
[8:56] an hour later your bags still haven't arrived,
you go back, you say, what's happening? Where are
[9:00] my bags? And of course, it says, hey, why don't
you go fill out this form? It doesn't remember
[9:05] a prior interaction. So what it speaks
to is the importance of being able to
[9:11] leverage historical context in making decisions
about what to do. And one of the tactics
[9:17] we use here, I'll talk in a second about our
evaluation platform, which is what we focus
[9:21] on, [clears throat] but one of the evals that we
develop are called long context reasoning evals.
[9:27] The idea behind this is simply to assess
how good an AI system is at identifying
[9:32] which historical context to use and how.
Cuz remember, the opposite can be true,
[9:36] too. If you over-rely on historical context, that
can create bad decisions and bad outputs as well.
[9:42] A classic example of this is ChatGPT or
really all of the chatbots suck at this,
[9:46] but I think especially ChatGPT over-relies
on memory. Right, you could be trying to ask
[9:51] it for you ask it for a recipe and it starts
like injecting information about your company
[9:55] and that creates just something that just doesn't
feel feel relevant or useful. And so that's why we
[10:01] built those evals, but beyond that there's a ton
of tools out there like graffiti, even just the
[10:07] out-of-the-box agent platform from like
Gemini OpenAI have good memory tooling
[10:11] and there's lots of really interesting research
happening here, real-time procedural learning,
[10:15] continual learning that I think we'll all be able
to benefit from in the near future hopefully.
[10:24] So, we have these tactics which on top of rules
give us the ability to build AI that performs the
[10:34] way we want it to, but the question still remains,
is it actually working? This is arguably the most
[10:38] important question because AI is really tricky
to get right. So you can try all these things,
[10:41] but you don't really know it's working you need
a way to do that. Now, I have quite a bit of
[10:46] conviction and certainly we do at Forum AI that
this should not be the role of technologists.
[10:52] Even though that's the easier default option,
right? Engineers are the ones building these
[10:55] systems, why not just also have them be the
ones that evaluate and assess its performance.
[11:00] Rather, it should be people with relevant
experience in the domain at hand making that call.
[11:06] Right? Should be experts in finance determining
if it's doing a good job at financial analysis. It
[11:11] should be experts in public policy determining if
it's doing a good job at giving elections advice.
[11:16] Healthcare professionals should decide if it's
doing a good job at behaving in a clinical
[11:21] setting. It should be experts in your company's
workflows determining if it's performing well
[11:25] in your company's workflows. And that's what
that's what we've built at Forum AI and there's
[11:31] really two parts to it. There's our standardized
evaluation platform and our tailored evaluation
[11:36] platform. Our standardized evaluation play
platform gives you access to our entire suite
[11:41] of expert trained evaluators. So, we have,
for example, uh evaluators that capture
[11:48] unauthorized financial advice trained by former
regulators. We have evaluators that look at the
[11:53] accuracy and the quality of financial analysis and
predictions built by Wall Street professionals.
[11:59] We have evaluators look at a variety of different
types of safety, quality, accuracy, and bias built
[12:05] by relevant domain professionals. And it gives you
the ability to build systems with confidence that
[12:11] actually work well and that have defensibility.
So, this is uh my my co-founder, Campbell, wrote
[12:19] this op-ed in the Wall Street Journal a couple
weeks ago called AI needs public quality testing.
[12:25] And what this speaks to is the risk,
not just to users, but to companies,
[12:31] of grading your own homework. And this talks about
the importance of being able to have external
[12:35] defensibility, especially if you're working
in high-stakes domains that may have risk. So,
[12:39] that's part of the the role of our standardized
evaluation platform. And standardized evaluations,
[12:46] while they're our foundational and important to
building good systems that work well and are a
[12:50] great shortcut to doing so, they're not enough.
And Satya Nadella had this quote um at a Microsoft
[12:56] event a few weeks ago where he says, "Private
evals may be the biggest IP a company owns."
[13:02] And what this speaks to is the power and the
importance of being able to capture your company's
[13:08] unique experiences, knowledge, expertise, and
evals, and the profound impact that can have
[13:14] on your ability to leverage AI in a way that
works well and that is uniquely differentiated.
[13:20] And so, what we've done is we took the system that
we built internally to build the with our experts,
[13:28] And we exposed that for companies to use to use
themselves. Now this is a screenshot. So early we
[13:33] probably spent now thousands of hours with experts
just trying to get in their brain and figure out
[13:38] how do you build the evals that really work and
reflect their judgement. This is an example with
[13:43] Fareed Zakaria from CNN is one of our experts.
We probably spent we spent many hours with him
[13:48] building evaluations around factual
accuracy on politics and geopolitics.
[13:55] But in all cases we should capture these
things and and build a system that allows
[13:59] you to build the evals that reflect the
people whose judgement you want to reflect.
[14:04] And part of the reason this is important is
that one of the things we learned early on
[14:10] building evals that capture what you want them to
capture is actually a lot easier said than done.
[14:16] It's really hard. And we ran this experiment
where we took a bunch of evals off the shelf
[14:23] like some open source evals some companies we were
working with and we we ran the evaluations on an
[14:29] AI system and so usually LLM judges said it's
good good bad output good output bad decision.
[14:36] Then we got relevant domain experts to come and
evaluate the exact same set of AI decisions or
[14:41] outputs. Good bad bad good. And what we found is
that in most cases there was actually very little
[14:47] statistical correlation between what the automated
judges produced and what the experts said.
[14:53] Now this may seem like technical jargon but the
implication is important and what it suggests is
[15:00] that automated monitoring tools governance tools
eval platforms may seem like magic but they may
[15:06] sometimes just be capturing noise. And you can see
here with our judges part of our what we do with
[15:11] our system is actually calibrated to the relevant
experts and you get certainly much higher chance
[15:16] adjusted accuracy as well as with that a higher
capture of failure or high-risk cases as well.
[15:25] The other other reason this is all
important is that LLMs are really brittle.
[15:29] This is a a recent example I saw a few days
ago um where GPT-56 solved someone made a few
[15:36] minor changes at the harness level and drastically
changed the performance on a frontier benchmark.
[15:42] Of course, the opposite can happen as well,
right? You can make a few small changes and it can
[15:46] drastically regress the performance of a
system. And so, what this speaks to is the
[15:51] importance to not just measure how good a system
is up front, but to continue doing it because as
[15:57] the models change as the context that's
available in your system changes,
[16:01] that can have a real impact on the performance.
[16:05] So, what do we talk about? Well, real
world's complicated and so sometimes
[16:11] black and white rules and instructions
aren't enough and our tactics like
[16:16] consequence-driven thinking, adapting information
over time that are important. But most of all,
[16:21] it's important that you have visibility and
oversight into how a system is performing
[16:26] so you can monitor and improve it over time and
get it to where it needs to be. And that's that's
[16:31] what we work on. I should um add one thing. Our
our standardized evaluation platform is available
[16:37] right now generally, so if you go to our website
and you want to try it out, just request access.
[16:42] The tailored evaluation platform is limited
access, but we did since there was quite a bit
[16:46] of interest yesterday opened up some more spots
um for AI 4. So, if anyone's interested in trying
[16:52] that out, we'd love to work together. You got my
information right there. Um thank you. [applause]
