## Background
Published on YouTube by the Ai4 channel on August 27, 2025, this talk features Rohit Patel, Director at Meta Superintelligence Labs. According to the page description, the session explores how modern LLMs work under the hood, examines real-world deployment challenges, and compares automated, judge-based, and human-rated evaluation methodologies to help teams quantitatively benchmark AI against human performance.
## TL;DR
Rohit Patel outlines the architectural frameworks and practical engineering methodologies required to rigorously evaluate generative AI systems in production. He emphasizes that because AI exhibits a non-convex intelligence envelope—where mastery in complex tasks does not guarantee competence in simpler adjacent tasks—teams must construct targeted evaluation suites rather than relying on generic benchmarks. The talk details the trade-offs between human raters, automated log-likelihood scoring, and rubric-based LLM judges, outlining a concrete strategy using 500-sample golden datasets to establish baseline quality before deployment. In the Q&A, Patel addresses complex frontiers including multi-turn evaluation drift, benchmark noise, and the deliberate overfitting of dedicated reward models to capture nuanced human preferences.
## Atmosphere & style
The presentation carries the brisk, pragmatic energy of an advanced technical masterclass, moving quickly past foundational tokenization concepts once Patel gauges the technical depth of the room. Patel delivers his insights with engaging clarity and engineering rigor, using sharp geometric analogies and transparent breakdowns of neural network mechanics. The concluding Q&A turns the session into an interactive, collaborative deep dive, tackling open frontier problems around multi-agent simulation and reward hacking with grounded realism.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=4s) Introduction and the Failure Modes of AI Deployment
Rohit Patel begins the session by setting the agenda around AI evaluations: what they are, how to construct them, and why organizations must care. To gauge the technical depth of the room, he asks for a quick show of hands: "how many people here have a sense of how tokenizers work?" Greeted by a room full of raised hands, he responds with enthusiasm—"Love it. Fantastic. Okay. So this is a great audience"—deciding to accelerate through the foundational mechanics of generative models so the session can dedicate maximum time to evaluation techniques before moving into audience Q&A.
Turning to the core theme of the presentation—addressing practical engineering and product challenges—Patel highlights that deploying AI into production surfaces multiple failure modes. These include integration failures, organizational friction, and feasibility problems where the underlying AI simply lacks the capability to perform the intended job (a pitfall he notes is far more common than people expect). The specific challenge this talk zeroes in on, however, is performance failure: situations where an AI system is successfully integrated and operational, yet fails to meet business expectations. The principal technical discipline required to diagnose and prevent this failure mode is rigorous evaluation.
- **AI Deployment Failure Types:**
- **Integration failure:** Technical issues wiring the model into existing engineering infrastructure.
- **Organizational failure:** Friction within the team or organization adopting the AI tooling.
- **Feasibility failure:** Expecting the model to solve a problem that lies entirely outside current technological capability.
- **Performance failure:** Deploying a system that technically functions but fails to satisfy quantitative quality standards.
---
[02:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=128s) The Non-Convex Intelligence Envelope
To justify why evaluations are indispensable, Patel introduces a geometric mental model: the "intelligence envelope" of artificial intelligence is fundamentally non-convex. When assessing a human colleague—such as someone holding a PhD in mathematics or physics—one can safely assume general competencies across adjacent domains. If that person demonstrates proficiency at task A and task B, common sense allows us to fill in the gaps and deduce that they can perform intermediate task C.
With modern large language and vision models, this heuristic completely collapses. A frontier model can achieve stunning, human-exceeding performance on highly complex reasoning tasks, only to fail catastrophically at elementary logic or basic comprehension. Because the envelope is not convex, taking two known points of capability does not guarantee competence on any convex combination between them. Consequently, teams cannot deduce upfront whether a model will succeed at task C simply because C seems easier than A and B. The only way to guarantee a product will work is to build an evaluation targeted precisely at the specific use case, rather than extrapolating from generic or adjacent benchmarks.
```mermaid
flowchart LR
accTitle: Convex Human Intelligence vs Non-Convex AI Intelligence
subgraph Human ["Human Intelligence (Convex)"]
direction TB
H_A["Mastery at Point A (e.g. Physics PhD)"] --- H_C["Interpolated Competence at Point C"]
H_C --- H_B["Mastery at Point B (Math Analysis)"]
end
subgraph AI ["AI Intelligence Envelope (Non-Convex)"]
direction TB
AI_A["Stunning Success at Task A"] -. "Unpredictable Failure".-> AI_C["Catastrophic Failure at Simple Task C"]
AI_C -. "No Linear Interpolation".-> AI_B["Stunning Success at Task B"]
end
```
---
[04:02](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=242s) Under the Hood: Turning Numbers into Language
Before deep-diving into evaluation architectures, Patel briefly revisits how neural networks generate text to inform subsequent evaluation design. Stripped of mystical framing, neural networks consist of stacked linear equations interwoven with non-linearities, executed at massive scale via matrix multiplications. Patel emphasizes an immutable rule: "Everything is numbers. These models are not capable of doing anything beyond producing numbers and ingesting numbers." Neural networks cannot read words or raw text; everything must be translated into numerical inputs, and the final output is purely a tensor of numbers.
The fundamental trick of generative AI lies in attaching human interpretations to those output numbers. Patel uses a toy binary classification example: suppose a network outputs two continuous values, representing a flower and a leaf. The network has no intrinsic understanding of botany; human designers simply establish a decision rule stating that whichever index has the larger numerical value represents the model's prediction. Expanding this logic produces a character-level generator: if the network outputs an array of 26 to 28 numbers, each index can be mapped to an alphabetical character, space, or exclamation mark. By taking the index with the maximum score, feeding that character back into the input, and repeating the cycle autoregressively, one arrives at a functioning character generation machine.
---
[07:11](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=431s) Modern Tokenization and Learned Embeddings
While a naive character model can ingest letters by arbitrarily assigning scalar numbers (such as setting $a = 1$, $b = 2$, and so on), this naive mapping proves inefficient for modern systems, even though training it on a sufficiently massive corpus will technically yield language generation. Modern large language models bypass character-level assignment in favor of tokenization. A tokenizer splits text into sub-word tokens drawn from a fixed vocabulary—typically around 50,000 unique units—where common words remain whole while rarer words are broken into sub-word fragments (for instance, sharing roots across singular and plural forms).
Instead of mapping these 50,000 vocabulary tokens to arbitrary static integers, modern architectures use learned embedding vectors. An embedding assigns a high-dimensional vector to each token and treats those values as trainable parameters within the network. Over the course of training on large corpora, the weights adjust automatically, positioning tokens in a geometric space that captures their contextual and semantic meaning. The model's final linear projection outputs a vector across the entire 50,000-token vocabulary, and the resulting distribution determines the next generated token.
---
[09:28](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=568s) Defining an Evaluation: A Monotone Mapping to the Real Line
Moving from generation to evaluation, Patel formulates what an evaluation actually is from a mathematical perspective. Whether one is ranking standalone foundation models or composite agentic systems for tasks like code generation or customer service, an evaluation is fundamentally "a mapping from the space of all possible models or systems to the real line." Critically, this function must be monotone with respect to capability: superior models must receive strictly higher scalar scores, while inferior models must receive lower scores.
Because the underlying space of neural networks is defined by billions of continuous weights across varied architectures, it possesses no natural ordering. One cannot inspect model weights directly to determine superiority. Instead, engineers must evaluate systems behaviorally by measuring output responses against an input test distribution. Constructing this mapping requires decomposing the problem into three foundational building blocks.
```mermaid
flowchart TD
accTitle: The Three Pillars of AI Evaluation
A["1. Curated Sample of Prompts\n(Representative subset of domain inputs)"] --> B["Model Generation Engine\n(Produce behavioral responses across systems)"]
B --> C["2. Scoring & Ranking Methodology\n(Human rating, pairwise comparison, or heuristics)"]
C --> D["3. Metric Aggregation\n(Reduction into a scalar score on the Real Line)"]
```
- **The Three Pillars of an Eval:**
- **1. A Representative Sample of Prompts:** Since collecting every possible user query is impossible, engineers must curate a high-signal, bounded sample that mirrors the operational use case.
- **2. A Scoring or Ranking Methodology:** A consistent decision rule to rate individual model outputs or rank competing outputs pairwise.
- **3. Metric Aggregation:** A mathematical formula that condenses individual evaluations into a reliable scalar metric representing overall system quality.
---
[13:58](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=838s) Evaluation Methodologies: Human Raters vs. Automated Metrics
Patel outlines the primary architectural paradigms for scoring model outputs, beginning with human-rated evaluations. Human evals are conceptually simple to design: once a test set of prompts is assembled, human reviewers inspect the outputs. Reviewers can score responses on an absolute scale, execute pairwise head-to-head comparisons between Model A and Model B, or benchmark model outputs against human-generated gold standards. A standard output metric derived from pairwise comparison is the "net win rate," which quantifies how frequently an AI system outperforms its competitor or human baseline.
Despite their qualitative fidelity, human evaluations carry severe operational drawbacks. They are slow and extraordinarily expensive to scale. Moreover, human raters exhibit high variance and personal inconsistency; without a sufficiently large sample size and multi-annotator redundancy, the evaluation risks measuring the noise of the raters rather than the capability of the model.
As an alternative, Patel introduces automated evals relying on statistical heuristics. A quintessential automated approach forces the model into structured multiple-choice questions (MCQs) to measure accuracy directly. Automated evaluations excel at assessing objective domains—such as factual knowledge recall, mathematics, or formal problem-solving—allowing teams to run high-volume, low-cost assessments instantly. However, their utility narrows significantly when applied to nuanced, open-ended, or subjective product tasks.
| Evaluation Method | Primary Advantages | Critical Drawbacks | Best Use Cases |
|:--- |:--- |:--- |:--- |
| **Human Rater Evaluations** | Direct qualitative assessment; captures nuanced, subjective criteria; supports head-to-head A/B testing and net win rate calculations. | Very expensive; slow turnaround; high inter-annotator variance and inconsistency requiring multiple raters per item. | Complex end-user workflows, conversational agents, subjective creative tasks. |
| **Automated / Heuristic Evaluations** | Highly scalable; zero marginal human cost; perfectly consistent and deterministic scoring. | Fragile when evaluating open-ended generation; largely restricted to structured formats (e.g., MCQs). | Objective benchmarks, factual knowledge retention, mathematics, deterministic coding tasks. |
[16:58](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1018s) The Eval Spectrum: From Automated Volume to AI Judges
During early model development, automated evaluations account for the overwhelming majority of compute and activity, constituting 90% to 95%+ of all evals run by a research team. In the beginning phases of training, models are not particularly smart, making rapid automated checks essential to ensure the model is actually making progress before deploying more expensive evaluation techniques. At the other end of the emerging spectrum are AI judge evaluations, which have surged in popularity as a way to simulate human raters by replacing the human evaluator with a secondary AI model.
```mermaid
flowchart LR
accTitle: AI Evaluation Methodology Spectrum
A["Automated Evals<br/>(90-95%+ volume, early training)"] --> B["AI Judge Evals<br/>(Frontier models, rubrics)"]
B --> C["Human Evals<br/>(Gold standard, high cost)"]
```
However, utilizing an AI judge introduces its own suite of failure modes and subtle biases. A critical pitfall occurs when teams use the exact model they are actively training or analyzing to judge its own outputs: doing so risks reinforcing the model's inherent blind spots and biases, leaving developers unable to diagnose what is actually going wrong. Standard industry practice therefore requires using an entirely separate, typically larger frontier model for evaluation rather than iterating on self-evaluating loops.
[18:39](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1119s) Inside Multiple-Choice Evals: MMLU and Log-Likelihood Scoring
To illustrate how automated evaluations work mechanically under the hood, standard multiple-choice benchmarks like MMLU (Massive Multitask Language Understanding) do not simply prompt the model and parse its conversational reply. Instead of letting the model freely generate text, the system feeds the prompt and question, then inspects the final output layer of the neural network—the layer containing roughly 40,000 logits/neurons corresponding to the vocabulary tokens.
```mermaid
flowchart TD
accTitle: Negative Log-Likelihood Choice Scoring Flow
P["Input Prompt + MMLU Question"] --> M["Neural Network Forward Pass"]
M --> L["Final Layer Output Logits (40,000 Neurons)"]
L --> R["Restrict Vocabulary to Options: A, B, C, D"]
R --> S["Calculate Negative Log-Likelihood / Softmax Probabilities"]
S --> W["Select Highest Probability Token as Predicted Answer"]
```
By restricting the output vocabulary specifically to the tokens representing options A, B, C, and D, the evaluation calculates the negative log-likelihood (NLL) across those candidate tokens. The option with the highest probability is scored as the model's selected answer without ever running full autoregressive generation. This technique avoids parsing errors, enables rapid computation of accuracy across thousands of benchmarks, and can similarly be extended to calculate the cumulative log-likelihood across longer target sequences.
[20:37](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1237s) Rubric-Based AI Judges and the Traps of Eval Noise
As frontier models grow more capable, AI-as-a-judge approaches are evolving from simple binary checks into sophisticated rubric evaluations. Teams can construct nuanced scoring rubrics that reflect explicit domain values and structural requirements, asking the judge model whether a response satisfies specific criteria, exhibits target characteristics, or handles complex constraints.
At the same time, practitioners must beware of over-indexing on minor fluctuations in benchmark numbers. Eval scores are susceptible to mechanical noise introduced by:
- Pipeline parallelism configurations
- Choice-based log-likelihood scoring versus free-form generation parsing
- Specific response regexes and parsing heuristics
- Hardware-level tensor core batch sizing and floating-point reduce operations
Because these infrastructure variations can easily cause a 1% to 2% swing in overall scores, model builders are cautioned against "reading the tea leaves where there's nothing to be read" and should avoid hyper-focusing on trivial statistical jitter.
[22:24](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1344s) Real-World Eval Strategy: Golden Datasets and Safe Production Iteration
When building real-world applications—such as an enterprise customer service agent—the single most important engineering investment is establishing an evaluation suite *before* writing application logic or attempting deployment. Because LLMs exhibit a "non-convex intelligence envelope" where performance does not degrade predictably across tasks, jumping straight into deployment without a robust benchmark invites catastrophic production surprises.
```mermaid
flowchart TD
accTitle: Real-World Evaluation and Deployment Workflow
D["Collect ~500 Real-World Complex Conversations"] --> G["Draft & Human-Review 'Golden Responses'"]
G --> B["Benchmark Human Reps vs. Golden Responses"]
G --> A["Benchmark AI System vs. Golden Responses"]
B & A --> S["Compute Quantitative Win-Rate / Score (0-100)"]
S --> I["Iterate Model / Prompts Until Target Met"]
I --> P["Deploy to Production with Confident Model Swap Capabilities"]
```
A practical recipe involves curating approximately 500 challenging real-world conversations—a sample size large enough to produce tight statistical confidence intervals—and crafting human-reviewed "golden responses" that capture thorny edge cases. Both human agents and the AI candidate system are benchmarked against these golden targets using automated metrics or LLM judges, yielding an absolute score (0 to 100) or a head-to-head win rate.
Establishing this quantitative baseline upfront delivers major long-term operational advantages:
- Teams can establish an objective threshold for when an AI system is ready for launch.
- Developers can swap backend models, test cheaper lightweight models, or switch API providers in production while verifying that response quality does not silently degrade.
- Engineering teams can decouple integration bugs from core modeling failures.
[27:02](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1622s) Q&A: Academic Benchmarks, Core Competencies, and Eval Evolution
Opening the floor to questions, an attendee asks whether existing academic benchmarks fall short and what missing benchmarks researchers wish existed. The issue is rarely that current benchmarks are defective; rather, teams often fail to apply them effectively to their specific problem spaces. While current evaluations focus heavily on siloed, narrow capabilities, the community is moving toward composite multi-task evaluations that measure performance across intertwined tasks simultaneously.
When asked which single evaluation is mandatory before entering production, Patel emphasizes that while a baseline reasoning eval is a sensible minimum for general tasks, evaluations must ultimately be tailor-made for the specific use case.
Responding to whether evals should iterate over time like software tests:
- Evals operate much like integration tests: they must actively evolve as live deployments uncover unforeseen failure modes.
- When an application encounters novel, thorny edge cases in production, those specific interactions should be sanitized, paired with correct outputs, and permanently integrated into the eval suite to systematically expand test coverage.
[30:06](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1806s) Q&A: The Multi-Turn Evaluation Dilemma and Conversational Drift
The discussion shifts to multi-turn interactions, where evaluating isolated turns without full context is notoriously difficult. While models perform well when providing a single 11th-turn response given 10 static historical turns, they frequently suffer from conversational drift when forced to sustain both sides of a continuous dialogue.
To test true multi-turn performance, researchers frequently set up two-agent simulations where a secondary model is assigned a specific agenda and hidden private information to interact with the target model. However, this introduces a severe "blame assignment problem": if the conversation derails because the simulator made an erratic move, the primary model is unfairly penalized.
One partial mitigation is to have the primary model role-play both personas simultaneously to remove asymmetric simulator errors, though multi-turn conversational evaluation remains an open, unsolved research challenge in the field. An audience member chimes in suggesting a rigid, rule-based state machine or router model as the conversational counterpart to provide deterministic inputs before pivoting to the challenge of evaluating subjective outputs that lack clear ground truth.
[33:27](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=2007s) Evaluating Subjective Responses and Overfitting Reward Models for Evaluation
When deploying agentic systems to replicate or approximate human behavior—particularly nuanced affective cues like valence signals—evaluating quality becomes challenging because there are often no straightforward black-and-white, right-or-wrong answers. An audience member asks whether methodologies from fields like quantitative psychology, such as structural equation modeling (SEM) for measuring latent variables in survey responses, can be adapted to benchmark how human-like an AI model's outputs are and refine evaluation rubrics.
```mermaid
flowchart TD
accTitle: Evaluation-Specific Reward Model Flow
A["Human Preference Data on Eval Set"] --> B["Train Targeted Reward Model"]
B --> C["Reward Model Overfit to Eval Questions (Permissible)"]
C --> D["Score Main Model Responses on Eval Benchmarks"]
E["Main Model Pre-training & Fine-tuning Data"] -. "Strict Data Isolation (No Leakage)".-> D
```
Rohit Patel explains that the most effective strategy developed so far is training a dedicated reward model on human preference data to capture subjective nuances. While training a full reward model solely to power an evaluation harness might feel like overkill, the investment pays off significantly. Patel highlights a crucial technical distinction when building reward models for evaluation versus general alignment:
* **General Alignment Reward Models:** Must generalize broadly across unseen domains without overfitting to a narrow training distribution.
* **Evaluation-Specific Reward Models:** Can be deliberately overfitted to the specific evaluation benchmark (such as a fixed set of a thousand target evaluation questions). Because the reward model is only ever used to grade that specific test suite, overfitting allows it to capture human preferences on those exact prompts with much higher fidelity.
* **Data Leakage Guardrail:** This strategy requires strict separation ensuring that the evaluation questions are never included in the primary generative model's training data.
---
[35:40](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=2140s) Mitigating Reward Hacking and Rubric-Based Advantage Modeling
A major challenge with reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) is reward hacking, where models exploit vulnerabilities in the reward function to generate degenerate or nonsensical text that nonetheless scores extremely high rewards. When asked how to prevent models from gaming subjective reward metrics, Patel notes that while evaluation harnesses are meant to be fast and cannot provide exhaustive real-world coverage, several architectural strategies can mitigate the issue during training.
```mermaid
flowchart LR
accTitle: Rubric-Based Scoring without Retraining Reward Models
A["Generative Model Output"] --> B["LLM Judges & Rubrics"]
B --> C["Direct Evaluation Scores"]
C --> D["Relative Advantage Function Calculation"]
D --> E["Policy Optimization / Reinforcement Learning"]
```
Rather than training a heavy, static reward model that is prone to overtraining, Patel suggests an alternative framework using modular LLM judges and rubric-based scoring. Under this setup, judges evaluate responses against structured rubrics to produce direct scores, which are then used to construct a relative advantage function without ever fitting a dedicated neural reward model.
- **Adaptability to Real-World Failure Modes:** "There is no real antidote to the real world," Patel emphasizes, noting that teams must continually observe new failure modes as they emerge in production.
- **Dynamic Rubric Updating:** In a rubric-and-judge architecture, addressing a newly discovered reward-hacking exploit simply requires drafting a new evaluation rubric or updating existing scoring rules, avoiding the time-consuming process of collecting preference pairs and retraining a separate reward model.
- **Flexible Optimization Framework:** Patel advises building policy optimization pipelines where the advantage function and reward calculations can be dynamically tuned and modified as rubrics evolve.
## Video description
Rohit Patel, Director, Meta Superintelligence Labs, Meta
Abstracts
This session looks at GenAI under the hood, breaking down how moden LLMs work and how neural networks generate language. We then discuss challenges to real-world AI deployment. We present an overview of AI evaluation and discuss why evaluation is key to ensuring success. We’ll compare automated, judge-based and human-rated methods for assessing AI performance. Attendees will learn how to build evaluations for their use-cases to assess AI performance and compare it to human performance, on their tasks, quantitatively.
AI评估入门：与Meta共同应对现实世界AI应用挑战
Rohit Patel 概述了在生产环境中严谨评估生成式 AI 系统所需的架构框架与实用工程方法。他强调，由于 AI 表现出非凸智能包络（即精通复杂任务并不保证能够胜任更简单的相邻任务），团队必须构建针对性的评估测试套件，而非依赖通用基准。本次演讲详细分析了人工评估员、自动化对数似然评分以及基于规则的 LLM 裁判之间的权衡取舍，并提出了在部署前使用包含 500 个样本的黄金数据集建立基线质量的具体策略。在问答环节中，Patel 还探讨了多轮评估漂移、基准噪声，以及为了捕捉微妙的人类偏好而对专用奖励模型进行刻意过拟合等复杂前沿课题。
针对 AI 展现出的非凸智能特性，团队须构建专门的评估套件，避免单纯依赖通用基准。
权衡人工评估、对数似然评分与 LLM 裁判的利弊，建议部署前利用 500 样本黄金数据集确立质量基线。
探讨了多轮评估漂移、基准噪声，以及对专用奖励模型进行刻意过拟合以捕捉人类细微偏好等前沿问题。
Reading summary
Rohit Patel argues that because generative AI exhibits a fundamentally non-convex capability profile, teams cannot extrapolate performance from adjacent tasks, making domain-tailored behavioral evaluation suites essential prior to deployment.
The Non-Convex Nature of AI Capabilities (2:08)
Patel categorizes AI deployment pitfalls into integration, organizational, feasibility, and performance failures. While feasibility failures involve expecting capabilities beyond current technology, performance failures occur when functioning systems fall short of quantitative business expectations. Diagnosing and mitigating this failure mode requires rigorous, targeted evaluation rather than relying on generalized intelligence assumptions.
To explain this necessity, Patel introduces the concept of a non-convex intelligence envelope. Unlike human professionals, whose mastery of advanced tasks allows safe assumptions about their proficiency in intermediate skills, large language models can master complex reasoning while failing elementary logic. Because capability does not degrade or improve predictably between two verified points, engineers cannot infer competence on a target task simply because it appears easier than an established benchmark.
Evaluation Architectures and Metric Noise (9:28)
Mathematically, Patel defines an evaluation as a monotone mapping from model space to the real line, ensuring superior systems receive strictly higher scalar scores. This architecture requires three pillars: a representative prompt sample, a scoring methodology, and metric aggregation. Scoring spans a spectrum: automated metrics handle 90% to 95% of early training volume, automated multiple-choice tests measure negative log-likelihood across restricted output logits without full text generation, and AI judges use rubrics to approximate qualitative human scoring.
Patel warns against self-evaluating models, noting that using the model under test as its own judge reinforces blind spots; teams should instead employ independent, larger frontier models. Furthermore, model builders are cautioned against over-interpreting 1% to 2% benchmark fluctuations, which often stem from mechanical noise such as pipeline parallelism, regex parsing variations, and floating-point batch reductions on tensor cores.
Production Readiness Through Golden Datasets (22:24)
For enterprise applications, Patel advises establishing an evaluation suite before writing application logic. A standard implementation curates approximately 500 complex, real-world interactions paired with human-reviewed golden responses. Benchmarking both human agents and candidate models against these baselines produces a quantitative net win rate or a 0-to-100 score, establishing objective criteria for launch.
This upfront benchmark allows engineering teams to swap backend models, experiment with lower-cost APIs, and separate integration defects from core model performance. Patel notes that evaluations must operate like software integration tests, continuously expanding by sanitizing and incorporating novel edge cases discovered during live production.
Multi-Turn Drift and Subjective Reward Modeling (30:06)
Evaluating multi-turn dialogues introduces severe complications, including conversational drift and a blame assignment problem when using two-agent simulations, where erratic inputs from a simulator model unfairly penalize the primary agent. Multi-turn assessment remains an unresolved challenge in deployment engineering.
For subjective domains lacking ground truth, Patel outlines the use of evaluation-specific reward models. Unlike general alignment models that must generalize widely, evaluation reward models can be intentionally overfitted to a fixed benchmark of evaluation questions to capture human preferences with high fidelity, provided strict data isolation prevents prompt leakage into pre-training. To mitigate reward hacking during policy optimization, Patel recommends modular rubric scoring and advantage calculations driven by LLM judges, which allows developers to update scoring criteria dynamically without retraining heavy reward models.
Patel emphasizes that automated evaluations and static benchmarks cannot fully safeguard against the complexities of the real world. Because conversational multi-turn dynamics and reward hacking remain persistent engineering hurdles, production teams must treat evaluations as living test suites that dynamically adapt to newly observed failure modes.
## Background
该视频由 Ai4 频道于 2025 年 8 月 27 日发布在 YouTube 平台，主讲人为 Meta 超级智能实验室（Meta Superintelligence Labs）总监 Rohit Patel。内容围绕生成式 AI 的底层运作机理展开，系统剖析了真实业务部署中的性能挑战，并全面介绍了如何构建自动化、AI 裁判与人工评估体系来定量衡量 AI 性能。
## TL;DR
演讲指出大语言模型的“智能包络是非凸的”，不能从某项高难度任务的表现推断其他任务的可靠性，因此构建专属评估体系是确保 AI 落地成功的核心基石。讲者详细对比了人工标注、自动化评估（如基于对数似然的 MMLU）和 AI 裁判三类范式，并演示了以 500 条真实棘手对话构建黄金基准（Golden Responses）的落地闭环流程。在问答环节，演讲深入探讨了多轮对话评估中的“动态漂移”与“责任归因”难题、主观场景下的偏好奖励模型拟合，以及利用细粒度评分规则（Rubrics）构建优势函数以缓解强化学习中的“奖励黑客”现象。
## Atmosphere & style
整个演讲节奏明快、干货密集，充盈着严谨而务实的顶级工业界技术工程研讨氛围。讲者在摸底观众技术背景后迅速切入数学本质与真实工程陷阱，风格生动自信、直击痛点，时而用形象的比喻化解抽象的底层运算逻辑。在后半段高密度的现场问答中，讲者毫不回避多轮评测与奖励黑客等业内前沿未决难题，展现出敏锐的技术洞察与极具实操价值的工程经验分享。
## Key takeaways
- [01:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=68s) 大语言模型的智能包络具有“非凸性”（Non-convex），在复杂任务上的优异表现无法推导出其在简单任务上必然可靠，因此必须为特定业务场景构建专属的定量评估。
- [04:02](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=242s) 神经网络底层本质上是纯粹的矩阵运算与数值映射，大模型的语言理解与生成完全依赖人类赋予的解释机制与 Token 词表概率分布。
- [10:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=608s) 评估（Eval）在数学上是将无序的模型空间映射到有序实数轴上的单调函数，核心由输入样本、评分机制与指标聚合三要素构成。
- [13:58](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=838s) 主流评估方法分为人工标注、自动化评测与 AI 裁判三类；自动化评估占据训练期 90% 以上的评测算力，而 AI 裁判严禁使用待训同款模型以避免强化自身偏见。
- [18:39](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1119s) 像 MMLU 这样的自动化评测可通过直接读取选项 Token 的负对数似然（Negative Log-Likelihood）完成，无需模型完整生成自由文本答卷，具有极高的计算效率。
- [20:17](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1217s) 硬件张量运算精度、批处理大小（Batch Size）及文本解析差异可能导致 1%~2% 的评估分值微小波动，开发者无需对这类系统底层噪声过度解读。
- [22:24](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1344s) 构建业务评估闭环的标准做法是收集约 500 条真实复杂用例并撰写标准参考答案（Golden Responses），这能在平替低成本模型时充当核心质量护栏。
- [25:25](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1525s) 完成自动化评估构建后，开发者必须亲自人工抽检得分与主观质量体感是否对齐，确保评测能够有效防范灾难性故障。
- [27:02](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1622s) 评估集应像软件工程中的集成测试一样，持续从生产环境中回流捕获的失败坏例（Badcases），实现用例覆盖面的动态扩展演进。
- [30:09](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1809s) 模型在多轮对话中容易产生漂移，引入模拟智能体评测时会面临责任归因难题（Blame assignment），可通过单模型分饰两角互博来降低评估噪声。
- [32:25](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1945s) 面对无绝对客观真值的主观任务，在不泄露给训练集的前提下，可在千道评测题库上过拟合训练专属的奖励模型（Reward Model）以精准捕捉人类偏好。
- [35:36](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=2136s) 针对强化学习中的奖励黑客（Reward hacking）漏洞，可采用大模型裁判结合细粒度 Rubrics 评分规则直接计算相对优势函数，并随线上新缺陷敏捷修订规则。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=4s) 开场与议程：为何要聚焦于 AI 评估
Rohit Patel 首先向现场观众介绍了本次演讲的核心主题：AI 评估（Evals）。在正式展开之前，他通过举手提问的方式摸底现场听众对“分词器（Tokenizer）如何工作”的了解程度。看到现场绝大多数人都非常熟悉基础概念后，他欣喜地表示这将是一个很懂技术的专业受众群体，因此他决定快速带过“什么是生成式 AI 以及语言如何生成”的基础铺垫，把绝大部分时间留给最关键的核心部分——“如何评估 AI”，并在最后预留充足的时间进行现场问答互动。
```mermaid
flowchart LR
accTitle: 演讲内容规划与时间分配
subgraph S1["快速回顾 (基础概念)"]
A["生成式 AI 原理"] --> B["语言生成与 Token 机制"]
end
subgraph S2["核心重点 (实践指南)"]
B --> C["AI 评估的核心定义与意义"]
C --> D["评估体系的构建与量化方法"]
end
subgraph S3["互动交流"]
D --> E["现场 Q&A"]
end
```
### [01:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=68s) 实际落地中的性能失效与“智能包络非凸”理论
在企业部署 AI 的实际场景中，团队常常会遇到多种实际挑战，包括模型集成失败、组织架构协调困难、技术可行性不足（即 AI 根本无法完成预定任务），以及最常见的“性能失效”（Performance Failure）——模型虽然成功部署上线，但实际业务表现却远未达到预期。针对这种性能无法达标的问题，最核心的技术解决方案就是系统化的评测（Evals）。
谈及为什么必须重视评测，Rohit Patel 提出了一个深刻的概念——AI 的“智能包络（Intelligence Envelope）是非凸（Non-convex）的”。当面对一个拥有数学或物理学博士学位的人类专家时，人们通常可以合理推断他在相关中等难度任务上的表现；但大语言模型（LLM）和多模态模型完全不同，它们可以在某些极其复杂的任务上表现惊艳，却在极其简单直观的任务上“惨败”（fail miserably）。
```mermaid
flowchart TD
accTitle: 人类能力与 AI 智能包络的凸性对比
subgraph Human["人类能力表现（具备凸性）"]
H1["精通高难度任务 A (如数学博士)"] --- H2["精通高难度任务 B (如物理研究)"]
H1 --> H3["自然掌握中间基础任务 C"]
H2 --> H3
end
subgraph AI["AI 智能包络（非凸特性）"]
A1["惊艳完成任务 A"] --- A2["惊艳完成任务 B"]
A1 -.->|"无法直接推导"| A3["可能在简单任务 C 上惨败"]
A2 -.->|"无法直接推导"| A3
end
```
- **非凸性推论**：即便模型证明了自己能解决任务 A 和任务 B，也绝对不能想当然地认为它能处理介于两者之间、看似更简单的任务 C。
- **评测的必要性**：不能依赖模型在相近场景或通用榜单上的表现，必须针对具体的业务场景（Use Case）构建专属的定量评测，才能确保最终交付的产品真正可靠。
### [04:02](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=242s) 神经网络的本质：纯粹的数字与人类赋予的“解释”
为了更好地理解评测指标的设计，Rohit Patel 深入到底层数学逻辑剖析了生成式 AI 的工作机制。神经网络的本质是由大量矩阵乘法（线性变换）与非线性激活函数交织而成的数学系统。这类模型根本不具备直接理解文本或真实世界概念的能力，它们只能接收数字作为输入，并输出数字。
人类之所以能通过神经网络实现分类或生成，完全依赖于人为构建的“解释机制（Interpretation）”。例如在区分“花（flower）”和“叶（leaf）”的二分类任务中，模型只是被要求输出两个数值。网络本身丝毫不懂什么是叶子或花朵，但人类制定了规则：如果第一个数值高于第二个数值，就将其“解释”为识别出了树叶。
```mermaid
flowchart LR
accTitle: 神经网络数值输入与输出的解释机制
subgraph InputStage["输入转换"]
A["现实概念 / 文本"] -->|"编码映射"| B["数值 / 向量"]
end
subgraph ModelStage["网络计算"]
B -->|"大规模矩阵乘法 + 非线性层"| C["输出数值 (Logits)"]
end
subgraph OutputStage["人工解释"]
C -->|"取最大值判定"| D["分类结果: 叶子 / 语言 Token"]
end
```
### [05:52](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=352s) 从字符生成到现代 Token 分词与 Embedding
沿着“解释数字”的逻辑，Rohit Patel 拆解了模型生成文本的演进过程。如果让网络输出 26 到 28 个数字（对应 26 个字母以及空格、标点符号），并将输出的最大数值解释为模型对下一个字符的预测，再将生成的字符作为自回归输入重新喂回模型，就构成了一台最基础的“字符生成机器”。早期如果采用最简单的数字编码（如 $a=1, b=2$），在足够庞大的语料库上训练，网络确实能学会生成可读的语言。
而在现代大语言模型中，这种机制被升级为基于 Token（词元）的体系：
- **Token 词表**：现代模型通常采用约 50,000 左右大小的词表，模型输出层包含 50,000 个数值，分别对应词表中的各个 Token，模型通过预测下一个 Token 的概率分布来连续生成文本。
- **词嵌入（Embedding）**：在输入端，现代模型不再采用固定的标量数字映射，而是为每个 Token 分配一个连续向量，并将其作为网络参数与模型一同训练，使向量能够精准捕捉词汇在语义空间中的丰富含义。
### [09:18](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=558s) 明确定义 AI 评估的目标
在理清了模型如何从数值生成语言之后，Rohit Patel 切入评估的正式定义：当企业试图构建一项特定能力时——例如写代码的能力，或是专属于自身业务的智能客服机器人——评估的核心目的就是对当前世界上存在的所有候选模型、Agent 架构或复合 AI 系统进行科学、定量的能力打分与排名，从而挑选出在目标任务上表现最优的技术方案。
### [10:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=608s) 评估的数学本质与核心三要素
评估（Eval）在数学上本质是一个映射函数：将“所有可能模型或系统的空间”映射到“实数线”（Real line）上。给定任意系统，评估流程会输出一个标量数值。这个映射必须在能力维度上保持“单调性”（monotone）——能力越强、表现越优秀的系统应当获得更高的分值，表现越差的系统得分越低。
```mermaid
flowchart LR
accTitle: 模型评估的数学映射流程
subgraph InputSpace["模型空间 (无序集合)"]
M1["模型 A"]
M2["模型 B"]
M3["模型 C"]
end
subgraph EvalProcess["评估映射 f(·)"]
P["采样测试集 Prompt"] --> R["评分方法 / 偏好对比"]
R --> S["聚合指标"]
end
subgraph OutputSpace["实数轴 (有序标量)"]
Score["得分: 92.5 > 84.1 > 70.3"]
end
InputSpace --> EvalProcess
EvalProcess --> OutputSpace
```
构建一个评估系统其实并不神秘，归结起来只有三个核心组件：
1. **输入提示词样本（Sample of input prompts）**：从所有可能的使用场景问题中抽取具代表性的子集。一个神经网络模型本质上由架构、权重参数以及输入/输出解释机制组成，由于模型拥有海量权重，所有模型构成的集合并不是一个天然有序的集合（not an ordered set），直接比较网络内部参数毫无可能。因此，实际做法是将问题降维：不直接评判模型内部，而是评判模型在特定问题上的生成结果；由于无法穷尽线上所有的可能输入，只能抽取一个高质量的小样本集合。
2. **评分机制（Methodology for ranking / scoring responses）**：确定如何评价和对比模型针对这批测试样本给出的具体输出质量。
3. **指标聚合（Metric combination）**：把模型对各个样本的偏好排序或分值，融合成一个最终的单一数值（single number）。
---
### [13:58](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=838s) 三类主要评估范式：人工、自动化与 AI 裁判
当前主流的评估方法主要分为三类，它们在成本、可扩展性和适用场景上存在鲜明对比：
| 评估类型 | 优势 | 局限与挑战 | 典型适用阶段 |
|:--- |:--- |:--- |:--- |
| **人工标注员评估<br>(Human-rater Eval)** | 直观、真实，支持模型间或模型与人类水平的直接对比。 | 运行成本极其高昂；标注员之间方差极大（inconsistent）；难以快速大规模迭代。 | 最终产品交付前对齐、小规模精细基准评测。 |
| **自动化评估<br>(Automated Eval)** | 计算开销极低、执行极快，能提供绝对一致的客观指标。 | 依赖统计启发式或客观答案，表达能力有限，难以评测主观生成任务。 | 模型预训练及早期训练阶段，占据 **90%~95% 以上**的评测算力与频次。 |
| **AI 裁判评估<br>(AI Judge Eval)** | 兼具人工评估对复杂文本的理解力与接近自动评估的高吞吐。 | 存在模型固有偏见（Biases）；若用待训模型自己当裁判，会强化自身盲区。 | 当前学术与工业界最热门的研究方向，正处于爆发式增长中。 |
在人工标注中，常见指标是**净胜率（Net Win Rate）**，即“模型 A 的回答优于模型 B 或优于人类回答的次数占比”。为了抑制标注员之间的打分波动，工程上往往需要引入多名标注员交叉验证，但这进一步成倍推高了时间与财务成本。
自动化评估则极其适合有客观标准的领域（如常识问答、数学推理等）。在模型训练初期，模型能力通常非常低，研究团队在启动昂贵的进阶评估之前，必须依赖自动化评估跑通第一轮筛选。
在引入 AI 裁判（LLM-as-a-judge）时，工业界的标准做法是**严禁使用正在迭代优化的同款模型去担任自己的评判者**，否则系统只会反复强化自身已有偏见，根本无法发现模型暴露的真实缺陷。
---
### [18:39](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1119s) 自动化评估实例：基于对数似然的 MMLU 评测机制
演讲以广泛使用的 MMLU（Massive Multitask Language Understanding）基准测试为例，深入剖析了自动化评估在底层是如何绕过漫长的自由文本生成直接完成打分的。
当给模型输入一道包含 A、B、C、D 四个选项的单选题时，系统在工程上其实**并不需要模型完整输出一段文本答卷**。
```mermaid
flowchart TD
accTitle: 基于 Log-Likelihood 的 MMLU 自动化评估机制
Prompt["输入题目 Prompt + A/B/C/D 选项"] --> LLM["大语言模型前向计算"]
LLM --> Logits["最终输出层：~40,000 个 Token 的 Logits 概率分布"]
Logits --> Filter["词表裁剪：仅提取 Token 'A', 'B', 'C', 'D'"]
Filter --> NLL["计算四者的负对数似然 (Negative Log-Likelihood)"]
NLL --> Best["挑选概率最高者作为模型预测答案，并对比标准答案"]
```
- 在模型的最后一层神经元中，通常会输出涵盖数万个 Token（例如 40,000 词表）的概率分布。
- 自动化评估系统会直接把词表严格限制在候选选项对应的 4 个特定 Token 上（例如 `'A'`, `'B'`, `'C'`, `'D'`）。
- 评测系统只读取这 4 个 Token 的负对数似然（Negative Log-Likelihood）或生成概率。
- 概率最高的选项即视为模型预测的答案；"模型根本没有真正生成回答，但如果它真的生成，概率最高的就是这个答案"。
这种基于对数似然的评估机制不仅极其廉价、高效，而且具备极强的扩展性——该方法同样可以推广到更长、结构更复杂的序列评估之中。
### [20:17](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1217s) 大模型作为裁判与评估中的底层“扰动”
在使用对数似然（Log-Likelihood）和自回归生成的常规准确率评估之外，利用前沿大模型作为裁判（AI Judge）正成为极其活跃的研究方向。随着前沿基础模型的能力越来越强，它们充当裁判的效果也显著提升。当下一种越来越流行的做法是编写非常详尽的评分标准（Rubrics），将开发者自身的价值观与业务标准注入到评测中。具体而言，裁判模型会逐项核对生成的回答是否满足特定准则、是否具备某项特定特征，从而形成结构化的评判。
然而，构建优质的评估体系存在不少潜在的陷阱与工程挑战。许多微小的评测波动并不代表模型能力的实质变化，而是由底层技术细节引起的：例如流水线并行（Pipeline Parallelism）的切分方式、选择生成式评测还是多项选择题评测、输出结果的文本解析（Parsing）逻辑差异，甚至具体硬件架构中张量核心（Tensor Cores）在不同批处理大小（Batch Size）下执行归约操作（Reduce Operations）时的微小数值差异，都会影响最终的评测得分。因此，开发者完全没必要为 1% 到 2% 的微小评测波动而焦虑纠结，在迭代模型时切忌“在没有痕迹的地方强行占卜解读”（"reading the tea leaves where there's nothing to be read"）。
- **AI 裁判评分机制**：通过制定细颗粒度 Rubrics，让裁判模型检查输出是否符合设定的多项核心特征与业务约束。
- **底层扰动因素**：硬件张量运算的归约精度、Batch Size、流水线并行切分及解析规则均可能导致 1%~2% 的数值浮动。
---
### [22:24](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1344s) 落地实战：以客服机器人为例构建评估闭环
在实际业务场景中（例如构建智能客服系统），团队应遵循“评估先行”的策略，在投入大量资源开发其他模块之前，先建立起可靠的评估基准。标准做法是提取约 500 场真实的复杂业务对话，这一样本量足以提供收敛且置信度较高的置信区间。接着，组织人工团队审核并撰写代表业务上限的高质量“标准参考答案”（Golden Responses），确保这些测试用例覆盖实际业务中最棘手、最复杂的真实问题。随后，让待部署的 AI 系统或人工客服分别对这些对话进行回答，并通过自动化规则或 AI 裁判对表现进行打分（可以是百分制绝对分数，也可以是针对标准答案的胜率 Win Rate）。
```mermaid
flowchart TD
accTitle: Real-world AI Evaluation and Deployment Workflow
accDescr: A flow diagram showing how real conversations are curated into golden datasets to benchmark AI vs human performance, enabling confident model iteration and production swapping.
subgraph Prep ["数据准备阶段"]
A["收集 500+ 真实棘手对话"] --> B["人工编写并审核 Golden Responses"]
end
subgraph Eval ["基准评测闭环"]
B --> C["系统/人工生成回答"]
C --> D["AI 裁判 / 自动化规则打分"]
D --> E["计算绝对得分或胜率 (Win Rate)"]
end
subgraph Deploy ["业务迭代与上线"]
E --> F{"是否达到业务准入阈值?"}
F -- 否 --> G["调整 Prompt / 微调模型迭代"]
G --> C
F -- 是 --> H["上线投产 & 无缝平替低成本模型"]
end
```
这种前置构建的自动化评估流程能够让团队清晰对比人类客服与 AI 系统的表现差距，并在明确设定的业务达标阈值指引下高效迭代模型。更为关键的是，即便系统已经上线投产，这套评测基准仍具有巨大的战略价值：当团队后续希望切换到成本更低的轻量模型、或者更换底层 API 供应商时，评估体系可以提供强大的质量护栏，确保在降低成本的同时核心模型生成质量不会出现意外退化。
- **样本规模**：推荐选取约 500 条真实样本以获得统计学上足够紧凑的置信区间。
- **评估指标**：支持百分制绝对打分或相对 Golden Response 的胜率对比。
- **平替保障**：在线上模型迁移至更低成本模型或第三方平台时，评测集能确保核心能力不发生灾难性回退。
---
### [25:25](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1525s) 评估核心原则总结与问答环节开启
大语言模型展现出的是一种典型的“非凸智能包络”（Non-convex Intelligence Envelope），这意味着模型在一个维度的优异表现并不能保证其他任务的可靠性，因此在项目初期优先构建评估体系至关重要。评估的构建逻辑本身非常清晰：收集真实 Prompt、撰写 Golden 参考答案，并借助前沿基础模型作为裁判进行对比分析。在评估体系搭建完成后，最关键的一步是开发者必须亲自审视评测数据与得分，确保评估分数的高低与人类对质量的真实感知高度契合——"Make sure the eval associates with how you perceive quality"。只有当人类的主观判断与自动化评分达成一致时，这套评估体系才能真正发挥防范未知灾难性故障的作用。
- **核心观点**：语言模型能力是非凸的，必须针对具体任务进行专属评测。
- **质量校准**：构建自动化评测后，必须人工抽检模型得分，确保评分结果与人工质量体感对齐。
---
### [27:02](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1622s) 问答：基准测试的局限、必测能力与评估的持续演进
在演讲后的问答环节中，现场观众就评测基准与实操细节提出了多个关键问题：
针对学术界现有公开 Benchmark 的有效性，讲者认为现存的大量基准本身非常优秀，问题往往在于使用者未能合理有效地使用它们。当前大多数基准往往孤立地针对单一任务，而更理想的发展趋势是涌现更多融合了多种任务的“复合型评估基准”（Combination Evals）。
当被问及“模型上线前是否存在一个必须测试的通用评估指标”时，讲者指出，虽然很难找到一个完全脱离业务场景的万能评测，但基础的“逻辑推理评估”（Basic Reasoning Eval）通常是所有模型都应具备的底线能力。不过，评估的核心始终必须高度绑定具体业务场景；如果是纯艺术类的图像生成场景，推理能力的重要性可能就会相对次要。
关于“评估集是否需要随着应用开发持续演进”，讲者给出了极其肯定的回答。评估集本质上就是一组代表性的提示词样本，在实际部署过程中，线上必定会遇到事前未预料到的复杂边缘案例与失败模式。开发者的标准操作应当是修复模型对这些样本的处理，并将这些失败样本抽样回流沉淀到评估集中，持续扩大评测覆盖面。因此，评测集不单单像传统非 AI 应用中的单元测试（Unit Test），它在某种程度上更像是一种能够随业务深入而不断扩展用例的“集成测试”（Integration Test）。
- **复合型基准**：行业需要更多将多项复杂任务混合评测的复合 Eval，而非单一割裂的基准。
- **底线能力**：基础逻辑推理评测通常适用于绝大多数模型，但终究要为具体用例让位。
- **测试演进**：将生产环境捕获的失败 Badcase 抽样补充进评测集，使评估如同集成测试般持续扩张覆盖面。
### [30:09](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1809s) 多轮对话评估的困境与双智能体模拟机制
在现场提问环节，有观众提出了关于长上下文与多轮对话评估的难题：在脱离前序对话上下文的情况下，单看基准黄金集（Golden datasets）很难对模型进行真实评测。Rohit Patel 坦言这是一个极其棘手且行业内尚未彻底解决的难题。他指出，大语言模型往往在单轮问答（Single-turn）上表现出色，即使面对看似属于多轮对话中的某个单一步骤，如果直接给定前 10 轮历史让模型生成第 11 轮回复并对其打分，评测结果通常很理想；但如果在真实场景中让模型自主接管并生成全部 10 轮交互，模型就会随着对话深入而逐渐出现“漂移”（Drift）。
```mermaid
flowchart TD
accTitle: Multi-Turn Conversation Evaluation Workflow
subgraph MultiTurnEval ["多轮对话评估方案"]
Sim["模拟智能体 / 固定规则模型"] <-->|"模拟交互 (包含私有信息/目标)"| Target["主评估模型 (Target Model)"]
Target --> Output["完整对话轨迹 (Trajectory)"]
end
subgraph EvaluationCheck ["评分与归因"]
Output --> Judge["判定检查点 (Checkpoints)"]
Judge --> Problem{"存在归因偏差？ (Blame Assignment)"}
Problem -->|解决方式| SingleAgentSelfPlay["让主模型同时扮演双方 Agent 消除归因误差"]
end
```
为了应对多轮对话评测中的动态漂移，业内目前尝试的一种思路是引入另一个模型充当对话对手来模拟真实交流。在这类评测设定中，两个模型各自被赋予特定任务目标（Agendas）和私有信息（Private information），随后由系统按照预设的检查点（Checkpoints）和评估标准对整场对话进行通盘打分，检验主模型是否犯错。然而，这种双模型对抗机制会引发棘手的“责任归因问题”（Blame assignment problem）——当对话出现差错时，难以界定究竟是主评估模型失误，还是协助模拟的对手模型给出了糟糕的输入。针对这一点，Rohit Patel 提出可以让同一个主评估模型分饰两角、同时扮演两套不同目标的智能体进行左右互搏，从而消除归因偏差，不过他也强调多轮评测依然是一个开放性挑战，期待业界提出更多新思路。
- **多轮漂移（Multi-turn drift）**：模型在独立回答给定上下文的问题时表现优异，但在连续自主生成多轮交互时极易偏离目标。
- **责任归因难题（Blame assignment problem）**：利用模拟模型（Simulator）进行对话评测时，模拟方的失误可能被错误归咎于受试模型。
- **单模型双角色方案**：通过 Prompt 驱动同一模型同时模拟对话双方，以此规避外部模拟模型能力不均带来的评测噪声。
---
### [32:25](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1945s) 无标准答案场景下的主观情感评测与偏好奖励模型
随后有观众顺着多轮对话思路提出补充建议：是否可以将模拟端设计为高度受控的规则化模型（Rule-based model）以确保输出稳定性；紧接着该观众抛出了自己的核心问题：当面对无绝对客观真值（Ground truth）的任务（例如量化心理学中测量受试者情绪效价 Valence 或潜在变量 Latent variables）时，应当如何评估模型回复的“类人度”（Human-likeness）？
Rohit Patel 回应称，解决缺乏客观标准答案（Black-or-white right/wrong answer）的最佳实践依然是基于人类偏好数据训练专属的“奖励模型”（Reward Model）。尽管为了单纯的评测（Eval）去专门训练一个奖励模型听起来有些“大材小用”（Overkill），但在具备人类偏好标注数据的前提下，这种投入回报极高。在评测场景下训练奖励模型有一个独特的优势：评测者无需过度担心泛化性，允许奖励模型在特定的千道评测题库上发生过拟合（Overfitting）。只要绝对保证这批评测问题不会泄露到主模型的预训练与微调数据集中，在固定评测集上高度拟合的奖励模型便能极其精准地捕捉并打分出符合人类偏好的细微倾向。
---
### [35:36](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=2136s) 奖励黑客防御与基于 Rubric 判规的优势函数评测框架
针对奖励模型，现场观众紧接着追问了模型训练与强化学习中常见的“奖励黑客”（Reward hacking）现象——在大语言模型进行人类反馈强化学习（RLHF）或 AI 反馈强化学习（RLAIF）时，模型往往非常善于钻奖励模型的漏洞，生成满嘴高分却毫无实际意义的“垃圾内容”（Absolute garbage inputs），该如何有效缓解这一失效模式？
```mermaid
flowchart LR
accTitle: Rubric-Based Reward and Advantage Framework
subgraph TraditionalRLHF ["传统模式"]
RM1["训练复杂奖励模型 (RM)"] --> Hack["容易被模型钻空子 (Reward Hacking)"]
end
subgraph RubricAdvantage ["免训练评判框架"]
Rubric["灵活的评判细则 (Rubrics)"] --> JudgeLLM["评委模型判分 (LLM-as-a-Judge)"]
JudgeLLM --> Scores["直接产出得分"]
Scores --> AdvFunc["计算相对优势函数 (Advantage Function)"]
end
```
Rohit Patel 解释道，奖励黑客本质上属于模型训练与强化学习对齐层面的挑战，评测本身很难作为单一解药，因为评测集的设计初衷是轻量、快速运行，无法完全覆盖所有潜在的边界漏洞。除了控制奖励模型的训练强度、避免过度拟合之外，他在近期思考并撰写过一种更具灵活性的替代架构：**跳过单独训练奖励模型的繁琐步骤，直接利用大模型作为评委（LLM-as-a-Judge）结合细粒度的评分规则（Rubrics）直接打分**。通过这些规则得分直接构建相对优势函数（Relative advantage function），不仅避免了训练参数化奖励模型的脆弱性，还赋予了开发者极高的灵活性。
面对真实业务中层出不穷的边缘失效模式，Rohit 总结道：“现实世界没有万灵丹（There is no real antidote to the real world）。”一旦在实际部署中观察到模型钻漏洞的新表现，最根本的解决路径依然是快速迭代框架：若采用奖励模型，就向其中追加标注样本；若采用评分规则体系，就即时扩充和修订 Rubric 判规，构建能够动态调整奖励函数与优势函数的敏捷评测闭环。
## Notable quotes
> [01:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=68s) fail miserably — Rohit Patel
> [10:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=608s) not an ordered set — Rohit Patel
> [10:08](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=608s) single number — Rohit Patel
> [13:58](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=838s) inconsistent — Rohit Patel
> [20:17](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1217s) reading the tea leaves where there's nothing to be read — Rohit Patel
> [25:25](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1525s) Make sure the eval associates with how you perceive quality — Rohit Patel
> [32:25](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=1945s) Black-or-white right/wrong answer — Rohit Patel
> [35:36](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=2136s) Absolute garbage inputs — Rohit Patel
> [35:36](https://www.youtube.com/watch?v=j9LwI9uYm6M&t=2136s) There is no real antidote to the real world — Rohit Patel
## Mentioned
Rohit Patel, Meta, Meta Superintelligence Labs, Ai4, MMLU
## Video description
Rohit Patel, Director, Meta Superintelligence Labs, Meta
Abstracts
This session looks at GenAI under the hood, breaking down how moden LLMs work and how neural networks generate language. We then discuss challenges to real-world AI deployment. We present an overview of AI evaluation and discuss why evaluation is key to ensuring success. We’ll compare automated, judge-based and human-rated methods for assessing AI performance. Attendees will learn how to build evaluations for their use-cases to assess AI performance and compare it to human performance, on their tasks, quantitatively.
[Source](https://www.youtube.com/watch?v=j9LwI9uYm6M)
Source quotes
[1:08] Rohit Patel: fail miserably
[10:08] Rohit Patel: not an ordered set
[10:08] Rohit Patel: single number
[13:58] Rohit Patel: inconsistent
[20:17] Rohit Patel: reading the tea leaves where there's nothing to be read
[25:25] Rohit Patel: Make sure the eval associates with how you perceive quality
[32:25] Rohit Patel: Black-or-white right/wrong answer
[35:36] Rohit Patel: Absolute garbage inputs
[35:36] Rohit Patel: There is no real antidote to the real world
Original transcript
[0:04] Uh so today we're going to talk about
[0:06] evals. Um hopefully you know a lot of
[0:09] people in this room have an idea of what
[0:10] this is but we are going to talk about
[0:12] we're going to try and define evals.
[0:14] We're going to talk about how to build
[0:15] them and why we should care about evals.
[0:19] Um and before I get started um just by a
[0:24] show of hands just a question mostly to
[0:26] get a sense for me to get a sense of how
[0:28] much I I need time I need to spend
[0:30] talking about u the first topic which is
[0:32] what is geni and how is language
[0:34] generated. Uh how many people here have
[0:36] a sense of how tokenizers work?
[0:41] Love it. Fantastic. Okay. So this is a
[0:44] great audience. Um so we're so I'm going
[0:46] to go maybe a bit faster through the
[0:48] first two and we'll spend bulk of our
[0:50] time on how to evaluate AI the last
[0:51] thing here. Um and then I'll try and
[0:54] save up some time towards the end for uh
[0:56] Q&A. Uh and that way we'll be able to
[0:59] you know sort of make a mental note of
[1:01] any questions you have popping up in
[1:02] your head as as I'm going through and
[1:04] then we'll take time to cover go through
[1:06] that.
[1:08] Um so the the title of the talk is you
[1:11] know uh addressing practical challenges.
[1:14] Uh here we're going to talk about one
[1:15] particular type of practical challenge
[1:17] which is uh model failure or product
[1:19] failure to deliver. There can be many
[1:21] different uh types of uh problems when
[1:24] you're trying to deploy AI. Uh you could
[1:26] have uh model integration failure. You
[1:28] could have organizational issues. Um you
[1:32] could have uh feasibility problems which
[1:34] is uh you know um maybe the AI is just
[1:37] not able to do the thing you're trying
[1:38] to get the AI to do uh more common than
[1:41] uh you'd expect. Uh and you could have
[1:44] performance failure that is the product
[1:45] doesn't do uh or meet expectations. So
[1:47] you've deployed it, you've integrated
[1:48] it, it's just not doing as well as you
[1:50] wanted it to do. This is the last thing.
[1:52] This is what we're going to talk about
[1:54] today. um and how to and we're going to
[1:56] talk about how do what what is the
[1:58] technical solution to to a problem like
[2:00] this uh which is eval right um but
[2:03] before we get into that um let's talk
[2:06] briefly about why do we care why should
[2:08] we care about eval in the first place um
[2:10] and the reason for that is that um what
[2:13] I call like the intelligence envelope of
[2:15] um AI is not convex uh what what I mean
[2:19] by that is that when you have um a
[2:22] person when you meet a person and
[2:24] they're capable of doing certain things.
[2:26] Um there is sort sort of a general
[2:28] understanding that if they can do uh
[2:30] maybe if they have a PhD in physics or
[2:33] math or something like that and and
[2:34] they're very good at certain other
[2:36] things, you can fill in the gaps and you
[2:38] can assume that they'll be able to do
[2:39] certain things in between. Um but with
[2:43] these um language models or or you know
[2:46] language vision models um that's not
[2:48] necessarily the case. uh they can do
[2:51] some stunning things and fail uh very
[2:54] miserably at some very simple task. Um
[2:56] so what that says is that uh the
[2:59] frontier or the the the envelope of
[3:01] intelligence isn't convex. If you take
[3:03] any two points uh it's not necessarily
[3:06] the case that and it does really well on
[3:08] those two things. It's not necessarily
[3:09] the case that anything in between is
[3:11] going to be good at. Um the implication
[3:14] of this is it's not obvious um upfront
[3:18] what you can do with AI. Um you know
[3:22] just because you think it can do A and
[3:24] B, you can't really necessarily always
[3:26] deduce that it'll be able to do C
[3:28] because C is much easier than A and B or
[3:30] is some sort of a convex combination of
[3:31] A and B or something like that. Um so
[3:35] how do you address that? You address
[3:36] that by sort of trying to build an eval
[3:39] for your particular use case. Um, and
[3:42] we'll talk briefly about what what Eval
[3:43] is, but um, what you want to make sure
[3:46] is you want to try and figure out how
[3:48] good it is at your use case, not at some
[3:51] adjacent use case, at exactly what it is
[3:53] that you're trying to get it to do. Uh,
[3:55] and that's when you can be reasonably
[3:56] assured that any product, any
[3:57] deployment, any anything you're building
[3:59] is actually going to work.
[4:02] Okay, so we'll come back to this. Um,
[4:04] I'm going to sort of quickly breeze
[4:06] through what's Gen AI. Um and the reason
[4:10] this is cuz we'll we'll talk a little
[4:11] bit about um how we did when we designed
[4:13] evals um some of this stuff. Um so
[4:16] judging by the earlier question in the
[4:18] room this should all be very familiar to
[4:19] everybody. So neural networks are um
[4:22] basically just a bunch of mostly linear
[4:25] equations interled with some sort of a
[4:28] nonlinearity which I'm not showing here
[4:30] on the board but ultimately the math is
[4:32] relatively simple. you're doing tons and
[4:34] tons of matrix multiplications
[4:37] and um the key thing here is um
[4:40] everything is numbers. Uh these models
[4:42] are not capable of doing anything beyond
[4:44] uh producing numbers and ingesting
[4:46] numbers. Uh they cannot they do not um
[4:50] ingest uh text or anything else. You
[4:52] have to convert everything to numbers
[4:53] before you feed it into a neural
[4:55] network. And what you get in the end is
[4:57] also a number. And um the key to Genai
[5:01] is try and figure out an interpretation
[5:03] um uh of of these numbers that allows
[5:06] you to interpret um those things as as
[5:09] something else as language as as
[5:11] whatever it is you may want it to be. So
[5:13] in this case, for example, if you were
[5:14] trying to build an AI that could
[5:16] classify data into two categories,
[5:19] flower and leaf, uh one possible
[5:21] interpretation here could be I could
[5:23] make it generate two numbers. Um, I can
[5:26] make my neural network output two
[5:27] numbers and I could say whichever number
[5:29] is higher that's going to I'm going to
[5:31] interpret that as the uh classification
[5:34] that the network has made for me. The
[5:36] network doesn't have an understanding of
[5:38] what a leaf is or a flower is. Um, but
[5:41] if it gives me the the number on the top
[5:43] as the higher number, I will assume it's
[5:45] a leaf. Uh, if it gives me the number.
[5:47] So, this is an interpretation I'm
[5:48] attaching to the numbers that the
[5:50] network is providing.
[5:52] And um as you can sort of maybe think
[5:55] follow up from this is that you can use
[5:57] this very nice fact to create uh more
[6:01] complicated networks and generate
[6:02] language. For example, instead of two
[6:03] numbers, you can maybe make it generate
[6:05] 26 or 28 numbers. Uh, you know, one for
[6:09] each character. And you can come up with
[6:12] a very nice interpretation, which is um,
[6:14] you know, whatever is the highest number
[6:16] out of these 26 or 28 numbers, let's
[6:19] let's throw in a space and an
[6:20] exclamation mark and some other symbols
[6:22] in there. Um, that's the prediction of
[6:26] the network for the next character. So
[6:28] this is an interpretation we could
[6:29] design that would give you a model that
[6:32] is capable of producing characters a
[6:34] single character at a time. You can feed
[6:36] this character back into the model and
[6:38] we'll talk briefly about how do we feed
[6:40] these character back into the model. Um
[6:42] bear bear with me for a minute. You can
[6:45] feed these characters back into the
[6:46] model and you can get the next character
[6:49] and so on and so forth. So you see you
[6:52] have a character generation machine of
[6:54] sorts just by making the model a neural
[6:57] network output 26 numbers and pretending
[7:00] that each of these 26 numbers
[7:02] corresponds to um a character. So we've
[7:06] gone from sort of like numbers to be
[7:08] able to generate characters. Um the
[7:11] problem is we still need to feed these
[7:13] characters somehow into the model. And
[7:15] uh maybe the stupidest thing we can do
[7:17] is just assign a number to each
[7:18] character like a= 1, b= 2, and we'll
[7:21] just put these in. Um, and if you
[7:24] actually try to do this, if you actually
[7:26] try to put a= 1, bals 2, that would be
[7:30] what we would call your embedding, um,
[7:33] and just had it generate 26 or 28
[7:36] numbers with the space and try to train
[7:38] a model. You're actually going to get
[7:40] something reasonable. if you trained it
[7:41] on a big enough corpus, this will give
[7:43] you a model that will generate language.
[7:45] And people have actually done that. Um,
[7:47] and it actually kind of works.
[7:49] Obviously, um, modern language models
[7:52] don't work like that. Um, modern models
[7:55] work on something called a token, which
[7:57] is, uh, you can think of it as worse.
[7:59] Um, maybe a smaller dictionary of words.
[8:03] Some words are broken up, so you don't
[8:04] really have a separate token for dog or
[8:06] dogs or cat or cats. Um but usually but
[8:11] essentially what we do here is you have
[8:13] maybe a vocabulary of tokens which is
[8:15] about 50,000 or so and you just sort of
[8:19] like go around creating models that will
[8:21] have 50,000 numbers in the final output
[8:24] and you pretend each one of them is
[8:26] corresponding to one of these words or
[8:28] tokens and that's how you generate
[8:30] language. So this is the current state
[8:32] of things. This is sort of how you
[8:33] generate language today. um in terms of
[8:36] feeding these words back instead of
[8:38] assigning a single number to each of
[8:39] these words uh you assign a vector and
[8:44] um what we call an embedding is
[8:46] something where you let the vector be
[8:48] trained along with the model. So, so you
[8:51] pretend that the input vector for that
[8:53] particular uh token isn't something that
[8:56] you're going to come up with in a very
[8:58] stupid scheme that we did earlier. You
[8:59] know, a equals b um equals 1, bals 2,
[9:04] etc. Instead, you're going to pretend
[9:07] they're sort of neurons in your network
[9:09] and you're just going to train them. So,
[9:10] that will give you a a sort of a vector
[9:13] for each word which is more fitting of
[9:14] what the word's meaning is. Um,
[9:18] cool. So we're past um how these models
[9:22] are and what they do and how we get
[9:23] language from just numbers, right? So
[9:25] it's it's purely about interpretation of
[9:27] these things. Um so let's talk about how
[9:30] do we evaluate them. Um first maybe
[9:33] we'll segue let's let's talk about what
[9:35] an eval is. So here what we're trying to
[9:39] do is with with an evaluation is let's
[9:41] say you're trying to build an evaluation
[9:43] for a particular capability. You want a
[9:44] model that is very good at um maybe
[9:48] coding or or you're trying to build a
[9:49] customer service bot for your particular
[9:51] business and and for that particular
[9:53] task you want to rank all the models uh
[9:56] in the world that exists on how good or
[9:58] or agents it could be model agent
[10:00] whatever some system of of some sort uh
[10:03] on on on your capability. So this system
[10:06] is the best and you want it to have the
[10:08] highest score. This system is is maybe
[10:10] the second best and it has slightly
[10:11] lower score and so on so forth. So what
[10:13] an eval is really is a mapping from the
[10:16] space of all possible models or systems
[10:19] to the real line. So you give it a
[10:21] system and it will give you a number and
[10:24] uh and this um basically what you want
[10:27] this mapping is to be monotone in terms
[10:29] of the capability. So models or systems
[10:32] that are better should get higher
[10:33] numbers. Systems that are worse should
[10:35] get lower numbers. So that's essentially
[10:37] an eval. This is what we're trying to
[10:39] build today. and uh and we want to talk
[10:42] about how we would build that. Right?
[10:45] So an eval is actually um not a very
[10:49] complicated thing to understand. Um it's
[10:51] really only three things. Uh first is
[10:53] you want a sample of input prompts. So
[10:57] why do you want this? Why why do you
[10:58] need a sample of input problems? So
[11:00] think think back to what we're trying to
[11:01] do. We're trying to take a set of an
[11:04] amorphous set of models, systems,
[11:06] whatever it is, and assign a score to
[11:08] each one of them. But as we talked about
[11:11] like way here uh what is a model? A
[11:14] model is a an interpretation a structure
[11:16] and parameters. It's a collection of
[11:19] some architecture the weights associated
[11:22] with this architecture and some input
[11:23] and output interpretation. So this is
[11:25] the the totality of of a neural network
[11:27] model. Now this is not a set that you
[11:31] can sort of like organize or or it's not
[11:34] an ordered set. The set of all models
[11:35] it's it's it's not an ordered set.
[11:37] There's no natural way of ordering this.
[11:40] Um, but here you're looking for an
[11:42] ordering on that set. Um, but you could
[11:45] have billions of weights. So, how do you
[11:46] know which one is better than the other?
[11:48] And, um, so you're going to have to make
[11:50] a determination of how I'm going to
[11:52] order this. And one way to make that
[11:55] determination is you say, I'm going to
[11:58] order this based on how good the
[12:00] responses are of this particular model
[12:03] on my um I I'll come up with a set of
[12:07] questions and on this set of questions,
[12:09] I'm going to determine how how good the
[12:11] responses are and the one with the best
[12:13] set of responses is uh is the one that
[12:17] I'm going to say is like maybe at the
[12:18] top and you know if and and so on and so
[12:21] forth. So that's one way of ordering it.
[12:24] Now there isn't a particular um way for
[12:28] us to collect all possible inputs that
[12:30] you know could come to this model in in
[12:34] the course of it being used in that
[12:36] particular use case that you're trying
[12:37] to build the eval for. Um so you're
[12:40] going to have to come up with a sample.
[12:42] So what you're really doing is you're
[12:44] saying uh I don't have a way of directly
[12:46] evaluating the model but I can evaluate
[12:48] the responses of the model on questions
[12:50] and then I can come up with some set of
[12:52] questions and because I also don't have
[12:54] a way of knowing all the possible
[12:55] questions I'll just take a small sample
[12:57] of questions. So you keep reducing the
[12:59] problem that you have at hand of
[13:00] ordering that set further and further by
[13:03] sort of like reducing the complexity of
[13:05] the problem. Um
[13:07] the second is once you do have the
[13:09] sample and you're sort of happy with it.
[13:11] So if you somehow were able to rank the
[13:13] models on the quality of responses on
[13:15] these sample questions, um you still
[13:18] need to sort of like figure out how to
[13:21] score or get the responses. You still
[13:23] need to figure out how do I score the
[13:24] responses. So you need to come up with
[13:26] some methodology for ranking them. And
[13:30] once you have these preferences, you
[13:32] need to combine that into some sort of a
[13:33] metric that sort of comes together and
[13:35] says um comes together and says like
[13:38] okay, what's the final score of the
[13:40] single number that I need to be able to
[13:43] get to for the metric. So that's sort of
[13:46] like what I would say is um is uh an
[13:50] eval in a nutshell these three things.
[13:53] So let's talk about how do we go around
[13:55] building these three things.
[13:58] Um so there are different types of evals
[14:01] uh there's human raider uh which is
[14:03] relatively easier to build uh but
[14:07] expensive to run uh which is uh you know
[14:10] you have this sample of questions you
[14:14] generate the prompts or responses or
[14:16] whatever like if you have a task you
[14:18] generate the output of the task um and
[14:21] you have humans take a look at it and
[14:23] you can either rate them or you can
[14:25] compare the task between to two options
[14:28] or two different models or you can
[14:30] compare them to work done by other
[14:32] humans and essentially you try and sort
[14:34] of get a ranking of sorts, right? Like
[14:36] so you have uh AB and then you sort of
[14:39] combine them and you can get a score you
[14:41] can get things like net win rate which
[14:43] is in in you have if you have like two
[14:45] different things uh net win rate would
[14:47] be in this case how many times response
[14:50] from a model was better than a human or
[14:52] a model A was better than model B or
[14:54] something like that. Um and uh so this
[14:58] is a very good way of doing an eval.
[15:00] This allows for direct u comparison
[15:02] between models. Um but there are some
[15:05] challenges uh which should be obvious
[15:07] which is kind of expensive to run them.
[15:09] Um so it's easier to sort of come up
[15:12] with the sample. Once you're once you
[15:13] have a sample you're pretty much good to
[15:16] do a human eval but it is kind of
[15:18] expensive to run these. Um there are
[15:20] other issues with human evals which is
[15:22] there's very a lot of variance. people
[15:24] aren't very consistent. Uh so you don't
[15:27] know unless you have a large sample, you
[15:29] don't know if you're observing sort of
[15:30] just the variance in raiders and you can
[15:33] sort of try and reduce it with multiple
[15:34] raiders and stuff like that. So that's
[15:36] maybe a good way to do it. Uh but that
[15:38] further increases your cost and your
[15:40] time and all of that. Um the second way
[15:44] of doing it is u is an automated eval
[15:46] which is basically uh you come up with
[15:48] some sort of a statistical uh heristic
[15:52] for um understanding how good a model is
[15:55] at something. And so we'll talk about
[15:57] one particular one which is a very
[15:59] common one uh which is uh and we will
[16:02] tie into um what we were doing what we
[16:04] were talking about earlier uh but but
[16:06] where you make the model give you
[16:08] options. So you do multiple choice
[16:09] questions and you you make the model
[16:11] give you options and you sort of try and
[16:13] um get um you know accuracy based on
[16:16] that. It's a very simple way of
[16:17] measuring this. Um
[16:20] this works relatively well for simpler
[16:23] things. So it works relatively well if
[16:25] you are trying to understand how
[16:27] knowledgeable a model is. It also works
[16:30] well if you're trying to do math or or
[16:32] or things like that which have a little
[16:34] bit more of an objective uh response or
[16:38] answer to them. Um you can do some
[16:41] subjective things with automated evals
[16:43] like fully automated evals uh with like
[16:46] sort of herist statistical heristics. Um
[16:49] but uh it's it's by and large limited.
[16:52] The upside of this is and this this is
[16:55] maybe from a volume standpoint point in
[16:58] terms of like during model building the
[17:00] the first row the automated evals is the
[17:03] highest volume of evals like maybe a
[17:05] research team would run uh during the
[17:07] phase of building because early on the
[17:09] models are not very smart when you're
[17:11] starting training them and because
[17:13] they're not very smart you start to you
[17:15] start with automated evals you make sure
[17:17] you sort of get somewhere first before
[17:19] you start to do any anything else. Um um
[17:22] so in terms of compute uh and the number
[17:25] that these eval maybe constitute like 90
[17:28] plus% of all like 95 plus percent of all
[17:31] the eval
[17:32] um the third category is AI judge evals
[17:36] um which is becoming more and more
[17:37] popular these days and they are more
[17:40] like human evals in the sense that
[17:42] you're essentially doing what you would
[17:44] be doing in the last row um but you're
[17:47] replacing your uh person or human with
[17:49] another AI model uh to do that. Now,
[17:53] this introduces its own issues. Um
[17:56] models can have biases. Models are not
[17:58] necessarily
[18:00] um you especially if you're using the
[18:02] same model that you're trying to train
[18:03] or you're trying to sort of understand.
[18:05] Uh this can create a bit of a problem
[18:06] for you because you might actually
[18:08] reinforce the bias that the model has
[18:10] with the eval. You'll never find out
[18:12] what's happening. So, you want to try
[18:13] and maybe at some point use a different
[18:15] model for this, which is kind of the
[18:17] standard practice. you don't want to use
[18:19] the model you're trying to iterate on or
[18:20] train for for that purpose. Uh but by
[18:23] and large this is maybe the area of most
[18:26] active research. Uh right now it's kind
[18:29] of exploding and and we'll talk about
[18:30] like maybe one or two other ways of
[18:32] doing these. Um but but um judge eval
[18:35] are becoming more and more popular these
[18:37] days.
[18:39] Um this is an example of an auto eval.
[18:42] Um so the first type of eval that we
[18:44] just talked about here. Um so if you
[18:47] have a if you have an input prompt um
[18:49] that you give to a model and and this is
[18:51] a this is particularly an example from
[18:53] the MMLG data set. Um and what what is
[18:57] happening here is you're asking the
[18:58] model uh you know a question and then
[19:01] you're giving it um four options and the
[19:05] way you score the answers is you don't
[19:07] actually ask the model to give you the
[19:09] response. What you do is instead um you
[19:12] know we talked about this last layer of
[19:14] in of of the models where you have like
[19:16] these 40,000 neurons that you generated
[19:19] and that is how you figure out what the
[19:22] next token is. Uh what you do is you
[19:24] just look at those 40,000 neurons and um
[19:27] you write the question and you make it
[19:29] generate um those neurons for what would
[19:32] have been the answer A B C D and you
[19:34] look at the probability of each one of
[19:36] those and whichever one is highest. So
[19:39] essentially what you're doing is you're
[19:40] restricting the vocabulary of the the
[19:42] final output of that particular one just
[19:45] to four particular um tokens and and
[19:49] you're just looking at the uh negative
[19:51] log likelihood of that those particular
[19:53] four tokens. Um which means the model
[19:56] never really generated an answer but if
[19:58] it had generated an answer hopefully it
[20:00] would have been one of these uh you know
[20:02] it would have been that the probability
[20:03] that was highest. Um so this allows you
[20:07] to very easily calculate the the log
[20:08] likelihood of the full full and you can
[20:10] do this for more longer more complex um
[20:14] generations as well. So it doesn't have
[20:15] to be just ABCD. You can actually look
[20:17] at the log likelihood of the entire
[20:18] sequence of things. Very common
[20:20] technique uh where you just look at the
[20:23] last sort of layer and you sort of keep
[20:26] auto reggressively generating more and
[20:27] more and and you get to um you get to a
[20:30] number. So that is your number accuracy.
[20:33] how many times did the model answer
[20:35] correctly?
[20:37] Um, so coming back to like sort of the
[20:39] AI judge, as I was saying, it's an area
[20:42] of active research. Um, frontier models
[20:45] are becoming more popular, powerful, so
[20:48] you can use them as judges more
[20:49] effectively. Um,
[20:52] what is catching on more is you can
[20:54] write more detailed rubrics. So you can
[20:55] enforce some sort of a value system of
[20:57] your own using these judge evals uh on
[21:01] the question. So you you take the
[21:02] answer, you take the response and you
[21:04] sort of start to say like hey is does
[21:08] the response contain these criteria or
[21:10] characteristics? Does the respond
[21:11] contain this other characteristics and
[21:13] so on and so forth.
[21:15] Um
[21:17] so um good evals can be hard. Uh they
[21:20] don't have to be but there are some
[21:22] gotchas and challenges in running evals.
[21:24] Um this slide is mostly just to uh make
[21:27] sure people don't worry too much about
[21:29] the small variations in eval like small
[21:32] variations in eval can be caused by a
[21:34] lot of different reasons uh depending on
[21:37] how you do pipeline parallel uh uh
[21:40] different results can be had um you can
[21:43] do uh generations or you can do choice
[21:45] evals which is what we just looked at
[21:47] earlier uh and you might get like sort
[21:49] of some differences in results um how
[21:52] you parse the the response responses can
[21:54] cause differences uh even so much so
[21:57] that depending on the architecture that
[21:59] you're using and the batch sizes uh you
[22:01] know how the reduce operations are
[22:03] happening within your tensor course can
[22:05] affect the result of your eval um so
[22:08] generally don't obsess over like a 1% 2%
[22:11] whatever changes or differences in eval
[22:13] because like there's a lot of factors if
[22:15] you're iterating on models make sure
[22:17] that you're like sort of not reading the
[22:19] tea leaves where there's nothing to do
[22:20] to to be read
[22:23] Um,
[22:24] all right. So, what could eval look like
[22:27] for some real world examples? Um, so
[22:30] here, like imagine you're trying to do
[22:32] one of these things. You're trying to
[22:33] maybe build a a a customer service bot
[22:37] or something like that. And you want to
[22:38] sort of like build an eval um, which is
[22:41] sort of the point of this talk is just
[22:42] to encourage people to first worry about
[22:44] building evals before you worry about a
[22:48] lot of other things like um, and invest
[22:50] up front. Make sure you have a good eval
[22:52] built. Um and and that way you're not
[22:54] going to have surprises uh when you
[22:56] actually end up um implementing it and
[23:00] it will perform the way that you sort of
[23:02] expected it based on the eval. Um so
[23:05] what you want to do for this is maybe
[23:06] take some real world conversations maybe
[23:08] 500 or so conversations enough that you
[23:10] get like a tight enough confidence
[23:12] interval and write really good responses
[23:16] to these conversations like have humans
[23:18] review them. make sure that these
[23:20] conversations represent like some really
[23:23] amazing and and as complex and thorny as
[23:25] they need to be for your use cases. And
[23:29] now you want to have the model or
[23:31] whatever your system it is that you're
[23:33] trying to deploy answer or respond to
[23:36] these conversations and use one of the
[23:38] techniques that we talked about earlier
[23:40] either automated or a judgebased eval to
[23:42] rank them. So you can get a score for
[23:45] how well humans are doing on on this. So
[23:47] like how well are your um customer
[23:50] service rep human customer service reps
[23:52] are doing compared to maybe the golden
[23:54] responses that you've written? How well
[23:56] is your AI system doing compared to the
[23:58] golden responses you've written? And
[24:00] that gives you a score. You can you can
[24:02] have an absolute score out of 100 or you
[24:04] can build a score that just gives you a
[24:06] win rate against the the best golden
[24:08] responses.
[24:10] And what you want to do is you want to
[24:11] automate this process and you want to do
[24:13] it up front. So what what that means is
[24:16] once you have this sort of like eval
[24:18] built out up front, it sets you up to
[24:20] iterate on your AI system, it allows you
[24:22] to then say, "Okay, now I'm going to um
[24:26] continue to iterate and improve on my AI
[24:28] system until I hit a certain number that
[24:30] is acceptable to me for my business for
[24:32] implementation." Um so that's sort of
[24:34] like what you ideally would um you know
[24:37] would enter into like before you get
[24:39] before you start thinking about a
[24:40] deployment and stuff. Um
[24:44] the other good thing about it is as
[24:45] you're iterating um on these things, it
[24:48] could allow you to actually switch out
[24:50] solutions, switch out models much more
[24:52] faster with more confidence even once
[24:54] you've entered production. So like
[24:55] you've gone into production and you have
[24:56] an EBA and you're evaluating, okay,
[24:59] maybe I want to move to a cheaper model
[25:00] or maybe I want to move to some
[25:01] something else or like a new uh new new
[25:04] provider or whatever it is. Um you can
[25:07] be reasonably sure that the quality of
[25:09] responses is not going down. Now things
[25:11] may still go wrong other other in other
[25:14] ways you know as as we talked about
[25:16] integration or whatever other reasons
[25:17] but at least you want to make sure that
[25:18] the core modeling part of it is is not
[25:21] the reason that you're you're facing
[25:22] those failures.
[25:25] Okay. So so the key things here are
[25:28] basically you know um language models
[25:31] have a non-convex um intelligence
[25:33] envelope. So, you want to first worry
[25:36] about building evals before you do
[25:38] anything else. Um, uh, or or maybe one
[25:41] of the first few things. Um, and they're
[25:44] critical to knowing um, you know, how
[25:46] the how the model or or the system is
[25:49] good at the task that you're trying to
[25:50] get it to do. Um, building them is not
[25:53] hard. Like we talked about how to do it.
[25:55] It's a very simple process. You find
[25:57] prompts, you write golden responses, you
[25:59] find, you know, you know, and you come
[26:00] up with some, you know, you can even
[26:02] have a judges. Maybe the simplest,
[26:04] fastest way these days, use a frontier
[26:06] model, have a judge look at them,
[26:08] compare them, tell you which one is
[26:09] better. Um um and eval up front. Make
[26:14] sure the eval associates with how you
[26:16] perceive quality. By the way, like once
[26:17] you have built it, look at the scores.
[26:20] Make sure the models that are doing well
[26:22] are actually you feel like they're
[26:24] actually better at the task that you're
[26:26] trying to get them to do. That's when
[26:27] you know like that's when you actually
[26:29] looked at the data yourself and you feel
[26:31] good about the eval you built. Um but
[26:34] after you've built the eval um you can
[26:37] sort of go and do all these other things
[26:39] um that you need to do and and you can
[26:41] be reasonably sure that okay like at
[26:42] least from a technical model standpoint
[26:45] um there isn't like sort of a failure
[26:47] mode that you're going to a catastrophic
[26:49] failure mode that you're going to enter
[26:51] um that will be unseen to you, right? Um
[26:55] and um yeah, that's pretty much it. That
[26:56] was like hopefully the I think the whole
[26:59] point of this talk. So we'll do
[27:00] questions if anybody has questions now.
[27:02] Yeah. Um so the question is there's just
[27:04] a lot of academic benchmarks in the
[27:06] community right now and um and are there
[27:09] are there benchmarks that we wish that
[27:11] existed that don't exist? Um yeah I'll
[27:15] say absolutely. Uh I think there's just
[27:18] a firstly there is a lot of really good
[27:20] benchmarks that you you know are already
[27:23] out there. Um my my view on those is
[27:26] it's not necessarily the case that that
[27:28] the benchmarks are are not good. It's
[27:30] it's that people sometimes don't
[27:32] effectively try and use them that well.
[27:34] Um in terms of what I wish existed more
[27:37] more is uh a lot of the benchmarks today
[27:41] are focused on certain tasks. Um and and
[27:44] I think there's this wave now of like uh
[27:48] uh the eval that are like more they they
[27:52] are a combination of a lot of tasks. So
[27:54] I I'm like hopeful that that will give
[27:56] us like so that that's what one of the
[27:58] things that I wish existed is like
[28:00] combination evals that contain a lot of
[28:01] tasks.
[28:02] >> What is that one eval which you would
[28:05] definitely want to test out before going
[28:07] into production?
[28:09] >> Yeah. Um I don't know if irrespective of
[28:12] the use case there exists an eval that
[28:14] like I would think that all models need
[28:16] to be good at I would think some sort of
[28:17] a basic reasoning eval is is a good like
[28:21] something that I would think like every
[28:23] model should do some somewhat reasonably
[28:25] well if it can uh uh if it's going to
[28:28] have but really if you're doing like an
[28:30] arts like image generation is your use
[28:33] case right like then maybe a reasoning
[28:34] eval doesn't really matter that much um
[28:37] or maybe it does matter because it has
[28:39] to reason through what you asked about
[28:40] the image. Um but either way um you know
[28:44] I think the one of the points here is
[28:47] actually that the that it should be
[28:49] specific to um to the use case that that
[28:53] you should build an eval specific to the
[28:55] use case that
[28:56] >> so basically you said uh we should uh
[28:59] build eval first. So should we also uh
[29:02] like iterate our evals over time when we
[29:05] are uh building that AI application?
[29:08] Yeah. Uh that's an excellent question. I
[29:10] think um all eval should evolve over
[29:13] time. So what is an eval? It's just a
[29:16] collection of sample of prompts, right?
[29:18] Like so as you're going through and
[29:19] you're in deployment and you see these
[29:21] thorny situations come in and you hadn't
[29:23] really thought of this before and you
[29:24] have some failure mode. I would take
[29:27] those samples. I would fix the model for
[29:28] those sample but then I would take those
[29:30] samples and put it in the eval some of
[29:31] them. And that way you know that in the
[29:34] future your eval sort of covers those
[29:36] use cases. uh there's no way up front
[29:38] you're going to know everything that is
[29:40] that is going to hit you in the real
[29:41] world. Um so you just continuously adapt
[29:44] your eval to increase coverage.
[29:47] >> So basically it's kind of unit test like
[29:49] for a simple application, right?
[29:52] >> Yeah.
[29:53] >> Basically it acts a unit test that we
[29:55] make for noni application.
[29:57] >> Yeah. I mean I I think more so than unit
[29:59] test like you eval itself is in in a way
[30:02] an integration test but yeah you're
[30:04] evolving the test to add more use cases
[30:06] in it. Yeah,
[30:06] >> most of the conversations that we having
[30:08] the these models are multi-turn
[30:09] conversations. So when you think about
[30:11] goldens, goldens in isolation without
[30:14] the context of previous turns is is a
[30:17] very hard problem to evaluate. So how do
[30:20] you think about context in the world of
[30:23] eval?
[30:25] >> Yeah, that's this is a very very good uh
[30:28] question and a very thorny problem. I
[30:30] don't think people have solved this yet.
[30:31] But the here's the problem that you're
[30:33] describing and then we'll talk a little
[30:34] bit about what potential solutions are.
[30:36] The problem is in multi-turn
[30:38] conversations I can't really have like a
[30:40] predetermined set of questions. Um I can
[30:43] even have multi multi-turn conversation
[30:45] and ask the model for a question. But
[30:47] what happens in reality is uh models are
[30:50] good at single turn answers even if it
[30:53] looks like it's an answer to a
[30:55] multi-turn conversation but they tend to
[30:57] drift when they're the ones providing
[30:59] all the turns. So it's not the same
[31:02] thing as I give you 10 turns and I I ask
[31:05] the 11th turn and I eval that 11th turn
[31:07] response. It's not the same thing as I
[31:10] make you generate all the 10 turns. I'm
[31:12] going to see start seeing some drift. So
[31:14] how do you solve this problem? Um and
[31:16] there are a couple of evals that address
[31:18] this uh try and address this as at least
[31:21] and the way they work is you essentially
[31:23] have another model simulate the other
[31:25] side of the conversation. you have a
[31:27] fixed model and you actually have the
[31:29] models um do the entire conversation and
[31:32] the models sort of have agendas and they
[31:34] have private information and stuff like
[31:36] that and you um evaluate the entire
[31:41] conversation on certain criteria uh and
[31:44] whether these checkpoints are head or
[31:46] not and and whether the models the
[31:48] primary model that you were evaluating
[31:49] made certain mistakes or not. This
[31:51] creates a blame assignment problem
[31:53] because the other model may have made a
[31:55] mistake in some turn which you're now
[31:56] assigning the blame to the primary
[31:58] model. Um so what you can do is in these
[32:01] situations you can try and sort of have
[32:03] the the primary model do both both
[32:05] agents. So you can you can have the
[32:06] model pretend to be two agents with two
[32:08] agendas and actually have play out a
[32:10] conversation and then you eliminate the
[32:12] blame assignment problem. Um but again
[32:15] it's it's not by any means a solved
[32:17] issue. This is a very problematic issue
[32:20] and there is just like if there are
[32:22] ideas on how to solve this, I'm all
[32:24] yours. Yeah.
[32:25] >> Um I have I have a question, but first
[32:28] can I piggyback off of that question
[32:30] really briefly? Um one idea with just
[32:34] the two models and the multi-turn
[32:37] kind of context window collapsing issue.
[32:40] I'm just wondering if you could make
[32:42] maybe the second model like a rigid like
[32:44] ruda type of rulebased model so that
[32:47] essentially it has very predictable
[32:49] outputs that then you're pitting the
[32:50] first model up against and then you're
[32:52] kind of like reinforcing it against this
[32:54] one that's essentially trained to always
[32:57] give these types of responses. So it's
[32:59] doing sort of the the input prompt
[33:01] comparison to the first model over time
[33:04] and then it's just an idea I wanted to
[33:06] piggyback that was a really good
[33:07] question. Um my actual question is when
[33:10] there's no like ground truth for the
[33:13] human version of whatever the the output
[33:16] is. So for example, one of the projects
[33:18] I'm working on, there can be kind of a
[33:21] an emotional or a sentimental veilance
[33:24] that we're trying to parse out of what
[33:25] we would get from a human respondent.
[33:27] And then when we're swapping in certain
[33:29] agent or agentic systems or models to
[33:32] try to replicate that or at least kind
[33:34] of approximate it in some way at least
[33:36] in those like veilance signals. We're
[33:39] wondering if one there's eval that can
[33:42] be borrowed from like other frameworks
[33:45] like you know how quantsych has a lot of
[33:47] frameworks for determining you know what
[33:49] this means what that means sem for
[33:51] measuring like latent variables when
[33:54] humans are responding to surveys and
[33:55] things can something like that be used
[33:58] to see like how humanlike your responses
[34:01] are um to kind of fine-tune the rubric
[34:04] or are there other frameworks that work
[34:06] for trying to get a human-like response
[34:09] that doesn't have a black or white right
[34:11] or wrong answer.
[34:13] >> Yeah, I mean I think the best ideas so
[34:15] far on on that are obviously training a
[34:17] reward model, right? Like uh which is
[34:19] something you take human preference data
[34:21] and you try and train a reward model
[34:22] that is able to actually try and capture
[34:24] those human preferences. Um it might
[34:27] feel like a little bit of an overkill to
[34:29] do like train an entire reward model
[34:30] just for an eval. Uh but it does pay
[34:33] off. So it is a worthwhile exercise if
[34:35] you have human preference data. Um, you
[34:38] can reduce the scope of of the model
[34:42] preference selection to just the
[34:44] questions of the eval. So, um, what I
[34:47] mean by that is, um, when you're trying
[34:50] to train a reward model for human
[34:52] preferences, you have to be very careful
[34:53] about, uh, not not overfitting it. You
[34:56] don't want it to overfit on that
[34:58] particular data set. Uh, you want it to
[35:00] generally try and capture pre human
[35:03] preferences. when you're doing it for uh
[35:05] an eval, you don't really care. Uh
[35:08] you're only ever going to evaluate on
[35:09] that set. As long as you're extremely
[35:11] extremely careful that you're actually
[35:12] actual model training data set does not
[35:14] contain the eval data set, your reward
[35:16] model being overfitit on the reward uh
[35:18] on on the eval set is perfectly fine. Uh
[35:21] and and that way you can get a much
[35:23] higher accuracy of what human preferred
[35:25] responses are on that particular those
[35:27] thousand questions. um as long as again
[35:30] those thousand questions are not in your
[35:32] um in your model training data set the
[35:34] main model training data set. So
[35:36] piggybacking actually kind of on that so
[35:38] perfectly.
[35:40] How do you think about trying to
[35:42] mitigate reward hacking? So with the
[35:44] sort of challenges that don't have a
[35:46] nebulous yes no answer. You can try to
[35:49] train a reward model or you can use
[35:52] reward or reinforcement learning from AI
[35:54] feedback. But we found that models are
[35:57] very very good at hacking the reward
[35:59] model and getting uh absolute garbage
[36:01] inputs that get really high rewards. How
[36:03] do you try to mitigate that?
[36:05] Yeah, I mean that goes into like sort of
[36:07] like model training itself and um I
[36:10] don't know that the solution necessarily
[36:12] lies in eval particularly. I think um
[36:16] evals by the nature of like they're
[36:18] they're meant to be run fast and and
[36:19] they're not going to have coverage on
[36:21] every possible thing. Um so I think what
[36:25] really what you want to do is you want
[36:26] to make sure that as you're training the
[36:28] reward models that you're not
[36:29] overtraining them. Um there are
[36:32] potentially like um optimization
[36:34] approaches where uh one of the things
[36:37] like I was writing about um like maybe
[36:40] yesterday or or something was was an
[36:42] approach where you can actually have um
[36:45] um you can avoid training reward models
[36:47] and you can actually have rubrics and
[36:49] and judges and scoring using other
[36:51] models. uh where you can just use the
[36:53] reward uh you can create a reward
[36:55] directly from the scores and you can do
[36:58] a you can create a relative advantage
[37:00] function on these rewards um um without
[37:04] actually training a reward model which
[37:06] might allow you more flexibility in
[37:08] setting up new rubrics and then being
[37:10] able to sort of like just cover whatever
[37:12] it is that you're seeing. Um but yeah, I
[37:15] mean there is no real antidote to the
[37:17] real world. Like you just kind of have
[37:19] to observe the failure modes and you're
[37:20] like, "Okay, now I'm going to capture
[37:21] these in in and so whatever framework
[37:24] you you build, if it is training a
[37:25] reward model, you have to throw more
[37:26] data into the reward model. If you're
[37:28] doing this other approach that I was
[37:30] talking about with rubrics, you're
[37:31] you're actually just building new
[37:32] rubrics. So you want to try and build a
[37:34] a framework that that calculates your
[37:36] advantage function in a flexible way
[37:38] that you can actually modify your reward
[37:40] function and advantage function.
