LangChain Advisers Shankar and Husain Link LLM Failure to Data Neglect

Getting Evals Right for LLM Applications | Interrupt 26 — LangChain

LangChain

Building reliable AI agents requires treating LLM judges as classic, imbalanced machine learning classifiers rather than trusting unvalidated, arbitrary scoring rubrics.

AI-assisted summary of Getting Evals Right for LLM Applications | Interrupt 26 — original source. Check the original for full context.

Source published:

Key ideas

  • Generic LLM metrics like 'helpfulness' or 'hallucination' are too ambiguous and context-dependent; teams must replace them with domain-specific failure categories derived directly from trace analysis. (3:28)
  • LLM judges should be treated as imperfect classifiers on imbalanced datasets, requiring train/dev/test splits and evaluation via precision and recall rather than raw accuracy. (7:03)
  • Continuous scoring scales such as 1-to-5 or 1-to-100 create uninterpretable metrics and should be converted into targeted binary pass/fail checks. (10:13)
  • Predefined evaluation rubrics routinely suffer from criteria drift because human reviewers only discover true edge cases and failure modes when inspecting actual production outputs. (12:00)

Detailed notes

## Background Published on YouTube by LangChain on June 12, 2026, from the Interrupt agent conference, this presentation features Shreya Shankar and Hamel Husain. Drawing from their experience teaching evals to over 4,500 practitioners across dozens of companies, the talk outlines five prevalent mistakes in evaluating LLM applications and demonstrates how adopting a data scientist's mindset resolves them.

## TL;DR Husain and Shankar argue that AI engineering has regressed into unrigorous, 'vibes-based' evaluation and that building effective agent harnesses fundamentally requires classic data science disciplines. They deconstruct five widespread industry traps: relying on ambiguous off-the-shelf metrics, uncritically trusting unvalidated LLM judges, poor experimental design with naive synthetic data and continuous scoring scales, delegating labeling to non-experts while ignoring criteria drift, and attempting to fully automate evaluations. To fix these issues, they advocate for hands-on exploratory data analysis using custom trace interfaces, reframing LLM judges as imperfect binary classifiers with train/dev/test splits, generating synthetic scenarios across explicit combinatorial dimensions, and continually inspecting raw data.

## Atmosphere & style The session is lively, candid, and direct, characterized by the natural back-and-forth rapport between Husain and Shankar as they alternate between diagnosing industry bad habits and prescribing practical engineering solutions. Infused with wry humor, relatable memes, and sharp critiques of shortcuts like 1-to-100 scales and blind prompting, the talk feels like a field-tested wake-up call for AI practitioners. Rather than abstract academic theory, it carries the urgent, grounded energy of seasoned mentors urging builders to drop the hype and return to empirical fundamentals.

## Bulletin - [03:28](https://www.youtube.com/watch?v=QDQT99csHJQ&t=208s) Generic LLM metrics like 'helpfulness' or 'hallucination' are too ambiguous and context-dependent; teams must replace them with domain-specific failure categories derived directly from trace analysis. - [07:03](https://www.youtube.com/watch?v=QDQT99csHJQ&t=423s) LLM judges should be treated as imperfect classifiers on imbalanced datasets, requiring train/dev/test splits and evaluation via precision and recall rather than raw accuracy. - [10:13](https://www.youtube.com/watch?v=QDQT99csHJQ&t=613s) Continuous scoring scales such as 1-to-5 or 1-to-100 create uninterpretable metrics and should be converted into targeted binary pass/fail checks. - [12:00](https://www.youtube.com/watch?v=QDQT99csHJQ&t=720s) Predefined evaluation rubrics routinely suffer from criteria drift because human reviewers only discover true edge cases and failure modes when inspecting actual production outputs.

## Key takeaways - [00:06](https://www.youtube.com/watch?v=QDQT99csHJQ&t=6s) The evaluation harness popularized by modern agent architectures is fundamentally grounded in traditional data science practices, encompassing logs, metrics, and traces. - [01:27](https://www.youtube.com/watch?v=QDQT99csHJQ&t=87s) Modern AI engineering has regressed toward intuition and vibes-based evaluation compared to the rigorous data science and machine learning practices of four years ago. - [03:28](https://www.youtube.com/watch?v=QDQT99csHJQ&t=208s) Off-the-shelf, generic evaluation metrics fail because abstract concepts like hallucination require completely different operational definitions depending on the product domain. - [04:47](https://www.youtube.com/watch?v=QDQT99csHJQ&t=287s) Building lightweight custom UIs using AI coding tools enables practitioners and product managers to step through traces and uncover concrete application failure modes. - [06:04](https://www.youtube.com/watch?v=QDQT99csHJQ&t=364s) Deploying LLM judges without validating them against annotated ground truth leads to arbitrary Likert-scale distributions that cannot support product decisions. - [08:09](https://www.youtube.com/watch?v=QDQT99csHJQ&t=489s) LLM judging is an imbalanced classification problem where measuring raw accuracy is misleading because real failures represent only a small fraction of traces. - [08:26](https://www.youtube.com/watch?v=QDQT99csHJQ&t=506s) Synthetic data generation must be structured around hypothesized dimensions of variation and combinatorial cross-products rather than open-ended prompts. - [10:13](https://www.youtube.com/watch?v=QDQT99csHJQ&t=613s) Replacing continuous scoring scales (1-to-100) with binary classification makes evaluation metrics statistically testable and actionable for debugging. - [11:27](https://www.youtube.com/watch?v=QDQT99csHJQ&t=687s) Delegating trace labeling to general developers or outsourced workers fails because they lack the domain expertise required to judge nuanced product behavior. - [12:00](https://www.youtube.com/watch?v=QDQT99csHJQ&t=720s) Criteria drift makes upfront rubric design ineffective; evaluation standards inevitably shift as reviewers encounter unexpected real-world production cases. - [13:03](https://www.youtube.com/watch?v=QDQT99csHJQ&t=783s) Automating the entire evaluation loop with LLMs misses critical failures because models lack the tacit institutional knowledge and user expectations that human teams hold. - [13:45](https://www.youtube.com/watch?v=QDQT99csHJQ&t=825s) Tuning LLM judge prompts repeatedly against a single static dataset leads to prompt-level overfitting, requiring held-out validation sets to confirm generalization. - [15:34](https://www.youtube.com/watch?v=QDQT99csHJQ&t=934s) Every modern LLM evaluation challenge maps to established data science disciplines, including exploratory data analysis, metric design, and model validation.

## Detailed walkthrough ### [00:06](https://www.youtube.com/watch?v=QDQT99csHJQ&t=6s) The Return of the Data Scientist and Harness Engineering

Hamel Husain opens the session by welcoming the audience and declaring the theme of the talk: "the return of the data scientist." Setting the stage, he asks the audience for a show of hands regarding who has seen the recent article discussing "harness engineering." About half the audience raises their hands, prompting Husain to recommend it to anyone who has not read it yet.

Husain explains the premise of the article: a team at OpenAI ran autonomous agents over several months to construct substantial internal software. Crucial to their success was an evaluation "harness" designed to keep the agents on track. Husain points out a frequent blind spot in how software practitioners interpret this concept: > “And so when people hear harness, they usually think about specifications and unit tests, but one detail that's really important to look at is, okay, the harness also contains logs, metrics, and traces.”

This encompasses the complete observability stack. Husain states his core thesis directly: > “And what I'm here to convince you of today is that the harness is data science, or at least a big part of it is data science.”

```mermaid flowchart TD accTitle: The Harness Engineering Observability Stack accDescr: Diagram illustrating how logs, metrics, and traces feed into data science practices. subgraph S1["Traditional View"] A["Specifications"] B["Unit Tests"] end subgraph S2["Observability Stack"] C["Logs"] D["Metrics"] E["Traces"] end subgraph S3["The Reality"] F["Data Science Foundation"] end S1 --> F S2 --> F ```

[01:27](https://www.youtube.com/watch?v=QDQT99csHJQ&t=87s) ML Engineering Four Years Ago vs. Vibes-Based AI Engineering Today

Taking a step back, Husain reflects on how the discipline arrived at its current predicament by comparing traditional machine learning engineering from four years ago with modern AI engineering practices. Four years ago, ML practitioners meticulously analyzed their data, produced visualizations, diligently aligned model predictions with human labels, selected the precise tool for each specific problem, and took great care to craft metrics tightly connected to actual business objectives.

In contrast, Husain observes that current AI engineering has regressed in critical ways, relying on intuition and shortcuts rather than rigorous evaluation: > “AI engineering today feels like we've taken a bit of a step back in some places. So now, a lot of times, we're just using vibes to know if we're doing the right thing.”

Engineers often resort to asking a model—sometimes an external model, sometimes the very same model that generated the text—whether it did a good job. Furthermore, metric design has been hollowed out; teams routinely instruct an LLM to assign an arbitrary score from 1 to 100 or blindly import off-the-shelf metrics packages without understanding their mechanics. Husain diagnoses the underlying cause: engineers frequently fear what they do not understand, causing core data science competencies—specifically evaluations and retrieval—to be repeatedly dismissed and revived. Having taught evals to more than 4,500 people across dozens of companies, Husain notes that organizations consistently fall into identical traps, before passing the presentation to Shreya Shankar to detail the specific failure modes.

[03:28](https://www.youtube.com/watch?v=QDQT99csHJQ&t=208s) Mistake 1: Relying on Generic, Off-the-Shelf Metrics

Shreya Shankar takes over to examine five critical mistakes practitioners make when building evals, framing the fixes through the lens of a data scientist. The first major pitfall is reaching for off-the-shelf, generic metrics to gauge agent accuracy or alignment: > “So the first one is kind of using metrics, what we call generic metrics or off-the-shelf metrics to measure the accuracy or alignment of your agents.”

While using generic operational metrics makes sense for traditional software—where measures like latency and uptime apply universally across applications—AI applications demand a bespoke approach. Off-the-shelf AI metrics typically revolve around abstract concepts such as "helpfulness," "coherence," or "hallucination." While these sound critical, Shankar points out that they suffer from two major flaws: 1. They are fundamentally ambiguous, making it near impossible to define what they objectively mean in practice. 2. They are deeply context-dependent. A hallucination in a healthcare application built by Husain requires an entirely different operational definition, different tools, and different data than a hallucination in a legal application built by Shankar. Evaluating both using a single off-the-shelf metric is ineffective.

[04:47](https://www.youtube.com/watch?v=QDQT99csHJQ&t=287s) Fixing Mistake 1: Exploratory Data Analysis and Custom Trace Interfaces

To break away from generic metrics, Shankar urges teams to act like data scientists by performing exploratory data analysis (EDA): directly inspecting the data, determining where the system breaks, and assigning concrete names to application-specific failure modes.

Rather than relying on static logs, Shankar highlights the value of using AI-assisted coding tools like Cursor, Claude Code, or Codex to quickly build custom user interfaces for inspecting traces. Practitioners can step through an agent's trace message by message, roleplaying as the end user, and work alongside product managers to identify where execution goes off the rails. Teams can record freeform open notes about mistakes, and as these notes accumulate, cluster them into bespoke failure categories.

Shankar illustrates this with an example from an automated real estate scheduling assistant: rather than tracking an abstract "hallucination" score, the team identified a precise, domain-specific failure mode—the agent was actively rescheduling tours when no open slots existed by hallucinating available appointment times. Shankar reinforces their core motto: > “Hamel and I always say, look at your data. And this is exactly what we mean by look at your data. It's not just, oh, look at it and don't say or think or do anything about it. It's build your own interfaces and try to find failure modes and try to do this at scale.”

```mermaid flowchart TD accTitle: Custom Interface Workflow for Failure Analysis accDescr: Step-by-step process of using custom tools to identify and categorize domain failure modes. A["Agent Execution Traces"] --> B["Build Custom UI<br/>(Cursor / Claude Code / Codex)"] B --> C["Read Messages Step-by-Step<br/>with PMs / Domain Experts"] C --> D["Write Open Exploratory Notes"] D --> E["Cluster into Bespoke Failure Modes<br/>(e.g., Fabricating Real Estate Tour Times)"] ```

[06:04](https://www.youtube.com/watch?v=QDQT99csHJQ&t=364s) Mistake 2: Blindly Trusting LLM Judges

The second widespread failure mode is using an LLM judge without verifying whether the judge itself is accurate or reliable. Shankar clarifies her stance on model-based judging: > “So I'm not saying that using LLM judges are bad. I'm saying that using LLM judges kind of blindly without validating your validators is bad.”

In standard industry practice, a team discovers a failure mode and immediately prompts an LLM to evaluate traces on a 1-to-5 scale. They plot a histogram of the resulting scores and attempt to make executive product and business decisions based on the distribution. Shankar warns that these arbitrary Likert-scale outputs are neither trustworthy nor translatable into actionable product insights.

[07:03](https://www.youtube.com/watch?v=QDQT99csHJQ&t=423s) Fixing Mistake 2: Treating LLM Judges as Imperfect ML Classifiers

A data scientist resolves this by framing the LLM judge as an imperfect machine learning classifier. Rather than relying on unchecked prompt outputs, practitioners should benchmark and develop the judge with the same rigor applied to classical machine learning: - Assemble an annotated dataset of traces containing confirmed examples of the specific failure mode. - Partition the labeled data into strict training, development (dev), and test splits. - Iterate on the judge prompt and model choice against the train and dev splits, validating that performance generalizes cleanly to the held-out test split without overfitting alignment.

Shankar emphasizes that LLM evaluation is fundamentally an imbalanced classification problem: > “And in many classification tasks, interestingly, LLM judging is often an imbalanced classification task, which is a fancy way, it's a lot of jargon for saying that only a small fraction of the trace set is a failure or a failure mode.”

Because failures represent only a tiny fraction of total traces, reporting raw accuracy is misleading—a judge that marks every trace as a success could trivially show 98% accuracy while missing every real failure. Consequently, teams must measure alignment using imbalanced classification metrics: - Precision - Recall - False positive rates - False negative rates

[08:26](https://www.youtube.com/watch?v=QDQT99csHJQ&t=506s) Mistake 3: Poor Experimental Design and Naive Synthetic Data

The third pitfall centers on flawed experimental design, particularly around synthetic data generation and scoring scale design. While synthetic data generation is valuable, naive implementations produce uniform, low-utility data. When teams ask an LLM with an unconstrained prompt like *"give me five generic questions"* to test a document question-answering bot, the generated traces collapse into homogeneous patterns lacking real-world variation.

To fix synthetic data generation, Shankar advises treating it as a structured experimental process where LLMs perform only constrained sub-tasks: 1. **Hypothesize dimensions of variation:** Inspect genuine application traces to determine how real user inputs vary across at least three distinct dimensions (such as user persona, ranging from complete novices to power users). 2. **Generate values per dimension:** Use LLMs to generate realistic, diverse variations across each individual dimension independently. 3. **Take the combinatorial cross-product:** Combine the generated dimensions across one another to construct diverse evaluation scenarios. 4. **Audit for quality:** Manually review the resulting synthetic suite to confirm structural diversity and data fidelity before deploying it into evals.

[10:13](https://www.youtube.com/watch?v=QDQT99csHJQ&t=613s) Fixing Metric Design: Replacing Continuous Scales with Binary Classifications

Turning to the metric design aspect of experimental design, Shankar explains that scoring traces across wide spectrums like 1-to-5 or 1-to-100 creates uninterpretable numbers that cannot guide engineering decisions. When a system scores an 82, teams cannot identify what broke or how to patch the underlying prompt or pipeline.

The data science solution is radical simplification: > “We'll try to make things a binary classification problem. We're trying to ask the judge for whether something fails or not, a binary classification problem, rather than one to 100 or one to five scoring for something like failure.”

By converting evaluations into targeted binary checks—determining definitively whether a specific, concrete failure mode occurred or did not occur—teams make metrics immediately interpretable, mathematically testable via classification statistics, and actionable for driving product iterations.

[10:55](https://www.youtube.com/watch?v=QDQT99csHJQ&t=655s) Aligning LLM Judges on Binary Tasks

Shreya Shankar emphasizes that aligning an LLM judge is an iterative process that cannot be solved in a single prompt attempt. Engineering teams constantly struggle with this step, and Shankar admits that even experienced practitioners run into the same barrier immediately: “Like we can never write a good LLM judge prompt on the first time.” Instead of expecting an out-of-the-box prompt to evaluate nuanced output accurately, teams must drastically simplify what they ask the model to do.

The most effective technique to overcome this difficulty is narrowing the judge's responsibility to strictly binary outcomes—such as deciding whether an output clearly passes or fails a single criterion—rather than grading on a broad or subjective spectrum. Once the task is scoped to a binary classification problem, engineers must hand-label a substantial volume of data themselves so they have a reliable ground-truth baseline against which to measure the judge's statistical alignment.

[11:27](https://www.youtube.com/watch?v=QDQT99csHJQ&t=687s) Mistake 4: Wrong Annotators and Criteria Drift

Passing the presentation back to Hamel Husain, the talk turns to a widespread human failure mode: delegating data annotation to people who lack the domain expertise required to evaluate system behavior. Husain observes that despite hearing repeatedly about the importance of inspecting data, teams consistently try to bypass the chore by outsourcing it or handing it off to the developers who wrote the application. Husain warns that this creates deeply flawed evaluation signals: “And unless you're building a coding app, it's kind of a bad idea, 'cause often those people don't have the domain expertise in whatever you're building or the problem you have.” Approaching evals like a rigorous data scientist requires skepticism toward both the labels and the annotators themselves; teams must confirm who is labeling, validate their domain qualifications, and audit the resulting annotations directly to catch unanticipated failure modes.

A primary reason hands-on inspection cannot be bypassed is criteria drift, a behavioral phenomenon documented in the research paper *Who Validates the Validators?*, co-authored by Shankar. When teams attempt to write an evaluation rubric in the abstract before viewing concrete system outputs, their expectations inevitably change the moment real data is placed in front of them. As Husain explains: “This idea that you're going to specify a rubric up front, and that's enough, is really problematic.” Rubrics written in isolation fall apart because practitioners discover edge cases, ambiguous language, and unexpected user intents only when forced to sit down and read actual production traces.

[13:03](https://www.youtube.com/watch?v=QDQT99csHJQ&t=783s) Mistake 5: Automating Too Much with LLMs

Teams frequently ask whether the entire evaluation loop can simply be handed over to a foundation model like Claude to avoid manual auditing altogether. Husain dismisses this as wishful thinking, explaining that while an LLM can catch obvious crashes, blatant formatting errors, or surface-level breakages, it has no window into domain-specific business expectations: “Like, Claude can't read your mind.”

Automated judges fail to detect critical product failures because they lack the implicit institutional knowledge and nuanced user expectations that human engineers carry. Uncovering why a response fails a user requires externalizing that tacit product context into explicit tests, rubrics, and data checks—work that no generic model prompt can invent on its own.

[13:45](https://www.youtube.com/watch?v=QDQT99csHJQ&t=825s) Rapid-Fire Pitfalls: ROUGE/BLEU, Raw JSON, and Overfitting

Husain runs through a rapid-fire catalog of recurring mistakes that derail production evaluation suites:

- **Misusing legacy NLP similarity scores:** Off-the-shelf evaluation frameworks frequently ship with classical metrics like ROUGE or BLEU enabled by default on dashboards, even though word-overlap metrics rarely correlate with actual response quality or semantic correctness in modern generative applications. - **Vague judge prompts:** Prompting an LLM judge with broad questions like "Is this helpful?" yields noisy, inconsistent judgments because the prompt is disconnected from the concrete failure modes of the specific product. - **Forcing annotators to read raw JSON:** Making human reviewers comb through unparsed JSON payloads adds unnecessary cognitive friction that degrades label quality; teams should instead build lightweight, custom annotation interfaces that make reviewing traces fast and delightful. - **Uncalibrated judge metrics:** Reporting scores from an automated judge without first measuring its statistical alignment against verified human labels turns metrics into pure guesswork. - **Ignoring criteria drift:** Failing to monitor how internal evaluation standards evolve over time as reviewers encounter new edge cases. - **Overfitting judges to development data:** Iteratively tuning a judge prompt against a single static dataset leads to prompt-level overfitting, identical to hill-climbing on training data in classical machine learning; teams must maintain held-out validation sets to confirm their judges generalize to unseen outputs. - **Poor data sampling and vanity metrics:** Sampling data carelessly or plastering dashboards with uncalibrated metrics that fail to provide real signal wastes engineering bandwidth.

Regarding the risk of tuning prompts without proper holdout sets, Husain stresses: “You need to set aside some data and make sure that your evals are generalizing.”

[15:34](https://www.youtube.com/watch?v=QDQT99csHJQ&t=934s) The Data Science Corollary and Final Takeaway

Stepping back from individual mistakes, Husain frames modern LLM evaluation not as an entirely new discipline, but as the natural resurgence of foundational data science practices. Every major failure mode in building LLM evals maps directly to a well-established data science skill: error analysis on model traces is equivalent to exploratory data analysis (EDA); scoping narrow pass/fail criteria is classic metric design; and measuring judge-to-human agreement is standard model validation.

```mermaid flowchart TD accTitle: Mapping LLM Eval Tasks to Classical Data Science Disciplines accDescr: A diagram connecting LLM evaluation tasks to their corresponding data science foundational skills.

subgraph LLM_Tasks["LLM Evaluation Workflow"] T1["Trace Error Analysis"] T2["Targeted Pass/Fail Checks"] T3["Judge-to-Human Alignment"] T4["Held-Out Test Sets"] T5["Production Observability"] end

subgraph DS_Skills["Data Science Foundation"] S1["Exploratory Data Analysis (EDA)"] S2["Metric Design"] S3["Model Validation"] S4["Dataset Curation"] S5["Monitoring & Telemetry"] end

T1 --> S1 T2 --> S2 T3 --> S3 T4 --> S4 T5 --> S5 ```

Building robust AI systems requires adopting a hypothesis-driven, scientific mindset where practitioners curate representative test sets, design specific measurements, and continually validate system behavior against reality. Husain directs attendees to an open-source evaluation skills repository designed to help teams audit their evaluation pipelines for these pitfalls.

Closing out the session, Husain delivers the single non-negotiable rule of the talk: “But most importantly, if you want to come away with something from this talk, is to always look at your data.” The presentation concludes with a QR code providing attendees access to the full slide deck alongside the collection of presentation memes.

## Notable quotes > [06:04](https://www.youtube.com/watch?v=QDQT99csHJQ&t=364s) I'm not saying that using LLM judges are bad. I'm saying that using LLM judges kind of blindly without validating your validators is bad. — Shreya Shankar

> [01:27](https://www.youtube.com/watch?v=QDQT99csHJQ&t=87s) AI engineering today feels like we've taken a bit of a step back in some places. So now, a lot of times, we're just using vibes to know if we're doing the right thing. — Hamel Husain

> [11:27](https://www.youtube.com/watch?v=QDQT99csHJQ&t=687s) This idea that you're going to specify a rubric up front, and that's enough, is really problematic. — Hamel Husain

> [13:45](https://www.youtube.com/watch?v=QDQT99csHJQ&t=825s) You need to set aside some data and make sure that your evals are generalizing. — Hamel Husain

## Mentioned Hamel Husain, Shreya Shankar, OpenAI, Cursor, Claude Code, Codex, Claude, LangChain, Who Validates the Validators?, ROUGE, BLEU

## Video description Shreya Shankar and Hamel Husain have taught evals to over 4,500 people across dozens of companies, and they keep seeing the same mistakes. This talk walks through five of the most common pitfalls and how thinking like a data scientist fixes them.

At Interrupt, the agent conference by LangChain, they covered: • Why generic metrics like "helpfulness" and "hallucination" are too ambiguous to use off the shelf • How to treat LLM judges as imperfect classifiers with train/dev/test splits • Why synthetic data generation goes wrong and how to fix it • Who should actually be labeling your data • What criteria drift is and why it happens • Why fully automating evals misses the product failures that matter most

The Return of the Data Scientist | Interrupt 26 0:00 Introduction 0:38 What is the harness? Logs, metrics, and traces 1:17 The harness is data science 1:27 How we got here: ML engineering 4 years ago 2:05 AI engineering today: vibes-based evaluation 2:37 What this talk covers: evals and common mistakes 3:28 Mistake 1: using generic or off-the-shelf metrics 4:47 How to fix it: explore data, build custom interfaces 6:04 Mistake 2: blindly trusting LLM judges 7:03 Treating LLM judges like ML classifiers 8:09 LLM judges as imbalanced classification problems 8:26 Mistake 3: bad experimental design 9:04 How to fix it: systematic synthetic data generation 10:13 Bad metric design: 1-to-100 scales 10:33 How to fix it: binary classification problems 11:23 Mistake 4: wrong people labeling data 12:00 Criteria drift 13:03 Mistake 5: automating too much 13:45 Other pitfalls: ROUGE/BLEU, unhelpful judge prompts, raw JSON, uncalibrated scores 15:34 The data science corollary: EDA, metric design, model validation 16:47 Summary: always look at your data 17:00 Where to get the slides

Extra resources: • Evals skills: https://github.com/hamelsmu/evals-skills • Everything mentioned during the session: https://maven.com/parlance-labs/o/0cb2fa • Everything we shipped at Interrupt: https://www.langchain.com/blog/interr... • Meet LangSmith Engine: https://www.langchain.com/blog/introd... • About LangChain: https://www.langchain.com/

[Source](https://www.youtube.com/watch?v=QDQT99csHJQ)

Reading summary

Hamel Husain and Shreya Shankar argue that effective evaluation for LLM applications requires abandoning superficial "vibes-based" shortcuts and returning to rigorous, classical data science practices like exploratory data analysis, binary metric design, and strict validator benchmarking. Harness Engineering as the Return of Data Science (0:06) Husain observes that while machine learning engineering four years ago prioritized careful data visualization, human alignment, and metrics tied directly to business outcomes, modern AI engineering often relies on unverified intuition or arbitrary 1-to-100 scores from external models. Highlighting recent reports of OpenAI using an evaluation harness to supervise autonomous agents over several months, Husain stresses that a harness cannot consist solely of unit tests and specifications; it inherently depends on an observability stack of logs, metrics, and traces. Replacing Generic Metrics with Exploratory Data Analysis (3:28) Shankar warns that off-the-shelf metrics targeting abstract concepts like "helpfulness" or "hallucination" are ambiguous and context-dependent, noting that hallucination in healthcare carries completely different operational requirements than in law. Instead of relying on generic dashboards or legacy NLP metrics like ROUGE and BLEU, teams should practice exploratory data analysis by inspecting application traces directly. Treating LLM Judges as Imperfect Classifiers (7:03) Prompting an LLM to rate performance on continuous scales or 1-to-5 Likert scores yields uninterpretable distributions that obscure concrete engineering fixes. Shankar and Husain recommend converting evaluations into targeted binary pass/fail checks aimed at specific failure modes, such as a scheduling assistant fabricating appointment times when no open calendar slots exist. Experimental Rigor, Synthetic Data, and Qualified Annotators (8:26) Shankar advises against naive synthetic data generation, recommending structured combinatorial cross-products across hypothesized dimensions of variation derived from genuine application traces. Husain cautions against delegating evaluation labels to software developers or third-party workers lacking domain expertise, noting that models cannot divine implicit business expectations without human domain context. Husain concludes that every major challenge in LLM evaluation maps onto established data science disciplines, including exploratory data analysis, metric design, dataset curation, and model validation. The primary non-negotiable directive for practitioners is to routinely look at their data rather than outsourcing evaluation to automated, uncalibrated pipelines.

Source quotes

[6:04] Shreya Shankar: I'm not saying that using LLM judges are bad. I'm saying that using LLM judges kind of blindly without validating your validators is bad. [1:27] Hamel Husain: AI engineering today feels like we've taken a bit of a step back in some places. So now, a lot of times, we're just using vibes to know if we're doing the right thing. [11:27] Hamel Husain: This idea that you're going to specify a rubric up front, and that's enough, is really problematic. [13:45] Hamel Husain: You need to set aside some data and make sure that your evals are generalizing.

Original transcript

[0:06] >> All right. Welcome, everyone. So today we are calling it the return of the data scientist. [0:14] We're going to -- great. I'm glad you all are excited about this. Okay. So just to set [0:22] the stage, you all probably have seen this article already about harnessing engineering. [0:28] Raise your hand if you've seen this. [0:31] Okay, so about half the people. [0:33] So if you haven't read this article, [0:35] I highly recommend taking a look. [0:38] But what it is about is a team at OpenAI, [0:41] they use agents autonomously for a few months [0:45] to build some very substantial software internally. [0:49] And what they did is using a harness [0:52] to keep the agents on track. [0:55] And so when people hear harness, [0:58] They usually think about specifications and unit tests, but one detail that's really important [1:05] to look at is, okay, the harness also contains logs, metrics, and traces. [1:13] So like the whole observability stack. [1:17] And what I'm here to convince you of today is that the harness is data science, or at [1:23] least a big part of it is data science. [1:27] So, okay, let's take a step back. [1:29] How do we get here? [1:32] Okay, so let's look at ML engineering data science [1:37] four years ago. [1:38] So we used to examine the data really carefully, [1:41] look at the data, visualize it. [1:44] Anytime you had a model or a prediction, [1:47] you would be very careful to make sure [1:49] you align it with human labels. [1:52] You're very careful to use the right tool for the task, [1:56] And when it came to designing metrics, we really take a lot of care to make sure those [2:01] are aligned with business goals and the metrics made sense. [2:05] AI engineering today feels like we've taken a bit of a step back in some places. [2:11] So now, a lot of times, we're just using vibes to know if we're doing the right thing. [2:16] A lot of times, we're just asking the model, another model, if it did a good job or the [2:20] same model. [2:23] And then we're not putting a lot of thought into metric design often, we're just using [2:27] an LLM to just grade something on a scale of 1 to 100, or just using someone's metric [2:32] package off the shelf without thinking about metrics. [2:36] And this is where people get into trouble. [2:39] And it shows up a lot, you can see it, because engineers often are scared of what they don't [2:49] understand. [2:51] The data science pieces of engineering -- evals and retrieval -- [2:55] Those have died so many times, [2:56] I can't even count it anymore. [2:59] But this talk is about evals. [3:02] And so, what we're gonna talk about today is, [3:05] okay, how do people go wrong with evals [3:08] and what mistakes do they make? [3:09] And how you can think like a data scientist [3:11] to overcome those mistakes. [3:14] And so we have taught over 4,500 people, [3:17] evals at lots of companies. [3:19] and we've seen the same mistakes over and over again. [3:22] So we're gonna highlight the most common ones. [3:25] And for that, I'm gonna give it over to Shreya. [3:28] - Great, hi everyone, I'm Shreya. [3:30] I'm gonna tell you five big mistakes [3:32] that people make coming into working with us [3:35] and how we can course correct it [3:36] by wearing a data scientist hat. [3:38] So the first one is kind of using metrics, [3:42] what we call generic metrics or off-the-shelf metrics [3:45] to measure the accuracy or alignment of your agents. [3:48] And it's tempting to use these generic metrics. [3:52] After all, we use generic metrics for software. [3:55] We measure things like latency or uptime. [3:57] But AI needs a different approach. [4:00] Even then, what are these so-called generic metrics? [4:03] You might have heard things like helpfulness, [4:06] or hallucination, or coherence. [4:08] And yeah, they sound like they're important to measure. [4:11] But when you really think about it, [4:12] they're quite ambiguous, one. [4:15] Can you really tell exactly what hallucination means? [4:19] Two, if you're building-- [4:21] Hamel is building an app maybe for a medical context, [4:24] or I'm building one for a legal context, [4:26] our definition of hallucination is going to be different. [4:29] Maybe different tools or different data that's [4:31] being used in these applications. [4:33] And it doesn't make sense to use the same definition [4:36] or the same way of evaluating hallucination [4:38] or an off-the-shelf evaluator to do this. [4:41] So how might we kind of circumvent [4:44] using off the shelf metrics. [4:46] How would we do this as a data scientist? [4:47] Well, we would actually explore the data, [4:50] look at what's breaking, and try to really specifically name [4:54] what the failure modes are to our system. [4:56] So more concretely, what we found that's really useful [5:00] is using tools like Codex or Claude Code or Cursor, [5:04] AI-assisted tools to build custom interfaces, [5:07] to load up traces for your agent, [5:09] actually read each message in each trace one by one, [5:13] And talk about what might be going wrong. [5:15] Pretend you're the user of your system. [5:18] Talk to your PM or talk to someone else [5:20] and be like, what are the mistakes here? [5:22] Write them down as open notes, which we don't show here. [5:25] But then over time, as you build these collections [5:27] of open notes, you can then categorize them [5:30] into the bespoke failure modes that you need. [5:32] So for example, in this real estate agent tour app, [5:37] maybe some of the bespoke failure modes [5:39] were it was rescheduling tours whenever there was no-- [5:43] They were hallucinating times for the tours. [5:46] That's something very specific to the app. [5:49] Hamel and I always say, look at your data. [5:51] And this is exactly what we mean by look at your data. [5:53] It's not just, oh, look at it and don't say or think [5:56] or do anything about it. [5:57] It's build your own interfaces and try to find failure modes [6:01] and try to do this at scale. [6:04] The second metric, or this is sorry, the second failure mode [6:06] that we commonly see is really using LLM judges [6:13] try to figure out what's going wrong. [6:15] Maybe you looked at your data and you found a failure mode, [6:17] but then you simply go and ask an LLM, hey, [6:21] how often is this occurring in my data [6:23] without any notion of trust in that LLM judge itself? [6:27] So I'm not saying that using LLM judges are bad. [6:30] I'm saying that using LLM judges kind of blindly [6:33] without validating your validators is bad. [6:36] So what do most teams do when they use LLM judges? [6:39] Well, they ask an LLM, hey, [6:42] I have this failure mode that I'm trying to measure, [6:45] maybe rate how good or bad something is on a one to five [6:49] scale, and then give me the numbers. [6:51] And then they kind of see this histograms of one and five, [6:53] and then try to make some business decision about that. [6:56] That's not really good. [6:58] We found that it's really difficult to trust those numbers [7:00] or even turn them into actionable business metrics. [7:03] So what would a data scientist do? [7:05] Or if you were to wear your data scientist hat, [7:07] what would you do? [7:08] Well, you would kind of say, OK, the LLM judges [7:10] an imperfect classifier. [7:12] And maybe I should treat finding or training this classifier [7:16] as I would any machine learning model in the past. [7:19] So concretely, that means taking examples of labeled traces [7:22] for each failure mode and trying to partition data [7:25] into train, development, and test sets. [7:28] If you've been in machine learning, [7:30] you're very, very familiar with this. [7:32] The idea of separating your data, [7:34] trying to figure out what prompt or what model works really [7:37] well in your train and development set, [7:39] and then making sure it doesn't overfit the alignment [7:42] on a test set. [7:43] So really what we found is there's no difference here [7:46] with LLM applications. [7:47] You still wanna go through the same rigorous process. [7:50] You wanna make sure that you don't overfit your LLM judges. [7:56] And in many classification tasks, interestingly, [7:59] LLM judging is often an imbalanced classification task, [8:03] which is a fancy way, it's a lot of jargon [8:04] for saying that only a small fraction of the trace set [8:07] is a failure or a failure mode. [8:09] So when you measure alignment with your preferences, [8:12] don't simply measure accuracy. [8:14] Use some of these metrics that we've designed [8:17] for imbalanced classification tasks, [8:19] like precision or recall or false positives [8:21] or false negatives to make sure [8:23] that your LLM judges are aligned. [8:26] So those are two pitfalls. [8:28] The third pitfall that we see is really, really broad, [8:31] but bad experimental design, really. [8:34] And I'll talk about two ways that we see it. [8:36] One is people like to generate synthetic data. [8:40] I love doing that. [8:41] But they'll do so in a way where all the traces end up [8:44] looking something like this. [8:46] Why does that happen? [8:47] Well, it's very easy to get there when you just [8:49] ask an LLM for synthetic data. [8:51] Like give me data or five generic questions [8:54] if you're trying to train a document question answering [8:56] chatbot. [8:58] OK, maybe that's not good. [9:00] So how might we think like a data scientist [9:02] to improve this process? [9:05] we'll might think about ways in which we want to systematically [9:08] generate the synthetic data and use [9:10] LLMs for very small parts of the data generation process. [9:14] So hypothesize which dimensions vary in the data [9:18] that users bring into the system, [9:20] generate combinations of this data, [9:23] review all your synthetic data for quality, [9:25] make sure there's good diversity, look at the data again. [9:29] And there's a lot of different ways [9:30] you can think about application-specific ways [9:34] to generate synthetic data. [9:36] So one exercise we like to do with people [9:39] is just get into your application, look at some traces, [9:42] and come up with at least three different dimensions [9:45] that vary across users. [9:46] So sometimes that's the persona of the user, [9:49] maybe they're a novice, or maybe there's somebody [9:51] who's much more experienced. [9:54] Use LLMs to generate different values for each dimension, [9:58] and then kind of, for lack of a better term, [10:00] take the cross product of all of these dimensions [10:02] to generate your synthetic data. [10:05] All right, so that was one way [10:07] in which you could have bad experimental design [10:09] and generating synthetic data poorly. [10:11] Another one is in designing metrics. [10:13] And I talked about why we need application-specific metrics, [10:17] but also how we evaluate the metrics is very important. [10:20] A lot of times people evaluate metrics [10:22] like score things on a one to five scale [10:24] or one to 100 scale, and that's not very interpretable [10:27] or actionable for us. [10:29] How might we revisit this problem as a data scientist? [10:33] Well, we'll try to make the problem as least complex [10:36] as possible. [10:37] We'll try to make things a binary classification problem. [10:40] We're trying to ask the judge for whether something fails [10:42] or not, a binary classification problem, [10:45] rather than one to 100 or one to five scoring [10:48] for something like failure. [10:49] We also want to make these actionable or aligned [10:51] with business outcomes. [10:53] And really, this is really difficult to do. [10:55] It's not going to be something that you can one shot or get. [10:58] First, especially when you're trying to align LLM judges, [11:02] often we see people really struggling, even ourselves. [11:05] Like we can never write a good LLM judge prompt [11:07] on the first time. [11:08] But some things that help us are make these LLM judges [11:11] very, very narrow in scope towards binary tasks, [11:15] like passing something or failing something, [11:17] and then label lots of data ourselves [11:19] to try to measure alignment with that. [11:23] All right, I'm gonna hand it back to Hamel [11:25] for the last two. [11:27] All right, some other pitfalls are, [11:29] and this one happens quite a lot, is, [11:32] okay, so as much as we talk about looking at data, [11:36] a lot of people for whatever reason [11:37] don't want to look at data, [11:39] and they just outsource it to someone else. [11:42] And one really common failure mode is, [11:44] okay, let's let the AI engineer [11:46] or the developers label data. [11:48] And unless you're building a coding app, [11:50] it's kind of a bad idea, [11:51] 'cause often those people don't have the domain expertise [11:54] in whatever you're building or the problem you have. [11:57] And so you don't want to do this. [12:00] So most teams make it someone else's problem, [12:03] but putting your data science hat on don't trust anything. [12:07] You don't trust the labels. [12:10] Don't trust the people labeling the data. [12:12] You make sure whoever's doing the labeling [12:14] actually has a domain expertise, and then [12:16] you need to look at the labels. [12:18] Because oftentimes there's some kind of problem [12:20] and what's happening that you are not anticipating. [12:25] And another reason to look at the data [12:28] is this phenomenon called criteria drift. [12:30] And this is from a paper who validates validators. [12:32] Shreya is an author on this paper, [12:34] so you should totally check it out. [12:35] But criteria drift is a very simple concept. [12:38] It's just they found that people don't know what they want [12:42] unless they look at some data. [12:45] This idea that you're going to specify a rubric up front, [12:48] and that's enough, is really problematic. [12:51] You need to look at data. [12:53] And you really need to force people to look at the data. [12:58] And it can feel like this. [13:00] It is kind of like this, honestly. [13:03] Another pitfall is automating too much. [13:06] So you might be thinking in your mind, like, oh, [13:09] all this eval stuff, can't just have Claude do it for me? [13:15] And the answer is no. [13:17] Like, Claude can't read your mind. [13:20] Claude doesn't know all the different product nuances [13:23] can go wrong. Sure, there's some low-hanging fruits that LLMs can find in terms of things [13:28] that are obviously errors, things that are obviously broken. But there's a lot of context [13:32] that you need to externalize to get to some of the more important things often that are [13:38] broken, especially the more product failures that might be happening with your users. [13:45] So, there's a bunch of other pitfalls. We don't have time to go through every single [13:49] pitfall exhaustively, but let me just give you a taste. So, misusing similarity scores, [13:55] okay? Like, ROUGE, BLEU, whatever, like, you see that on a lot of eval dashboards and it [14:01] doesn't make any sense. Like, do you really want to be measuring, like, similarity? You [14:08] know, but they come with a lot of off-the-shelf eval frameworks. I always see also number [14:15] Number two here, asking the judge, is this helpful? [14:17] This is like really bad LLM judge prompts that are not specific to your product. [14:24] Another pitfall is making annotators read raw JSON. [14:27] You should take all the friction out of looking at data. [14:30] So you should build your own data annotation interface that makes reading data delightful. [14:36] Fourth one, reporting uncalibrated scores. [14:39] So you also want to make sure, okay, like if you have an LLM judge, you want to make sure [14:42] like you study the alignment of that judge [14:44] with the human being. [14:45] Otherwise, it's just anybody's best guess. [14:48] You have to also be careful, number five, [14:51] to make sure you understand what data criteria drift is [14:54] happening. [14:56] Number six, you want to make sure you're not [14:57] overfitting your judges to data. [14:59] So kind of think of it like machine learning. [15:01] You don't want to just take a set of data [15:03] and hill-climb against that data iteratively and overfit. [15:07] You need to set aside some data and make sure [15:10] that your evals are generalizing. [15:13] You want to also make sure that you're sampling data [15:16] effectively. [15:17] And you want to make sure, if you're [15:19] going to put a metric on a dashboard, [15:21] be really careful that that metric is earning its place [15:25] and actually has signal. [15:27] There's not necessarily a problem just with AI. [15:29] There's always been a problem, but it's definitely [15:31] more acute in AI. [15:34] And so if you really take a step back, [15:37] Okay, a lot of the things that we talked about today, [15:40] there's a corollary kind of skill in data science [15:44] that helps alleviate this problem. [15:46] So, first thing we talked about, error analysis, [15:50] or data analysis, looking at your data. [15:53] You wanna find patterns in your traces, [15:55] you wanna make sure you're reading those traces [15:58] and analyzing the patterns in them. [16:00] That's a lot like EDA, or exploratory data analysis. [16:04] We talked about making sure you have good metrics that [16:08] are scoped to actual problems. [16:11] And that's metric design. [16:14] Validation, like a model of validation, how you would do [16:17] in machine learning, you want to make sure your LLM as a [16:20] judge is aligned with human judgment. [16:24] And then you want to make sure that you are curating your [16:27] data properly, your test data. [16:29] And then also you want to make sure you do your monitoring [16:33] and observability. [16:34] And then you want to approach this whole thing [16:36] with a scientific mindset. [16:38] You want to experiment, and then you [16:40] want to measure and try to improve. [16:42] And that muscle of being a data scientist [16:44] is really helpful here. [16:47] So we talked a lot today about a lot of different things. [16:51] I have these skills that help you audit your evals [16:54] to see if you're doing anything incorrectly. [16:57] So don't worry about writing this down. [17:00] I'll share a link. [17:02] But most importantly, if you want to come away with something [17:06] from this talk, is to always look at your data. [17:10] Now, there's one question that everyone has. [17:13] Anytime Shreya and I give a presentation is, [17:17] where are the slides? [17:18] Can I appreciate you sharing the slides? [17:20] So you can get the slides by looking at this QR code. [17:23] So we'll send you the slides. [17:24] We'll also send you all the memes. [17:26] There's like a lot more memes, which is really cool. [17:30] So thank you. [17:32] [APPLAUSE] [17:35] [APPLAUSE]