The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks

The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks — AI Engineer

AI Engineer

AI-assisted summary of The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks — original source. Check the original for full context.

Source published:

Key ideas

  • Starting enterprise AI projects with model selection is an anti-pattern; in an 8-week turnaround for an £85,000 failed banking PoC, the team deferred model selection to week seven, spending the first six weeks on evaluation data, tracing, and data foundation. (24:35)
  • Production agent evaluation requires three distinct layers: deterministic code checks (regex, classic ML for PII/NER), semantic scoring (LLM-as-a-judge), and behavioral evaluation to detect redundant tool calls and runaway API loops. (7:33)
  • Data foundation typically absorbs 60% of project effort because autonomous agents do not forgive pipeline errors and output wrong data with high confidence, necessitating separate schemas for 'question data' and 'tracking data'. (15:06)
  • In regulated environments like European banking, end-to-end decision tracing is legally mandated for automated dispute resolution and runtime safety, such as enforcing human handoff after three failed tool executions. (11:25)

Detailed notes

## Background Published by AI Engineer on YouTube on June 18, 2026, this talk features Sandipan Bhaumik, a technical lead for Data and AI at Databricks. The description outlines how Bhaumik's team rescued an £85,000 retail bank chatbot proof-of-concept that stalled for six months, establishing a five-pillar enterprise deployment playbook centered on evaluation, observability, data foundations, multi-agent orchestration, and governance.

## TL;DR Sandipan Bhaumik addresses the common failure mode of enterprise AI projects—the 'model-first trap'—where teams prematurely debate LLM selection before defining how to observe, measure, or govern agent behavior. To solve this, he outlines a five-pillar framework spanning deterministic-to-behavioral evaluation layers, end-to-end decision tracing for regulatory compliance, and a bifurcated data foundation separating question data from tracking telemetry. Through a real-world retail bank case study, Bhaumik demonstrates how deferring model selection until week seven of an eight-week sprint enabled teams to intercept 47 PII leaks and rapidly diagnose post-launch retrieval failures caused by stale embeddings. He concludes with a structured production incident playbook alongside practical guidance on prompt commit governance, living test suites, and tiered CI/CD testing budgets.

## Atmosphere & style The session has the measured, pragmatic authority of an engineering post-mortem delivered from the enterprise trenches. Bhaumik speaks with calm, candid clarity, replacing generative AI hype with the unvarnished realities of distributed systems, compliance audits, and silent production failures. The narrative moves deliberately from cautionary tales of squandered budgets to structured architectural blueprints, creating a reassuring, highly practical atmosphere for engineers navigating the friction of moving prototypes to production.

## Bulletin - [24:35](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1475s) Starting enterprise AI projects with model selection is an anti-pattern; in an 8-week turnaround for an £85,000 failed banking PoC, the team deferred model selection to week seven, spending the first six weeks on evaluation data, tracing, and data foundation. - [07:33](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=453s) Production agent evaluation requires three distinct layers: deterministic code checks (regex, classic ML for PII/NER), semantic scoring (LLM-as-a-judge), and behavioral evaluation to detect redundant tool calls and runaway API loops. - [15:06](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=906s) Data foundation typically absorbs 60% of project effort because autonomous agents do not forgive pipeline errors and output wrong data with high confidence, necessitating separate schemas for 'question data' and 'tracking data'. - [11:25](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=685s) In regulated environments like European banking, end-to-end decision tracing is legally mandated for automated dispute resolution and runtime safety, such as enforcing human handoff after three failed tool executions.

## Key takeaways - [01:52](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=112s) Enterprise AI initiatives frequently stall post-launch due to three systemic deficits: the Observability Gap, the Evaluation Gap, and the Governance Gap. - [04:54](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=294s) The five pillars of enterprise agent deployment are Evaluation, Observability, Data Foundation, Multi-Agent Orchestration, and Governance, all of which must be scoped before writing application code. - [07:33](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=453s) Evaluation must serve as the formal software specification, defining precise numerical thresholds such as deflection rates and acceptable false-positive metrics upfront. - [07:33](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=453s) Deterministic evaluation tasks like format validation, regex matching, and PII masking should be resolved upstream in low-cost classic code rather than via expensive LLM reasoning. - [10:01](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=601s) Behavioral evaluation uncovers silent operational failures, such as duplicate database queries and infinite loops, that pass semantic accuracy checks but create severe latency and cost bottlenecks under enterprise scale. - [11:25](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=685s) Granular execution tracing is mandatory in regulated jurisdictions like Europe, enabling teams to audit decisions, resolve customer disputes, and enforce circuit breakers or human escalations. - [15:06](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=906s) Enterprise data strategies must split into 'question data' for answering domain queries and 'tracking data' for streaming telemetry into compliance, auditing, and automated evaluation systems. - [16:27](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=987s) Centralized governance platforms like Unity Catalog provide critical metadata discovery, column descriptions, and PII tags that autonomous agents need to query data without exfiltrating sensitive information. - [20:05](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1205s) Multi-agent orchestration requires choosing between orchestrator-worker patterns for centralized debugging, choreography patterns for parallel low-latency execution, or human-in-the-loop workflows for confidence-based fallback. - [22:25](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1345s) AI governance requires rigorous pre-validation filtering—which caught 47 pre-launch PII leaks in a retail banking client—as well as strict version control and change management for operational prompts. - [26:07](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1567s) An evaluation dataset is a dynamic, living asset built from messy domain-expert interactions that continuously expands as new production edge cases and regression failures emerge. - [29:41](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1781s) A post-launch CSAT drop caused by stale interest rate answers demonstrated how production tracing instantly identifies silent RAG failures like un-embedded policy updates. - [30:53](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1853s) An enterprise production incident playbook must define a structured four-stage cycle—Detect, Diagnose, Contain, and Fix—integrated into existing IT Service Management (ITSM) systems. - [33:26](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=2006s) To control rising CI/CD costs, teams should run lightweight evaluation smoke tests on branch pull requests and restrict exhaustive Layer 3 behavioral test suites to main branch merges.

## Detailed walkthrough ### [00:15](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=15s) From Proof-of-Concept to Production: Lessons from the Trenches

Sandipan "Sandy" Bhaumik introduces himself as a technical lead for Data and AI at Databricks, following a five-year tenure at Amazon Web Services as a principal architect specializing in distributed data and AI platforms. While artificial intelligence has existed for decades, he reflects on how the industry has entered a phase of exponential experimentation over the past two years. Working directly with enterprise clients across B2B software and heavily regulated industries such as financial services, he has spent these years tackling the friction of translating flashy proof-of-concept demos into reliable, production-grade applications.

Bhaumik explains that the goal of the session is to synthesize those hard-won lessons from the trenches into a practical deployment playbook. Acknowledging the conference schedule, he notes that an afternoon session offers the ideal vantage point for attendees to take the granular concepts heard throughout the day and organize them into an overarching operational architecture. Rather than treating AI engineering as an ad-hoc series of experiments, teams need a concrete framework to evaluate where their tooling, observability, and models actually belong.

As Sandy Bhaumik frames the current industry inflection point: “In the past couple of years, specifically, I've been working with customers trying to figure out what we do with this new AI technology.”

- Prior to joining Databricks, Sandy served for 5 years as a principal architect for data and AI at AWS, focusing on scaling distributed systems. - His enterprise engagements center on taking prototypes out of sandbox environments and deploying them inside strict B2B and financial regulatory environments.

---

[01:52](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=112s) The "Model-First" Trap and the Three Gaps to Production

Looking back at client engagements over the prior two years, Bhaumik outlines a recurring anti-pattern driven by top-down pressure. Organizations felt immense urgency from executive leadership to deploy AI solutions immediately, which caused teams to begin almost every technical discussion with model selection. Stakeholders would exhaustively debate whether to use GPT or Claude before understanding the underlying operational requirements. Once a model was selected, engineers would build features around predictable datasets in tightly controlled sandbox environments. The demos inevitably looked impressive, leadership signed off on deployment, and the project shipped into production.

Within weeks of launch, the illusion broke down as real users introduced inputs that the system was never equipped to handle. Organizations were left baffled when their systems behaved erratically and failed to answer routine inquiries the way they had during presentations. This premature push into production yielded little to no return on investment while squandering substantial capital and engineering bandwidth on architectures that could never scale.

Reflecting on the frantic post-launch confusion inside these enterprises, Sandy Bhaumik recalls: “Then, after a few weeks, people would start asking questions that what the hell is AI doing?”

From analyzing these stalled initiatives, Bhaumik identified three fundamental structural gaps that doom enterprise AI projects from the outset:

1. **The Observability Gap:** If engineers cannot observe the internal behavior of an agent in real time and trace every downstream tool call or decision branch, the system is fundamentally unsafe and unusable in production. 2. **The Evaluation Gap:** Organizations speak abstractly about accuracy, latency, and groundedness, but fail to define exact business-level numerical thresholds or build continuous automated systems to track whether outputs are improving or degrading over time. 3. **The Governance Gap:** Teams fail to establish ownership and operational procedures for production incidents. When a model hallucinates offensive content or serves bad guidance at 3:00 a.m., there is often no clear protocol for who owns the underlying data assets or who is held accountable.

---

[04:54](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=294s) The Five Pillars of Enterprise AI Deployment

To counter these systemic failures, Bhaumik developed a battle-tested enterprise deployment framework composed of five core pillars. While organizations ideally tackle these steps sequentially, he acknowledges that real-world pressures often force teams to build them concurrently. Regardless of execution order, all five components must be planned before writing application code.

The framework starts with **Evaluation**, establishing rigorous numerical success criteria upfront. The second pillar, **Observability**, enforces end-to-end tracing across every agent decision—an absolute prerequisite for companies operating in Europe and other heavily regulated jurisdictions where AI systems cannot be legally onboarded without full auditability. The third pillar is the **Data Foundation**, which Bhaumik bifurcates into "question data" (pre-training, post-training, and real-time retrieval APIs) and "tracking data" (the high-volume decision traces generated by running hundreds of autonomous agents). The fourth pillar addresses **Multi-Agent Orchestration**, noting that while single-agent systems are straightforward, coordinating five or more agents introduces exponential complexity around message routing, state synchronization, and execution dependencies. The final pillar is **Governance**, which formalizes accountability, data security, prompt-injection defense, and brand protection when failures occur.

Emphasizing that engineering priorities must be inverted, Sandy Bhaumik states: “Before touching any code, before discussing about any models, any features, you have to think about when we build this system, how do we measure?”

```mermaid flowchart TD accTitle: The Five Pillars of Production AI Deployment accDescr: Sequential flow showing the five structural pillars required to take enterprise agents to production. subgraph Foundation ["System Readiness"] P1["1. Evaluation<br/>(Metrics & Benchmarks)"] P2["2. Observability<br/>(Tracing & Auditability)"] P3["3. Data Foundation<br/>(Retrieval & Trace Telemetry)"] end subgraph Execution ["Scale & Safety"] P4["4. Orchestration<br/>(Multi-Agent Coordination)"] P5["5. Governance<br/>(Security, PII & Ownership)"] end P1 --> P2 P2 --> P3 P3 --> P4 P4 --> P5 ```

- **Regulatory Compliance:** In European jurisdictions and regulated sectors like banking, automated decision systems cannot be deployed to production without deep execution tracing in place. - **Trace Telemetry Strategy:** Managing the observability logs of hundreds of interacting enterprise agents requires a standalone data infrastructure strategy comparable to core business intelligence pipelines.

---

[07:33](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=453s) Pillar 1: Evaluation as Specification and the First Evaluation Layer

Bhaumik dives into the first pillar, insisting that evaluation must function as the formal software specification for any AI system. Rather than using vague qualitative goals, teams must translate target outcomes into hard numbers: acceptable false-positive rates, minimum accuracy thresholds, and business-specific performance metrics. In a retail banking chatbot, for instance, the primary business objective is frequently query deflection—measuring the exact percentage of routine customer questions the agent can successfully resolve without escalating to an expensive human support representative.

Building an effective evaluation framework requires assembling a golden dataset grounded in frontline reality rather than synthetic assumptions. Bhaumik advises engineers to sit directly with customer support agents and domain experts to document how actual edge cases, ambiguous questions, and gray areas are handled on the phone or over chat. Once compiled, this dataset powers an automated evaluation pipeline that continuously scores live agent responses against golden test cases, providing immediate feedback on whether the model is meeting its numerical benchmarks.

Architecturally, Bhaumik highlights that evaluation systems across mature organizations break down into three distinct operational layers. The first layer consists of **Deterministic Evaluation**. This layer handles straightforward, computationally cheap verifications that software engineering has solved for decades: regular expressions for checking phone number and email syntax, alongside classic machine learning models for named entity recognition (NER), intent classification, and personally identifiable information (PII) detection. Bhaumik stresses that teams should strip these deterministic checks out of LLM prompts and resolve them upstream via traditional code and lightweight models before deploying expensive, non-deterministic reasoning.

Setting the baseline definition for building reliable AI pipelines, Sandy Bhaumik insists: “Evaluation is basically specification for your AI system.”

- Numerical metrics must replace qualitative descriptions: define acceptable false-positive rates and deflection targets upfront. - Evaluation sets must represent messy ground-truth scenarios captured directly from human domain experts rather than clean, idealized benchmarks. - Deterministic checks (format validation, regular expressions, NER, and PII masking) should be resolved in the first evaluation layer using classic, low-cost code and classic ML models rather than LLMs.

[10:01](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=601s) Semantic Evaluation and Behavioral Checks

Evaluating non-deterministic semantic responses requires automated evaluation techniques such as LLM-as-a-judge. In this setup, a dedicated secondary model evaluates the output of the primary model against specific criteria, including groundedness, safety, and relevance to the user's intent. The evaluation dataset informs the judge model by providing expected answers and criteria to measure against. In enterprise platforms like Databricks MLflow, these custom judges run automatically across production traces at scale, evaluating whether the generated text aligns with the intended prompt specifications and factual grounding.

```mermaid flowchart TD accTitle: Three-Layer Agent Evaluation Architecture accDescr: Diagram illustrating the three evaluation layers from deterministic to semantic and behavioral checks.

subgraph L1["Layer 1: Deterministic"] D["Regex, schema checks, format validation"] end

subgraph L2["Layer 2: Semantic (LLM-as-a-Judge)"] S["Groundedness, safety, answer relevance"] end

subgraph L3["Layer 3: Behavioral"] B1["Tool execution & routing"] B2["Infinite loop detection"] B3["Duplicate call minimization"] end

L1 --> L2 L2 --> L3 ```

Beyond deterministic and semantic checks lies behavioral evaluation, which scrutinizes the execution mechanics of agents—specifically how they interact with external tools and systems. Sandipan Bhaumik points out that an agent can appear completely successful on the surface while masking severe operational flaws underneath. For instance, when a customer asks, *"What is my account balance?"*, the agent may respond with the exact, correct dollar amount, passing both deterministic formatting tests and semantic accuracy checks. However, examining the behavioral telemetry reveals that the agent executed three redundant calls to the underlying customer database because previous calls failed or lacked proper termination logic.

While multiple repeated queries can pass unnoticed in a small demo, Bhaumik warns that they become catastrophic under enterprise load: “Now, three API calls in demo environment is fine, but in production, when you get thousands of queries from users every day, and there's like duplication in API calls, that's an expensive operation.” Because behavioral evaluation directly impacts cost, latency, and reliability, overlooking this layer poses a major risk to enterprise deployments.

- **Primary judge criteria**: Measures groundedness, safety filters, context recall, and relevance to the question. - **Behavioral failure modes**: Unnecessary tool retries, looping agent logic, and silent duplicate database queries. - **Enterprise impact**: Multiplied cloud infrastructure and API costs when thousands of concurrent users trigger redundant backend transactions.

---

[11:25](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=685s) Observability and Decision Tracing in Regulated Environments

The second pillar of the production playbook is observability, powered by comprehensive decision tracing. In an enterprise system, every autonomous step an agent takes must be captured and structured into an inspectable timeline. Bhaumik illustrates this with a real-world scenario from a retail banking chatbot handling an overdraft fee waiver request:

```mermaid flowchart LR accTitle: Retail Bank Agent Decision Trace accDescr: Step-by-step trace of how a banking agent handles an overdraft fee dispute.

A["User Request: Waive Fee"] --> B["Intent Classification"] B --> C["Customer DB Lookup"] C --> D["RAG Vector Search"] D --> E["Reasoning & Policy Check"] E --> F["Guardrails & Output"] ```

When a customer contacts support claiming an overdraft charge was illegitimate and asking for it to be waived, observability infrastructure traces every step: 1. **Intent Classification**: Classifies the query as a fee dispute, logging the exact latency (in seconds) and the model's confidence score. 2. **Account Integration**: Calls a backend API to retrieve the customer's account records from the core banking database. 3. **Policy Retrieval**: Performs a RAG lookup against a vector database to verify current bank rules regarding overdraft exemptions. 4. **Reasoning Engine**: Synthesizes the customer's account standing against the retrieved policy guidelines to determine eligibility. 5. **Guardrail Check**: Runs final output filters to verify compliance and safety before returning the final response.

Without granular decision tracing, teams are defenseless when customers dispute automated decisions. As Sandipan Bhaumik emphasizes, “Now, if you did not set up a system that helps you look visualize all of these traces, when the customer comes to you and raises a dispute, you have no way to check what the AI did.” Teams often default to writing off fees or issuing discounts because they cannot verify what occurred. Beyond operational recovery, European and global regulators explicitly mandate auditable decision trails, making tracing a prerequisite for going live. In production, this telemetry feeds online monitoring systems that trigger fallback strategies—such as capping automated API retries at three attempts before handing off the session to a human agent.

- **Regulatory compliance**: Financial service watchdogs require step-by-step auditability for automated decisions. - **Dispute resolution**: Detailed traces record whether the agent consulted stale policy documentation or misinterpreted account metadata. - **Fallback enforcement**: Automated guardrails prevent runaway loops by routing interactions to human agents after three failed tool executions.

---

[15:06](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=906s) The Data Foundation: Question Data vs. Tracking Data

Data foundation is the most critical pillar of any production rollout, routinely consuming 60% of project time. Enterprise data was historically collected and formatted for human consumption, but AI agents interact with data in a fundamentally different way. When a human encounters bad or missing information in an internal dashboard or report, they rely on intuition, cross-reference peers, or submit a data correction request.

Agents lack human intuition and cannot compensate for underlying pipeline defects. Explaining this gap, Sandipan Bhaumik observes: “Data was always built for humans, and humans are always forgiving. You find the wrong data in a report, you just go and ask someone to correct it. Agents don't forgive you, right? Agents will go, find it wrong, they'll give you the wrong answer confidently.” Because an agent presents inaccurate data with high confidence, data quality and governance dictate the success of the application.

Enterprises must split their data strategy into two operational categories: 1. **Question Data**: The domain-specific knowledge bases, structured tables, and transactional records needed to answer user queries and execute tools. 2. **Tracking Data**: The observability, telemetry, and execution traces produced by the agent during every interaction.

Tracking data requires a dedicated data schema plan. It must be structured so it can be streamed to auditors, queried by compliance regulators, scanned by online monitoring pipelines, and evaluated by automated LLM judges without slowing down production runtime.

- **Project allocation**: Approximately 60% of total implementation effort centers on cleaning, structuring, and governing agent-facing data. - **Human vs. Agent data tolerance**: Humans spot flaws and adapt; agents ingest pipeline errors and assert hallucinations with total certainty. - **Operational requirements for tracking data**: Strict schema enforcement, persistent audit storage, and low-latency availability for both automated judges and external compliance auditors.

---

[16:27](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=987s) Enterprise Data Architecture and Centralized Agent Tracing

To run autonomous agents reliably at scale, organizations rely on a multi-layered data foundation built on cloud storage (across AWS, Azure, and Google Cloud) and modern lakehouse architecture. Raw files—such as images, text documents, and video—are stored in object storage and wrapped by Delta Lake. Delta Lake adds ACID transaction support, manifest-driven table structures, and incremental batch or streaming data loading to unstructured and semi-structured assets.

```mermaid flowchart TD accTitle: Databricks Enterprise Data and AI Stack accDescr: Layered architecture showing cloud storage up through Delta Lake, Unity Catalog governance, and agent applications.

subgraph Apps["Applications & Tooling Layer"] M["Mosaic AI (LLMs & Apps)"] G["Databricks Genie (Text-to-SQL)"] A["Custom Databricks Apps"] E["MLflow & Agent Bricks (LLM Judges)"] end

subgraph Gov["Governance Layer"] U["Unity Catalog (Permissions, Metadata, PII Tagging)"] end

subgraph Storage["Data Foundation Layer"] D["Delta Lake (ACID, Manifests, Incremental Loads)"] C["Cloud Object Storage (AWS, Azure, GCP)"] end

C --> D D --> U U --> Apps ```

On top of Delta Lake, Unity Catalog serves as a centralized governance layer. It enforces role-based access controls, tracks end-to-end data lineage, and supports external access via Delta Sharing. Critically for AI agents, Unity Catalog manages metadata discovery, table descriptions, column definitions, and PII tags. When an autonomous agent or text-to-SQL model queries enterprise tables, it relies directly on these schema descriptions and PII tags to navigate schemas safely and prevent data exfiltration.

On top of this foundation, enterprise tools operate across multiple functions: Mosaic AI for model development and fine-tuning, Databricks Genie for natural language text-to-SQL querying, custom Databricks Apps for operational user interfaces, and MLflow with Agent Bricks for automated LLM judges. In large enterprises, developer teams rarely deploy on a single framework; they run combinations of LangChain, CrewAI, and native SDKs across multiple clouds. Centralizing observability and tracking telemetry into a shared data platform gives first-line defense teams, support engineers, and automated evaluators a single pane of glass to audit system health and agent behavior.

- **Unified governance**: Unity Catalog supplies rich semantic descriptions and column-level PII tags directly to autonomous agents during tool execution. - **Cross-framework consolidation**: Centralizes telemetry across disparate orchestrators (LangChain, CrewAI, custom agents) into structured lakehouse tables. - **Downstream observability use cases**: Powers automated compliance audits, operational health dashboards for first-line support, and ad-hoc troubleshooting via natural language SQL queries using Databricks Genie.

[20:05](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1205s) Pillar 4: Multi-Agent Orchestration Patterns (Orchestrator, Choreography, Human-in-the-Loop)

Moving past data unification, Sandipan introduces the fourth pillar: selecting the right multi-agent orchestration architecture. Operating a single agent is relatively straightforward, but as soon as an enterprise introduces multiple collaborating agents, systemic complexity explodes. Sandipan outlines three primary interaction topologies teams must evaluate based on their operational requirements: the orchestrator-worker pattern, the choreography pattern, and human-in-the-loop workflows.

In the orchestrator-worker pattern, a centralized control plane acts as the single point of entry and coordination. It decomposes incoming tasks and routes requests to specialized worker agents based on their discrete domain capabilities. Because every request passes through the central brain, debugging failure modes is clean and deterministic: operators simply inspect the orchestrator's central audit logs to see exactly where a sub-task stalled or returned bad output. Conversely, the choreography pattern eliminates the central orchestrator entirely in favor of an event-driven message bus. Independent, autonomous agents subscribe to specific domain events and execute concurrently rather than sequentially. In an enterprise mortgage application workflow, for instance, one agent can inspect customer profile details while another parallel agent evaluates underwriting approval criteria. By cutting out back-and-forth orchestration roundtrips, this pattern significantly drives down latency. Finally, human-in-the-loop patterns serve as essential safety valves, automatically routing transactions to a human supervisor whenever an agent's confidence score dips below an established operational threshold.

- **Orchestrator-Worker:** Centralized routing, strict execution control, and unified logging, making post-incident forensic analysis straightforward. - **Choreography:** Autonomous agents communicating asynchronously over an event bus, executing parallel workflows (e.g., identity verification and credit checks) to minimize end-to-end response latency. - **Human-in-the-Loop:** Automated escalation paths that suspend autonomous execution and request human sign-off when model uncertainty or risk thresholds are breached. - **Deep-Dive Reference:** Sandipan notes he recorded a dedicated technical talk for the conference's online track covering state management, fault tolerance, failure handling, and enterprise scaling patterns for multi-agent architectures.

> “As I said, one agent is good, multiple agents increases complexity.” — Sandipan Bhaumik

---

[22:25](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1345s) Pillar 5: Enterprise Governance, PII Detection, and Change Management

Pillar five shifts focus to AI-specific governance. Sandipan clarifies that standard enterprise data governance is merely baseline table stakes; production AI demands an entirely separate layer of operational controls, auditability, and risk mitigation. Regulatory compliance requires end-to-end audit trails that capture every user interaction, connection attempt, tool execution, and intermediate reasoning step. Before prompts ever reach an LLM, a strict pre-validation layer must scrub personal data using named entity recognition (NER) and regular expressions. In real-world testing with his retail banking client, this filtering layer caught 47 PII breaches before the system ever saw production traffic.

Furthermore, enterprise rigor must extend to prompt versioning and foundational model lifecycle management. Tweaking a prompt cannot simply be an undocumented Git commit pushed directly by a developer; prompts dictate application logic and must follow the same formal change management, automated testing, and release gating as production source code. Similarly, when LLM vendors release upgraded model checkpoints, teams cannot assume higher leaderboard scores translate to better internal performance. Public third-party evaluation benchmarks rarely correlate with proprietary enterprise domains. Organizations must maintain private evaluation suites to test model updates against their own data, preserving the architectural agility to swap providers rather than suffering lock-in to a single vulnerable upstream dependency.

- **Auditability & Traceability:** Immutable logging of all user requests, agent actions, and downstream integration calls for regulatory oversight. - **PII Guardrails:** Front-line NER and regex filters that successfully intercepted 47 PII leaks during pre-production validation. - **Prompts as Code:** Rigorous version control and peer-reviewed change management applied to all operational prompts. - **Model Risk Mitigation:** Continuous regression testing on internal benchmark datasets whenever upstream foundation models are updated or replaced. - **Platform Packaging:** Databricks incorporates these foundational governance, tracing, and evaluation pillars natively into its Agent Bricks framework to provide out-of-the-box enterprise readiness.

> “In our example, the work that I was doing with the customer that I mentioned, we already detected 47 PII breaches during the testing phase by applying this layer.” — Sandipan Bhaumik

> “You have to treat prompt versioning as change management in enterprise grade solution.” — Sandipan Bhaumik

---

[24:35](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1475s) Case Study: Rescuing an £85,000 Failed Retail Bank Chatbot PoC

To illustrate how these engineering pillars operate in practice, Sandipan walks through an engagement with a major retail bank that had spent approximately 18 months attempting to deploy a customer-facing conversational agent. The bank handled around 20,000 customer service calls every month. An internal audit revealed that roughly 60% of these calls were routine, low-complexity inquiries—such as checking current account balances or asking standard questions about overdraft fees—that tied up expensive human agents unnecessarily.

In an effort to deflect this call volume, the bank spent £85,000 over six months building a proof of concept (PoC) with an external partner. The project stalled completely. In pre-production and staging, the chatbot continually failed, yet none of the engineers or business stakeholders could diagnose why it was failing, measure its actual accuracy, or establish who was accountable when downstream integrations broke. Sandipan’s team took over the engagement under an aggressive eight-week reset timeline. Rather than rushing into prompt engineering or debating foundation models on day one, they anchored the project to explicit, measurable business outcomes: deflect 60% of simple tier-one customer queries with a target accuracy rate of at least 85%, supported by strict latency guarantees.

- **Baseline Call Volume:** ~20,000 customer service calls per month entering the bank's contact center. - **Automation Opportunity:** 60% of incoming inquiries consisted of simple, repetitive tasks (account balances, overdraft policies). - **Initial Sunk Cost:** £85,000 and 6 months of development resulting in a failed PoC with zero observability and no clear accountability. - **Reset SLA Targets:** Autonomous handling of 60% of routine inquiries, an 85% accuracy benchmark, and strict latency bounds. - **Contrarian Schedule:** Model selection was deliberately deferred until week seven of the eight-week engagement.

> “The key difference in this project that we when we did is that we selected the model in week seven, like in a eight weeks POC.” — Sandipan Bhaumik

---

[26:07](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1567s) Weeks 1–2: Building the Living Evaluation Dataset and Pipeline

During the first two weeks of the rescue engagement, the team wrote no agent application code. Instead, they focused entirely on constructing an evaluation dataset and automated measurement pipeline. They curated an initial golden dataset of 200 real-world customer inquiries, pairing each prompt with the exact verbatim transcripts of how high-performing human bank agents resolved them. This established an empirical ground truth for tone, factual precision, and procedural correctness.

Next, they engineered an automated evaluation harness. Whenever an agent generated a response, the pipeline compared the candidate text against the ground-truth dataset, assigned an automated score, and flagged any output falling below the designated quality threshold for human inspection. When errors surfaced—whether due to ambiguous phrasing, missing context, or flawed tool invocations—engineers adjusted the prompt or tool schemas and immediately converted the failed interaction into a permanent test case. Sandipan stresses that an evaluation dataset is not a static benchmark frozen at launch; it is a dynamic, living asset that must expand continuously as new edge cases appear in production.

```mermaid flowchart TD accTitle: Automated Evaluation and Feedback Pipeline accDescr: An incoming query and agent response are scored against a golden dataset, routing low-scoring answers to humans and updating the test suite.

subgraph INGEST["1. Execution"] Q["User Question"] --> AG["AI Agent"] AG --> RESP["Agent Response"] end

subgraph EVAL["2. Automated Grading"] RESP --> COMP["Eval Pipeline"] GOLD[("Golden Dataset\n(200+ Cases)")] --> COMP COMP --> SCORE{"Score > Threshold?"} end

subgraph REMEDIATION["3. Human Review & Expansion"] SCORE -->|"Yes"| PASS["Approved Response"] SCORE -->|"No"| REV["Human Review"] REV --> FIX["Fix Prompt or Tool Schema"] FIX -->|"Append New Test Case"| GOLD end ```

- **Golden Ground Truth:** Compiled 200 verified human-agent interaction transcripts representing standard retail banking queries. - **Automated Feedback Loop:** Responses scoring below acceptance thresholds trigger human review and root-cause analysis. - **Continuous Hardening:** Every discovered regression or edge-case failure is converted into an automated regression test, expanding the evaluation suite over time.

> “The summary of that story is that your evaluation data set is a living system. You start with 200, maybe there is no correct number here, but once you start, as you start building in production, this is a living system.” — Sandipan Bhaumik

---

[27:45](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1665s) Weeks 3–8: Infrastructure Tracing, Late Model Selection, and Launch

Entering the middle weeks of the engagement, the team laid down the core infrastructure layer before evaluating models. Long before standards like the Model Context Protocol (MCP) existed, the team implemented direct API connections between the agent and the bank’s distributed databases. Crucially, they instrumented end-to-end distributed tracing across every database query and API hop. When mock customer load was applied during testing, this tracing fabric immediately exposed hidden bottlenecks, including duplicate API calls that inflated latency and directly damaged customer satisfaction scores.

By weeks seven and eight, with comprehensive tracing and the golden evaluation dataset fully operational, model selection became an objective benchmarking exercise rather than an emotional debate. The team ran candidate foundation models across their curated test suite, calculated empirical accuracy scores against the expected outputs, and selected the winning model in a matter of days. With observability, automated evaluation, and governance guardrails unified into a visible, measurable, and accountable system, the bank successfully launched the agent into production. Six weeks after deployment, the operational metrics—accuracy, deflection rate, latency, and CSAT—were meeting targets, positioning the infrastructure to immediately catch subtle drift when policy changes loomed.

- **Infrastructure First:** Built direct database API query integrations with distributed trace capture prior to selecting any LLM. - **Immediate Defect Detection:** Trace telemetry surfaced redundant, duplicate API calls that had been degrading response times and CSAT. - **Empirical Model Selection:** Replaced subjective model debates with empirical test runs against the evaluation harness, completing model selection rapidly in week seven. - **Production Readiness:** Integrated evaluation, tracing, and governance layers to ensure the production deployment remained visible, measurable, and auditable from day one.

[29:41](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1781s) Post-Launch Production Incident: Stale Embeddings and Dropped CSAT

Sandipan Bhaumik illustrates the critical value of production observability by walking through a real incident that occurred after the retail bank’s chatbot went live. The bank altered its interest rate policies and notified its user base through emails and push alerts within the mobile banking application. Naturally, customers navigated to the in-app chatbot to ask follow-up questions about how the rate changes impacted their accounts. The chatbot, however, failed to provide accurate updates, prompting a wave of thumbs-down ratings and a noticeable slump in customer satisfaction (CSAT) scores.

Without an active telemetry and measurement system, diagnosing such a silent degradation would have been nearly impossible. Because the engineering team had implemented end-to-end tracing and real-time evaluation metrics, they immediately flagged the drop in CSAT and inspected the execution traces. The inspection revealed that the agent’s retrieval mechanism was retrieving an outdated policy document; the newly announced rate documentation had never been embedded and indexed into the vector database. Bhaumik emphasizes that resolving the bug was straightforward once spotted: “Because it did not come through, it was giving it stale answers. And that's when we went and fixed that. But it it was all possible because we built that those systems that that led us to to detect this.”

```mermaid flowchart TD accTitle: Production Incident and Recovery Flow accDescr: Flowchart showing how an interest rate policy change led to stale answers, user feedback, detection via tracing, and resolution.

subgraph Trigger ["1. Bank Policy Update"] A["Bank updates interest rate policy"] --> B["Customers notified via email and mobile app"] B --> C["Customers query chatbot for details"] end

subgraph Failure ["2. Silent RAG Failure"] C --> D["New policy document missing from vector DB"] D --> E["Chatbot retrieves stale policy embeddings"] E --> F["Incorrect answers served to users"] end

subgraph Remediation ["3. Telemetry and Fix"] F --> G["Thumbs-down feedback causes CSAT drop"] G --> H["Eval dashboard and tracing flag stale retrieval"] H --> I["Re-embed new policy into vector DB"] end ```

[30:53](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1853s) The Production Incident Playbook: Detect, Diagnose, Contain, and Fix

To ensure teams are prepared for runtime failures rather than caught off guard, Bhaumik introduces the production incident playbook, an operational standard often omitted in enterprise generative AI initiatives. “This is something that a lot of us tend to miss when we work in AI projects. And this playbook is basically a definition of what needs to happen when things fail in production,” he states. The playbook establishes a structured four-stage response cycle: detect via evaluation dashboards, diagnose root causes through granular tracing, contain runtime damage, and implement permanent fixes via regression test suites.

During containment, teams must maintain the ability to roll back faulty prompt versions immediately, divert failing conversations to human operators, or trigger programmatic fault-tolerance patterns. Bhaumik references architectural patterns such as saga patterns, compensating transactions, and circuit breakers to handle multi-agent cascade failures cleanly. For the remediation phase, engineering teams examine LLM-as-a-judge reports and test case logs to diagnose the defect, deploy the fix, and incorporate the failing scenario back into their living evaluation suite. Finally, Bhaumik insists that this entire workflow must be plugged directly into established IT Service Management (ITSM) platforms, ensuring automated alerts reach on-call engineers without disrupting downstream systems.

- **Detect:** Automated alerts triggered by real-time eval dashboard anomalies and CSAT drops. - **Diagnose:** Inspect agent execution traces, tool invocations, and retrieval chunks. - **Contain:** Roll back prompts, deflect interactions to human agents, or activate circuit breaker and compensation patterns. - **Fix:** Debug using LLM-as-a-judge reports, update the evaluation test library, and retrain or reprompt. - **Enterprise Integration:** Connect triggers to enterprise ITSM systems to automate escalation paths.

[32:46](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1966s) What to Do Tomorrow: Grounding Success in Business Realities

Turning to practical implementation steps that attendees can take immediately, Bhaumik advises engineers to resist the temptation to jump straight into model selection or complex agent frameworks. Instead, any new project must begin by formulating a clear, non-technical definition of what success looks like to the business. Teams should assemble a curated collection of representative user questions alongside explicit examples of what constitutes an ideal, acceptable answer.

Once this ground-truth sample is collected, developers can write simple, lightweight Python scripts to build an automated evaluation harness. Whenever an agent generates a response, this pipeline automatically compares the model's output against the reference dataset, establishing a baseline measurement before real users ever interact with the system.

[33:26](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=2006s) Three Hard-Won Lessons in Evaluation and Governance

Reflecting on field deployments with enterprise clients, Bhaumik details three operational lessons that are frequently overlooked until projects hit production scale. The first lesson centers on test case library governance. Because an evaluation dataset is a living system that constantly accumulates regression cases, it rapidly becomes unwieldy without formal ownership. Bhaumik recommends categorizing every row by business or technical domain—such as tagging authentication failures where an agent neglected to request login credentials under a dedicated "security" bucket. Clear categorization ensures that whenever an incident occurs, engineers can isolate and run only the relevant slices of the test suite.

The second lesson addresses prompt versioning discipline. While engineering teams routinely commit prompt templates into Git repositories, commit histories are frequently polluted with vague or meaningless messages. Bhaumik stresses that prompt changes require strict commit governance: “You have to put governance around what kind of commit messages you are putting in when you're changing these prompts because you need to understand when a prompt was changed, for exact what reason it was changed, right? What was the failure that caused this prompt to be changed? What kind of failure would it address and what would it correct, right? In the next version. That needs to be documented.”

The third lesson targets the escalating cost of Layer 3 behavioral evaluations. Evaluating complex agent behaviors, tool selection, and argument parsing across hundreds of test rows consumes significant API tokens and compute budgets. If CI pipelines trigger a full 300-to-500-row behavioral evaluation on every single prompt commit, testing costs quickly spiral out of control. Bhaumik recommends establishing tiered CI checks: run a fast, inexpensive subset of the evaluation dataset on feature branch commits and pull requests, reserving the exhaustive, expensive behavioral test suite for merges into the main branch.

- **Test Suite Governance:** Assign explicit ownership to the test library and classify rows into structured categories (e.g., security, compliance, retrieval, tool execution). - **Audit-Ready Commit Messages:** Require commit metadata detailing the exact incident, the observed failure mode, and the intended corrective behavior for every prompt modification. - **Tiered CI/CD Evals:** Run lightweight smoke tests on feature branch PRs; execute the full 300+ row Layer 3 evaluation suite exclusively on main branch merges to optimize operational spend.

[35:58](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=2158s) Tooling Artifacts and Closing Remarks

Bhaumik concludes his talk by presenting downloadable resources via on-screen QR codes to help practitioners implement these production patterns. The provided repository includes enterprise evaluation checklist templates, sample incident playbooks, and an implementation guide for configuring telemetry and tracing using open-source frameworks. This setup allows developers to establish tracing in staging environments prior to committing to commercial vendor tooling. He invites attendees to follow his weekly newsletter, where he regularly documents architectural patterns and operational lessons derived directly from deploying enterprise AI agents in the field, before closing to audience applause.

## Notable quotes > [07:33](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=453s) Evaluation is basically specification for your AI system. — Sandipan Bhaumik

> [04:54](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=294s) Before touching any code, before discussing about any models, any features, you have to think about when we build this system, how do we measure? — Sandipan Bhaumik

> [15:06](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=906s) Data was always built for humans, and humans are always forgiving. You find the wrong data in a report, you just go and ask someone to correct it. Agents don't forgive you, right? Agents will go, find it wrong, they'll give you the wrong answer confidently. — Sandipan Bhaumik

> [22:25](https://www.youtube.com/watch?v=ObTPqBGsEbA&t=1345s) You have to treat prompt versioning as change management in enterprise grade solution. — Sandipan Bhaumik

## Mentioned Sandipan Bhaumik, Databricks, Amazon Web Services, GPT, Claude, MLflow, Delta Lake, Unity Catalog, Delta Sharing, Mosaic AI, Databricks Genie, Agent Bricks, LangChain, CrewAI, Model Context Protocol

## Video description A retail bank spent £85,000 over six months on a chatbot PoC that could not reach production. No one could explain why it was failing. When Sandipan Bhaumik's team got involved, they picked the model in week seven of an eight-week engagement — the first six weeks went to evaluation data, tracing infrastructure, and a measurement pipeline. Six weeks post launch, when the bank updated its interest rate policy and customer satisfaction dropped, the tracing system caught the cause: the new policy document had not been reembedded and the agent was serving stale answers.

The talk covers the five pillars he built from that and similar engagements: evaluation (define success numerically before touching code), observability (trace every agent decision — European regulators require it), data foundation (agents do not forgive bad data the way humans do), multi agent orchestration patterns, and governance (47 PII breaches caught in testing before launch). The evaluation data set is a living system, not a fixed benchmark. The production incident playbook connects all five.

Speaker info: - https://www.linkedin.com/in/sandipanbhaumik

---

Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/ObTPqBGsEbA-the-production-ai-playbook-deploying-agents-at-enterprise-scale

Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):

- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney

---

[Source](https://www.youtube.com/watch?v=ObTPqBGsEbA)

Reading summary

Sandipan Bhaumik argues that enterprise AI initiatives routinely fail because organizations succumb to a 'model-first' trap, neglecting the foundational evaluation, observability, data architecture, orchestration, and governance required for production environments. The Model-First Trap and the Three Gaps (1:52) Executive pressure often leads teams to debate foundation models like GPT or Claude before understanding operational realities. According to Bhaumik, systems developed on predictable datasets inside sandbox environments rapidly collapse once deployed to real users. This premature push squanders engineering bandwidth and capital without delivering return on investment. A Three-Layer Architecture for Continuous Evaluation (7:33) Bhaumik asserts that evaluation must serve as the formal specification for an AI system, establishing strict numerical metrics such as query deflection rates and false-positive thresholds before code is written. Mature systems deploy three operational layers: deterministic checks using regular expressions and classic ML for PII and formatting; semantic evaluation using LLM-as-a-judge to assess groundedness and safety; and behavioral evaluation to detect flawed tool interactions. Observability and the Two Enterprise Data Streams (11:25) Deploying agents in regulated sectors requires end-to-end tracing across every intent classification, database query, and reasoning step. Without granular traces, teams face regulatory non-compliance in jurisdictions like Europe and remain unable to verify agent behavior during customer disputes. Bhaumik notes that data preparation consumes roughly 60% of project time, requiring an operational split between 'question data' used to answer queries and 'tracking data' generated by agent telemetry. Case Study: Rescuing an £85,000 Banking Chatbot (24:35) To demonstrate this framework, Bhaumik details an engagement rescuing a retail bank's chatbot that handled roughly 20,000 monthly calls, of which 60% were routine inquiries. An external partner had spent £85,000 over six months on a stalled proof-of-concept that lacked observability and diagnostic capabilities. Bhaumik's team reset the engagement around an eight-week plan, deliberately delaying model selection until week seven. Incident Playbooks and Operational Governance (30:53) Production agents require formal incident procedures built on a four-stage cycle: detect via evaluation dashboards, diagnose through granular traces, contain via prompt rollbacks or human escalation, and fix by adding failing scenarios to permanent regression suites. Bhaumik highlights that prompt edits must follow strict commit governance detailing the root failure and intended behavior. Bhaumik concludes that enterprise agent deployments succeed only when organizations replace model obsession with rigorous measurement, structured tracing, and disciplined governance. However, managing the compute and token costs of comprehensive behavioral evaluations in continuous integration pipelines remains an ongoing operational challenge.

Source quotes

[7:33] Sandipan Bhaumik: Evaluation is basically specification for your AI system. [4:54] Sandipan Bhaumik: Before touching any code, before discussing about any models, any features, you have to think about when we build this system, how do we measure? [15:06] Sandipan Bhaumik: Data was always built for humans, and humans are always forgiving. You find the wrong data in a report, you just go and ask someone to correct it. Agents don't forgive you, right? Agents will go, find it wrong, they'll give you the wrong answer confidently. [22:25] Sandipan Bhaumik: You have to treat prompt versioning as change management in enterprise grade solution.

Original transcript

[0:07] [music] [0:15] >> All right. Um, thank you for joining my [0:17] session. [0:17] >> [applause] [0:20] >> Thank you, man. [0:21] Uh, I'm Sandy. Uh, I'm a technical lead [0:24] uh, for data and AI at Databricks. Um, [0:27] prior to working in Databricks, I worked [0:30] in Amazon Web Services uh, for 5 years [0:32] as a principal architect for data and [0:33] AI. [0:34] Uh, [0:35] in the past few years, I worked [0:36] extensively uh, building and scaling [0:39] data and AI platforms using distributed [0:41] systems and technology. [0:43] And in the past couple of years, [0:46] specifically, I've been working with [0:48] customers trying to figure out what we [0:49] do with this new AI technology. [0:51] Uh, when I say new AI, AI has been here [0:54] for a long time, but we all started [0:57] experimenting quite exponentially uh, in [1:00] in the past couple of years, right? And [1:03] I have learned a great deal of lessons [1:05] on from building demos and how to take [1:08] those demos to production working with [1:10] different customers in [1:12] uh, B2B software industries and then uh, [1:14] regulating industries like financial uh, [1:16] services. [1:18] So, in this session, [1:20] I want to share [1:22] a playbook, a framework that I put [1:24] together from lessons that I have [1:27] learned working in the trenches [1:29] uh, that you can take and apply on when [1:32] you think about how to put your AI [1:34] systems into production. And I think [1:36] this session is nicely placed in the [1:38] afternoon because what you can do now is [1:40] in this framework, you can fit the [1:42] different [1:43] um, knowledge, the knowledge that you've [1:45] gathered attending these different [1:46] sessions throughout the day and see [1:48] where they fit in each of these [1:50] you know, elements in the framework. [1:52] So, when I started 2 years ago, this is [1:54] the pattern I noticed in every customer [1:56] conversation, right? So, everyone wanted [1:59] to do something with AI. Uh, there was [2:01] immense pressure from the top to do [2:04] something, to build a demo. [2:06] And every conversation started with [2:07] let's choose the model, right? And it [2:09] was nobody's fault because the market [2:11] was like that. We were talking about [2:12] models, the models were new technology [2:14] for us, right? And every conversation [2:16] started, shall we use GPT? Shall we use [2:19] Claude? [2:20] You know, there was huge debate with [2:22] with within organizations. Then, you [2:24] would choose a model, you'll build some [2:26] features, offsets of over what features [2:28] to build for that application. [2:30] Uh, you would build that in a controlled [2:32] environment, so predictable data sets, [2:35] you know, um, limited scenarios, and [2:39] then it looked great as a demo, and then [2:41] leadership would get happy, they would [2:42] sign it off, and they'll put it into an [2:45] environment in a production environment. [2:48] Then, [2:49] after a few weeks, people would start [2:51] asking questions that what the hell is [2:54] AI doing? [2:55] Right? Why is it not answering the [2:57] questions the way we expected it to [2:58] answer when we were doing the demos? [3:01] Uh, it would result in not only [3:04] less you know, no realization in return [3:06] on investment, but also loss of money [3:08] and effort in building these demos that [3:10] can never scale to production. [3:15] Throughout these uh, meetings, I [3:17] gathered three insights that connect to [3:20] everything that you we are talking about [3:22] when thinking about taking uh, AI to [3:25] production. The first one is the [3:26] observability gap, right? When we use AI [3:30] and put it into production, if we can't [3:32] see what it is actually doing, if we [3:34] can't trace every decision that it's [3:35] making, it's no use in production. [3:38] Second is the evaluation gap gap. A lot [3:41] of these conversations that we were [3:42] doing, we were not actually thinking [3:44] about what is what is that one thing [3:47] that we are measuring. Yes, we talk [3:49] about accuracy, we talk about latency, [3:52] we talk about groundedness, but we were [3:55] not defining what is that exact thing [3:57] like that matters to the business, and [4:00] how can we build a system that can [4:01] continuously measure that, whether it's [4:04] improving, whether it's not improving, [4:05] like what what is that system that we [4:07] need to build. And that was that [4:08] evaluation gap that I noticed. And the [4:10] third is the governance gap. Like we [4:13] were not actually thinking what happens [4:15] when AI fails in production. Who's [4:17] accountable? Who do I go to when [4:19] something happens at 3:00 a.m. in the [4:21] morning, right? Who needs to own the [4:24] data assets that feed some AI responses? [4:27] What happens if AI [4:30] you know, uh, talks um, uh, um, [4:34] nonsense to a customer, right? What what [4:35] happens, right? So, there is no [4:37] accountability, no governance around it. [4:39] And these three insights led me to [4:42] build a framework [4:44] on how [4:46] I think AI should be taken to [4:48] production, and this has been [4:49] implemented across multiple customer [4:51] organizations, and I think this is [4:53] something that you can pick up from [4:54] here. [4:55] These are the five pillars, and these [4:57] are absolutely what you need to think [4:59] about even before starting a project, [5:01] right? Then you start build them [5:03] gradually, [5:04] preferably in sequence, but in real [5:06] life, I know that this sequence don't [5:08] work, but these are the pillars that you [5:10] have to know about and you have to think [5:11] about when start building. First one is [5:13] evaluation. [5:15] Before touching any code, before [5:16] discussing about any models, any [5:18] features, you have to think about when [5:20] we build this system, how do we measure? [5:23] What does success look like, and what is [5:25] that system that will help us [5:26] continuously measure what success looks [5:29] like for us? [5:31] Second is how do we trace each and every [5:34] decision that AI makes. It's not only [5:36] important for the performance of the AI [5:39] system, it is also important for the [5:40] regulators. In Europe or in a lot of [5:44] companies, especially in regulated [5:45] industry, [5:46] you cannot even onboard AI into [5:49] production without having tracing and [5:51] observability in place. [5:52] So, this is a must-have. [5:54] The third is the data foundation, [5:56] right? Uh, I I I I think of data [5:58] foundation in in two ways. One is the [6:01] question data, so that is basically [6:04] the data needed for the AI to answer [6:06] questions that users ask to it. So, it [6:09] could be your pre-training data, post [6:11] post-training data, data that you use [6:13] APIs to hook onto and and get to the [6:16] answer that the user needs. The other [6:18] one is the tracking data, related to the [6:19] tracing data in observability, but when [6:21] you think from the data foundation and [6:23] and uh, data strategy perspective, this [6:26] needs to be handled in this pillar [6:28] because you need a whole data strategy [6:30] now with tracing data, especially when [6:32] you run hundreds of agents in your [6:34] organization. [6:36] Fourth is orchestration. One agent would [6:39] work pretty well. You don't need to [6:40] think about orchestration. But when you [6:42] onboard five agents, the the complexity [6:45] increases exponentially, right? You will [6:48] have multiple coordination patterns [6:49] between these agents, they will need to [6:52] talk to each other in multiple different [6:54] ways, they will need to each wait for [6:55] each other's responses, there's a lot of [6:57] complexity that comes in. And that's [6:59] where orchestration patterns and [7:01] thinking about how you will orchestrate [7:02] your agents in a particular system [7:05] becomes really important. [7:06] Fifth is governance. This is where [7:08] you think about what happens when [7:10] something fails. Who's accountable? How [7:12] do we govern data? How do we secure it? [7:14] How do we secure our systems? How what [7:16] do we make sure how do we make sure that [7:17] no one injects into our agent and leads [7:21] uh, to you know, misbehavior, right? Or [7:23] loss of reputation. [7:27] So, in the rest of the session, I will [7:28] dive a bit deeper into each of these [7:30] pillars and tell you how how you can [7:31] think about when you start working with [7:32] them, right? The first one is [7:34] evaluation. [7:35] Evaluation is basically specification [7:37] for your AI system. [7:39] You define success. As I mentioned, it's [7:41] not like [7:42] talking about accuracy. You have to [7:43] define it with numbers, like what [7:45] accuracy is is is good for your business [7:49] use case, [7:50] right? [7:51] Define it in numbers. [7:52] Uh, define what kind of you know, false [7:54] positives you can handle. What should be [7:57] the deflection? So, this is this is an [7:58] example from a [7:59] a retail chatbot, right? A banking [8:01] chatbot where [8:02] when you implement a chatbot with an AI [8:04] agent, one of the main goals is to [8:07] deflect simple queries [8:09] um, to the agent so that a human agent [8:12] don't need to uh, deal with them, right? [8:15] And so, you need to uh, [8:16] you need to track those queries and [8:18] track those numbers and put that system [8:20] in place. Second is building those test [8:23] test cases, like the evaluation data [8:25] set. You've heard about golden data sets [8:26] in evaluation. Talk with the domain [8:28] experts and find what is actually [8:30] happening in real life on the ground. [8:32] Like what answer would a support human [8:34] support agent [8:36] um, give to a customer on a particular [8:38] question. Collect those information. [8:40] What happens in gray areas, in edge [8:42] cases, like what happens when a human [8:45] sees a customer asking a confusing [8:46] question, right? Collect those into a [8:48] data set, and then automate your AI [8:52] testing, right? So, you put a question [8:54] to AI, it answers, take that answer, [8:57] compare against the test set, and [8:59] automate this whole pipeline so that [9:01] when you put AI in production, that [9:03] pipeline can actually take live [9:04] responses and evaluate against the test [9:07] data set that you're building, and then [9:09] give you the result in terms of how AI [9:12] is performing against those numbers and [9:14] the goals that you've defined. [9:17] When we talk about evaluation, [9:19] there are three main layers that I see [9:22] appear across organization, and this is [9:24] an architectural decision that you need [9:26] to make when you build these evaluation [9:27] systems, right? The first layer is [9:29] deterministic. These are the easy stuff, [9:31] like you know, checking formats, you [9:33] know, checking email formats, phone [9:35] formats, the regular expression things [9:37] that we have already been doing with our [9:39] coding systems, right? The uh, the the [9:41] other other is like you know, you could [9:43] use a classic ML models for name entity [9:46] recognition to for intent [9:47] classification, [9:49] for understanding what is first name, [9:52] last name, PII detection, etc. [9:54] So, the these these are easy stuff, [9:56] cheap stuff, you should get them out of [9:57] the way. We have already been doing this [9:59] for years. The second layer is the [10:01] non-deterministic semantic stuff, all [10:03] right? This is where groundedness comes [10:04] in. This is where we implement [10:06] technologies like LLMs judges. We all [10:08] know what LLMs judges are, right? [10:11] Everyone? Okay, I see a lot of nods. So, [10:14] um [10:15] Again, this This is a pretty simple [10:18] version of how a uh how a prompt would [10:21] look for an LLM as a judge. Um [10:24] So, with LLM as a judge, you you you use [10:27] a separate LLM from the primary LLM to [10:29] judge the response of the primary [10:32] model. [10:33] And when you do that, you tell the [10:35] secondary, the judge model, [10:37] on how it should uh judge the primary [10:40] model's output. So, it could be around [10:41] safety, groundedness, you know, [10:44] relevance to the answer, etc. etc., [10:46] right? Again, that can feed from a lot [10:49] of uh the evaluation data set that you [10:51] have created, right? To look at what are [10:53] the expected answers, and then it can [10:55] check against that. This is a sample [10:57] prompt on how these things work, but I'm [10:59] sure you've attended some of these [11:00] sessions where you've seen vendors doing [11:02] this automatically at scale. Uh for [11:05] example, in Databricks we in MLflow [11:06] you'll find automatic LLM as judge, uh [11:09] where you can create these custom LLM as [11:11] judges that run automatically on traces. [11:15] That's your second layer. The third [11:16] layer is behavioral, right? This is [11:18] where uh you think about a tool calls, [11:20] like is our agents calling the right [11:22] tool? Are they getting into loops? So, [11:25] for example, um you know, the first [11:28] layer you you can have a user ask a [11:29] question, "What is my account balance?" [11:31] And [11:32] you could go and check that, okay, this [11:34] there is no deterministic problem with [11:36] it. The seman- the agent answered right, [11:39] that, "Okay, your account balance is [11:40] this many dollars." [11:42] And that was right, and you can see this [11:43] is this is right, but when you go into [11:45] the behavioral checks, you will see that [11:46] the agent uh actually made three calls [11:49] to the database to find that answer. [11:52] Right? And that is because it was doing [11:54] making duplicate calls for whatever [11:55] reason. [11:56] Calls failed, [11:58] you know, calls did not work, it went [11:59] and retried and stuff like that. Now, [12:01] three API calls in demo environment is [12:04] fine, but in production, when you get [12:05] thousands of queries from users every [12:07] day, and there's like duplication in API [12:09] calls, that's an expensive operation. [12:11] And that's where you need to think about [12:12] behavioral evaluation. And this layer is [12:15] very, very important. I see a lot of [12:17] organizations, a lot of teams miss them [12:19] when when talking about this. [12:25] The second layer is observability, [12:27] right? Uh and in this pillar, what we're [12:30] talking about tracing, right? So, you [12:31] collect [snorts] all the decisions that [12:34] an agent is making. So, I want to [12:35] explain this with a scenario here, [12:37] right? And this is a scenario from an [12:39] actual project I worked on with a [12:41] banking uh retail retail banking [12:43] chatbot. Now, obviously, if you've seen [12:45] tracing data, it's not as beautiful as [12:47] this slide, right? So, I've simplified [12:49] it and made it beautiful for this slide. [12:51] But what this slide says is basically, a [12:52] user comes in and says, uh "You know, I [12:55] have been charged an overdraft fee, can [12:56] you waive it for me?" Because the user [12:58] thinks that the customer thinks that [13:00] that is not legitimate. [13:01] So, the agent does an intent [13:03] classification, [13:05] and you all you know about this because [13:06] you've enabled observability, you're [13:08] capturing traces, and you're actually [13:09] seeing what the agent is doing, right? [13:11] What AI is doing. [13:12] Intent classification, it is done, this [13:15] it took this many seconds, this was [13:16] confidence score. Then it goes and [13:18] connects to the customer's account, [13:19] maybe in a database, a customer [13:21] database, call calls an API, connects to [13:23] the customer database, gets the account [13:24] details. [13:25] It retrieves policy documents. It checks [13:28] from a rag vector database, [13:30] um what is uh what is the policy around [13:33] overdraft, right? Is what the customer [13:36] claiming is legitimate? So, it checks [13:38] for policy documents. Then it goes and [13:40] does a reasoning on what should be [13:43] uh [13:44] you know, responded to the customer, and [13:45] then it does some final guardrail [13:47] checks, and responds to the customer. [13:49] Now, if you did not set up a system that [13:52] helps you look visualize all of these [13:55] traces, [13:56] when the customer comes to you and [13:58] raises a dispute, [13:59] you have no way to check what the AI [14:01] did. [14:03] Right? You have nowhere to go, and you [14:04] end up saying that I don't have have any [14:06] idea. Let's Let's give the customer a [14:08] discount or something, and then make [14:09] them happy. [14:11] So, this is why you need this, and this [14:13] is why regulators are are are basically [14:15] mandating, because otherwise there's no [14:17] production system if you cannot do this [14:18] kind of stuff. [14:22] So, this is where um [14:24] you know, you you you detect this the [14:26] example that I gave around duplicate API [14:28] calls. This is where you start detecting [14:29] this stuff. So, when you when you enable [14:32] these traces, you can actually go and [14:33] see duplicate calls, and then take [14:35] relevant actions based on that. Not only [14:38] that, you can actually do that in online [14:39] monitoring. So, when it's happening in [14:41] production, in on you can set up online [14:43] monitoring, and at that point, if it is [14:45] doing duplicate calls, you can apply [14:47] fallback strategies. Or even if it is [14:49] doing a call that is failing, you can [14:51] actually go and apply a strategy where [14:53] it will say, "Okay, go and retry for [14:55] three times, not more than three times. [14:57] If it if it is more than three times, [14:58] then report somewhere, or pass it to a [15:00] human to take some action." [15:06] The third pillar is the most important [15:07] pillar, in my opinion, is the data data [15:09] data foundation, right? Uh [15:12] in my typical project projects, I spend [15:15] 60% of my time, uh and I see I see a lot [15:18] of organizations spending a lot of time [15:20] here, because [15:21] no one expected agents to come suddenly [15:24] in the market and start querying data. [15:27] Data was always built for humans, and [15:29] humans are always forgiving. [15:31] You find the wrong data in a report, you [15:32] just go and ask someone to correct it. [15:34] Agents don't forgive you, right? Agents [15:36] will go, find it wrong, they'll give you [15:38] the wrong answer confidently. [15:40] Right? And you wouldn't know what's [15:42] happening. [15:43] And this is why data quality, [15:45] setting the right data strategy, has [15:47] become so important for enterprises now. [15:50] I divide it into two sections. One is [15:52] the question data, as I was explaining, [15:54] like data needed for actually serving [15:56] the AI's uh outcome. [15:59] And the other one is the tracking data. [16:01] This is the observability data, the [16:02] tracing data I was talking about [16:03] earlier. [16:04] You need a proper plan on how you [16:07] collect this tracing data, and how you [16:10] serve it to auditors, to regulators, to [16:13] do online monitoring, to run LLM as [16:15] judges on the tracing, and everything [16:17] else, right? So, there it needs a proper [16:19] strategy on how you structure the schema [16:20] and everything on the tracing data. [16:24] Um [16:27] On Databricks, [16:28] um we [16:31] create a robust data foundation for our [16:33] customers [16:34] using uh some of the technologies that [16:36] we provide. If you don't know [16:38] Databricks, Databricks has been built on [16:39] some open-source technologies like [16:40] Apache Spark, MLflow, and Delta Lake. [16:44] Uh we provide a bunch of capabilities on [16:46] top of it. So, the blue layer at the [16:48] bottom is basically your cloud storage. [16:50] Databricks works on the three major [16:52] clouds, Google, AWS, Azure. [16:57] Okay. [16:59] I thought it was for me. [17:01] So, so once you once you store raw data [17:03] on your cloud storage, [17:05] uh the data is then um [17:08] um we we we bring in a a layer called [17:10] the Delta Lake layer, which uh which [17:12] basically brings in database-like [17:14] properties on top of your raw data. So, [17:16] you have got images, text files, video [17:18] files, or whatever. We we help you [17:20] create this um [17:22] you know, uh table-like structure on top [17:24] of it using manifest files, right? And [17:27] and we help you to uh incrementally load [17:30] data, do all of those um data management [17:32] tasks in a structured way. [17:35] On top of that, we bring in Unity [17:36] Catalog, which is a data catalog. Uh [17:39] with Unity Catalog, you can centrally [17:40] apply permissions on top of the data. [17:43] You can [17:44] um you can uh share the data using uh [17:47] Delta Sharing, but also uh what happens [17:50] with Unity Catalog is uh you you can [17:53] enable discovery and um you know, um [17:56] ownership, metadata tagging capabilities [17:59] at the catalog level. What that means [18:01] is, when you apply table a description, [18:03] column description, [18:05] uh tag columns uh like PII columns with [18:07] metadata, it becomes really easy for AI [18:10] to then get that context when it queries [18:12] these tables on top of Unity Catalog. [18:15] So, everything is governed at one layer [18:17] through Unity Catalog, and on on top of [18:19] that we bring in different uh [18:21] applications. So, whether it's AI [18:23] through Mosaic AI, so to build LLM, tune [18:26] LLM, or even build AI applications, we [18:28] bring in uh data warehousing [18:29] capabilities, BI capabilities, and um [18:33] uh and some of the other text-to-SQL [18:35] capabilities. We have got Genie that uh [18:37] helps you write natural language to do [18:39] SQL querying, etc. [18:42] And one application of that in the [18:43] observability and tracking tracking [18:45] data, as I was showing, is is this. So, [18:48] basically, think about when I was [18:49] talking about the tracking data [18:50] strategy. [18:52] Organizations, especially enterprises, [18:54] will not be running AI in just one [18:56] framework. They'll be using different [18:58] frameworks, [18:59] CrewAI, LangChain, etc. etc. They'll be [19:02] using different cloud platforms. [19:04] And once they do that, you need a [19:06] centralized layer of collecting that [19:08] tracing data, so that you can serve sev- [19:11] several use cases on the right hand [19:13] side. So, whether it's for operational [19:14] dashboarding, for first line support, uh [19:18] a lot of these uh first line [19:20] um first line of defense teams need [19:22] health monitoring uh sort of dashboards, [19:25] right? These teams can also write SQL [19:27] using Databricks Genie to do [19:29] text-to-SQL. But they can also build [19:31] Databricks apps using coding agents uh [19:34] to create common workspaces or custom uh [19:38] UIs that customers might need for [19:39] different uh different use cases. [19:42] And then we've got Agent Bricks and [19:44] MLflow that serves you uh LLM out of the [19:46] box LLM as judges, [19:48] and uh proactively monitor a [19:51] The idea is, [19:52] no no no matter where your AI runs, you [19:55] can create this kind of strategy [19:57] bringing in data in one common place and [19:59] serving uh different teams from one [20:01] shared location. [20:05] The fourth pillar is multi-agent [20:07] orchestration patterns. As I said, one [20:09] agent is good, multiple agents increases [20:11] complexity. That's where you start [20:13] thinking about, okay, what pattern is [20:15] good for my use case. The first one I [20:17] describe here is the orchestrator worker [20:19] pattern. Where you have one orchestrator [20:22] which orchestrates all the work, which [20:24] controls all the work from a centralized [20:26] plane, and then distributes this work to [20:29] different agents based on their [20:30] specialized skills. [20:32] And then every request goes through the [20:34] orchestrator, so you have got central [20:36] control. If something goes wrong, you [20:38] can go to the orchestrator logs and look [20:40] into them and see what has happened. [20:42] Right? So, that's the orchestration data [20:44] uh pattern. [20:45] There is this choreography pattern where [20:48] each agent is independent, they're [20:50] autonomous, they don't depend on an [20:51] orchestrator. All of them talk to a [20:54] message bus [20:55] and they listen to the events that they [20:57] are interested in. [20:58] Right? So, think about agents that are [21:00] independent of each other, right? They [21:01] can run parallelly. So, they are not [21:03] sequential, like one agent is not [21:05] dependent on another. So, they run [21:07] parallelly, they listen to the message [21:08] bus for the [21:10] for the events that they are interested [21:11] in. Maybe it's a trigger for, let's say, [21:13] a mortgage application, and it says, uh [21:16] you know, uh the mortgage application [21:17] agent [21:19] uh one of the agent uh looks customer [21:21] details, right? The other agent looks at [21:23] approval details and everything else, [21:24] right? They can work in parallel, and [21:27] the advantage it brings you is the [21:28] latency is reduced because they are not [21:30] dependent on an orchestrator and sending [21:32] messages back and forth. Right? So, this [21:35] is the choreography pattern. And the [21:36] third one is human in the loop, which is [21:39] where when an agent crosses a threshold [21:42] or serves below threshold a confidence [21:44] threshold, then a human is called in the [21:46] workflow to look into the pattern uh so, [21:49] look into the looking into what the [21:50] agent has done and then take action [21:53] based on that. [21:55] I have [21:57] done a deep dive video on multi-agent [21:59] orchestration pattern [22:01] uh for the online track of this [22:02] conference. Uh it's already on YouTube, [22:04] so you can look into it. I talk about [22:07] the real implications of when you think [22:09] about multi-agent patterns. [22:11] One is uh state management, the other is [22:15] fault tolerance, like what happens when [22:16] things fail, like how do you manage [22:17] them? I talk about different patterns. [22:19] And then talk about how how you think [22:21] about scaling them in large scale on [22:22] enterprises. [22:25] Pillar five is governance, right? Now, [22:27] here I'm not talking about data [22:29] governance at all. That's given, we need [22:31] that, right? From AI perspective, what [22:33] what what are we thinking about? [22:34] Regulatory, right? Audit trails, have we [22:36] got the trail of every action, every [22:39] user connection, every request, [22:41] everything that happens in the system? [22:43] Are we capturing everything? [22:45] Are we doing pre-validation of personal [22:48] information? Are we using name entity [22:50] recognition? The the easy stuff, the [22:52] rejects and all of those things, right? [22:54] In our example, the work that I was [22:56] doing with the customer that I [22:57] mentioned, we already detected 47 PII [22:59] breaches during the testing phase by [23:02] applying this layer. So, that's that's [23:04] really important. [23:06] Fourth is um prompt versioning. You have [23:08] to treat prompt versioning as change [23:10] management in enterprise grade solution. [23:13] It cannot be just change to a prompt and [23:15] commit to get. It has to be is it has to [23:18] go through proper change management [23:19] processes as you do with code. So, [23:21] basically treating prompt as code. [23:24] Third is model change management. So, as [23:26] models change, the model providers [23:28] upgrade these models, you have to have a [23:30] system to understand whether that [23:31] upgraded model will be good for your use [23:33] case, for your data. Right? Model [23:36] providers you will put evaluation [23:38] benchmarks on three uh benchmark uh [23:41] boards, [23:42] but those are not really useful when you [23:44] put them in your context, in your [23:46] enterprise. So, that's where these [23:48] evaluation data sets come in handy, [23:49] where you try these different models on [23:51] this evaluation data set and try to [23:53] understand which one performs better. [23:55] And that management needs to be done [23:56] because [23:58] from a risk perspective, you cannot [23:59] really rely on one single model. You [24:01] have to have the flexibility to switch [24:03] to different models and also test them [24:06] on your own data. [24:07] That management needs to be done. [24:12] Uh in Databricks, uh we have taken all [24:14] of these these points, these pillars [24:16] that I've been talking about into Agent [24:18] Bricks. We are building Agent Bricks to [24:21] make uh all of these operations out of [24:24] the box for you, [24:25] uh so that it's easy to implement [24:27] production grade AI applications on uh [24:30] on in in your enterprises. [24:35] So, I wanted to quickly touch upon a [24:37] case study, just to give you a uh a [24:40] flavor of how these things go, right? [24:42] So, when I was working with this client [24:45] um [24:46] they were a retail banking they were [24:48] building a retail banking chatbot you [24:49] know, one and a half 18 months ago. [24:52] Uh [24:53] their their problem the the problem they [24:55] wanted to solve is they had got around [24:56] 20,000 odd calls per month from [24:58] customers on their chatbot. They wanted [25:01] to deflect they they they saw that there [25:04] were like 60% of them were simple [25:05] queries, what is my account balance, you [25:08] know, what do I do with my overdraft and [25:09] all of those stuff, like that can be [25:11] answered simply. So, they wanted to the [25:13] reliance on human agents for those [25:14] answers. So, [25:16] they identified those queries and they [25:18] wanted to automate them. Right? [25:22] They spent around 85K in 6 months doing [25:24] a POC which did not succeed. When we got [25:26] involved, we found those insights that I [25:28] was talking like no one knew why things [25:30] were failing when it was in production [25:32] when when when when it was [25:34] in production. No one could actually [25:37] measure [25:39] why [25:40] why it's not succeeding and no one could [25:42] actually understand who is accountable [25:44] for what when things go wrong. Right? [25:47] So, the goal we set for them is AI agent [25:50] handles 60% of user queries, right? [25:53] Which were simple user queries and then [25:55] a way to identify and track them. The [25:57] key difference in this project that we [25:59] when we did is that we selected the [26:00] model in week seven, like in a eight [26:03] weeks POC. [26:04] Right? And this is how it turned out. [26:07] For the week one and two, we built the [26:09] evaluation layer. We collected 200 [26:12] cases on their actual human agents [26:15] answering to their customers on simple [26:17] queries and understand how they are [26:19] responding to them. [26:21] We created that database. [26:23] Then we defined the success metrics. [26:24] What does success look like to you? So, [26:26] out of let's say 100 queries, you need [26:28] 60 queries or the 60% of the queries [26:31] that are simple queries [26:32] to be [26:35] uh to be handled by the agent, right? [26:36] They needed some sort of accuracy. So, [26:38] 85% it was around 85% accuracy target. [26:40] They needed latency, all of the [26:42] operational targets that you need. They [26:44] were there. [26:46] Then we created this automated [26:48] evaluation pipeline for them. And what [26:50] what I mean by that is an automated [26:51] system where you can capture a user's uh [26:54] a user's question and the AI agent's [26:56] response. You take that, compare that [26:59] against your evaluation data set. [27:02] You rate that, and if the rating is [27:04] below certain threshold, you get it [27:06] checked by a human. And you if if [27:08] something goes wrong, you make sure that [27:11] you find the solution. So, it could be a [27:12] change to the prompt, it could be change [27:14] to a tool calling system or something [27:15] else. Once you have done that, you add [27:18] that test case in the test data set. So, [27:20] that when it happens next time, the test [27:22] cases cases catch them. So, the the the [27:26] the summary of that story is that your [27:27] evaluation data set is a living system. [27:30] You start with 200, maybe there is no [27:32] correct number here, but once you start, [27:34] as you start building in production, [27:36] this is a living system. This will keep [27:39] growing. And the [27:40] and the bigger it grows, the better your [27:42] system will be. [27:45] In the second week, we talked we thought [27:47] about the foundational layer, right? So, [27:48] the the question data, we thought we [27:50] thought that, okay, [27:51] if you have to call the database, have [27:54] you got the API connections right? Have [27:55] you got a system in place that can trace [27:57] the API connections? Are those secure, [27:59] right? We were not talking about MCP at [28:01] that time. [28:02] Right? It was just direct API calls to [28:04] database to run queries. [28:08] Have you got the distributed storage? [28:09] Have you got the Have you Are you [28:11] collecting traces? [28:12] And this is where when when we started [28:14] testing after building these systems, we [28:16] could catch those duplicate API calls, [28:19] right? We could catch why customer [28:20] satisfaction was dropping and stuff like [28:22] that. [28:24] And then comes in week seven to eight, [28:26] we started talking about models. Now [28:28] that we had the evaluation data set, we [28:30] could run different models on that data [28:32] set to see the responses, compare them [28:34] against the expected responses, and [28:36] calculate a number on on on the [28:38] accuracy, right? That helped us to [28:41] decide which model to use. [28:44] Now, that decision didn't take long, [28:45] right? We in as I explained in the [28:48] introduction, like we spent weeks [28:49] debating on which model to use, [28:52] but when you took the other approach, [28:55] you can actually [28:56] uh do that in a very quick way. [29:00] So, [29:01] once once that's done, we we stitched [29:04] everything that I was talking about [29:05] around observability, evaluation, the [29:07] layers of evaluation. Once we had that [29:09] system that can make AI visible, [29:11] measurable, and accountable, that's when [29:14] we started launching it to production. [29:18] And that's when uh so, this is this is [29:19] the result uh six weeks post launch, [29:22] uh we we calculated the operational [29:24] metrics, of course, you know, the [29:25] accuracy, the deflection rate, the [29:28] response time, uh the customer CSAT. But [29:31] what's important here is [29:33] in few weeks time, when uh there was a [29:35] problem with uh so, one of the one of [29:38] the things that happened was that the [29:39] bank changed some uh [29:41] interest rate related policies. So, when [29:43] that when they changed the policy, they [29:45] actually sent emails to customers or [29:47] notifications in the application in the [29:49] in the in the mobile banking app about [29:51] the policy change. [29:52] But when the customers came and queried [29:55] on the chatbot for further questions, [29:57] they couldn't get the right answers and [29:59] they were like [30:00] putting thumbs down on the answers. So [30:02] they were getting this feedback, right? [30:03] So feedback decreased. [30:06] The problem with this kind of system, if [30:07] you did not have this measurement [30:08] system, is that you couldn't actually [30:10] know what's happening. [30:11] But because we had the measurement [30:13] system in place, [30:14] these [30:15] the drop in C set was detected, right? [30:18] Because we were getting negative [30:19] feedback from customers. [30:21] We could actually look into the tracing [30:22] decisions and see that the agent was [30:24] looking at a policy policy document that [30:27] was outdated. So the the new policy [30:29] document was not updated in the vector [30:31] database. The embeddings did not come [30:33] through. [30:34] Because it did not come through, it was [30:36] giving it stale answers. And that's when [30:39] we went and fixed that. But it it was [30:41] all possible because we built that those [30:43] systems [30:45] that [30:46] that led us [30:47] to to detect this. [30:53] Before you go, I I generally in these [30:55] sessions I share different [30:57] artifacts that you can take away. I have [31:00] a QR code at the end for you to download [31:02] and you will find multiple artifacts. [31:04] One of the important artifacts that I [31:06] want to talk about is the production [31:07] incident playbook. This is something [31:09] that a lot of us tend to miss when we [31:12] work in AI projects. And this playbook [31:15] is basically a definition of what needs [31:17] to happen when things fail in [31:19] production. [31:21] First, you detect using your eval [31:22] dashboard. [31:23] Then you diagnose using your tracing as [31:25] I explained. [31:26] Then you contain. So basically, you you [31:29] are versioning your prompts. Is there a [31:31] Is there is a If there is a problem with [31:32] the prompt, you you take that prompt [31:35] out, right? And start the changes, [31:37] deflect it to a human. Or in my [31:40] multi-agent orchestration video, I've [31:41] talked about multiple fault tolerance [31:44] failure recovery patterns [31:46] around saga pattern, compensation [31:48] pattern, [31:49] and circuit breaker pattern that you can [31:51] look into the video. I've explained them [31:52] in details on how you can handle them. [31:55] And then you use the test case library [31:56] to fix. So you look into LLM's judge [31:58] reports, you look into your evaluation [32:00] data set reports. [32:02] Then you fix your problem. Once you fix [32:05] your problem, you put those test cases [32:06] in your data set, right? And and create [32:09] that eval suite that is a living system [32:11] that will keep growing. [32:13] And [32:14] and and you you keep improving your AI [32:16] system based on that, right? But this [32:18] playbook needs to be in place. When it [32:20] runs in production, you will need to [32:22] integrate it with your ITSM system so [32:24] that it alerts the right person at the [32:26] right time. You know, a lot of these [32:27] organizations would have existing [32:30] ITSM systems, right? So which [32:32] which is used for alerting and you know, [32:35] making sure that the downstream systems [32:36] don't get affected, etc. [32:38] So [32:39] once once you have this in place, you [32:41] can go and stitch it together to other [32:43] systems. [32:46] So what can you do tomorrow, right? [32:49] Start with If If you have a project in [32:51] mind, start with defining success. [32:53] Success not from the technical sense, [32:55] from the business sense. What it means [32:56] means for the business, right? Come up [32:58] with a few examples of what good answers [33:01] look like. [33:02] And and create a data data set of that. [33:04] And then build that pipeline using [33:07] simple Python code. See if you can [33:09] automate that so that when you run AI [33:11] and get some response, it can become it [33:13] can go and compare You can go and [33:15] compare the answer against that data set [33:18] and then that can be delivered [33:20] to to the to the customer. [33:26] Now, these are three lessons that I have [33:28] learned while doing these things with my [33:31] you you know, my easily miss. [33:34] The test case library, as I explained, [33:35] is a growing system. It will grow over [33:37] time. [33:38] And because it grows over time, you need [33:40] some sort of governance around it. You [33:41] need a owner, right? You need to You [33:44] need to figure out which test cases [33:48] relate to what kind of problem. So that [33:50] whenever you go back to it, you can [33:53] you can relate your answers to those [33:54] sort of problems, right? If it is a [33:55] security If it is login, so you can say [33:57] that the agent did not ask for login [34:00] credentials when the customer asked the [34:01] answer. And all those kind of issues can [34:03] be put under a security category within [34:05] that data set. So to categorize the rows [34:07] in your data set so that you can pick up [34:10] what changed and compare it with them. [34:12] The second is prompt versioning. Now, [34:14] when you start versioning prompts using [34:16] Git, you know, we all know when you put [34:18] Git message commit messages [34:20] tend to be [34:21] simple commit messages. But you have to [34:24] put governance around what kind of [34:26] commit messages you are putting in when [34:28] you're changing these prompts because [34:29] you need to understand when a prompt was [34:31] changed, for exact what reason it was [34:33] changed, right? What was the failure [34:35] that caused this prompt to be changed? [34:38] What kind of failure would it address [34:40] and what would it correct, right? In the [34:42] next version. That needs to be [34:43] documented. [34:45] Otherwise, it becomes difficult because [34:46] when you go back and look into prompt [34:48] versioning and look at different [34:49] versions and you cannot trace why why [34:51] those changes were made, then it becomes [34:53] difficult to track what's happening. [34:55] The third, the layer three evals, right? [34:58] So the behavioral evals that I was [35:00] talking about around tool calls and [35:01] stuff like that, they can be really [35:03] expensive as you grow your eval data set [35:06] as well. So when you have a wrong tool [35:08] call, for example, and you want to [35:10] correct that system, when you correct it [35:12] and run it against the eval data set, [35:16] you have to basically run it against [35:17] let's say if you have got 300, 400, 500 [35:20] rows in the data set, you have to run it [35:21] against them. And you do all the testing [35:23] again and again and again and again and [35:25] again, [35:26] that can cost you a lot of money. So you [35:27] have to put some governance around that. [35:29] So for example, when in your continuous [35:32] integration pipeline, when you do the [35:34] prompt [35:34] change, [35:36] you can actually put [35:39] some checks around [35:41] just just selecting a small subset of [35:43] the eval data set to do the testing. And [35:45] you only do the full test when you merge [35:47] to the main branch. [35:48] So you can put these kind of decisions [35:50] in place so that you can reduce cost [35:52] around [35:53] around [35:55] you know, expensive eval decision. [35:58] If you scan this QR code, it'll take you [36:00] to a Google Drive link where I have put [36:02] some examples on [36:04] some of these how these templates look [36:06] like, what evaluation checklist should [36:08] look like. [36:09] I've given you some guide on [36:11] set setting up tracing [36:13] with open source technologies [36:16] so that you can quickly set up some [36:17] tracing and start testing in the test [36:19] environment before you [36:20] decide on what kind of tools you want to [36:22] use. [36:25] Thank you very much for listening to me. [36:28] This is This QR code will take you to my [36:30] LinkedIn profile. So I share [36:33] I have a newsletter where I share this [36:35] kind of topics every week. So if you're [36:38] interested, you can join. It's free. [36:40] I basically share what I learn in in the [36:42] field working with customers, right? So [36:44] it might be useful for you. [36:46] Thank you very much. [36:47] >> [applause]