DagsHub Warns Enterprise AI Apps Stall on Premature Model Training

Why Your LLMs Won't Make It to Production: Best Practices for Customizing and Evaluating LLMs — Ai4

Ai4

Enterprise LLM deployments collapse in production because engineering teams overengineer complex architectures instead of relying on curated datasets, prompt engineering, and lightweight parameter tuning.

AI-assisted summary of Why Your LLMs Won't Make It to Production: Best Practices for Customizing and Evaluating LLMs — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 24, 2025, this presentation features Dean Pleban, Co-Founder and CEO of DagsHub. The talk explores why most enterprise LLM applications fail to transition from promising internal prototypes to real-world deployment, outlining core best practices for model customization and evaluation.

## TL;DR Many engineering teams fall into the 'prototype trap,' watching functional internal demos crumble in production under the weight of hallucinations, domain mismatch, compliance risks, and operational complexity. Rather than jumping straight to expensive full-model retraining or complex multi-agent frameworks, organizations should follow a disciplined customization ladder starting with prompt engineering, RAG, and lightweight parameter-efficient fine-tuning (PEFT/LoRA). Success hinges on prioritizing compact, high-quality golden datasets over noisy mass data. Finally, teams must embed evaluation as a day-one priority, combining deterministic guardrails, Small Language Models as judges, and human feedback loops to ensure production reliability.

## Atmosphere & style The presentation carries a pragmatic, grounded tone that cuts through generative AI hype with technical realism. Pleban speaks with direct, accessible authority, framing production pitfalls with relatable humor and hard-earned engineering lessons. The talk feels structured and clarifying, transforming the overwhelming complexities of enterprise AI deployment into an actionable, disciplined roadmap.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) Most enterprise LLM prototypes fail when exposed to real users due to hallucinations, domain mismatch, safety risks, and operational complexity. - [02:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=167s) LLM customization should be approached as a ladder, exhausting prompt engineering, few-shot examples, and RAG before attempting fine-tuning. - [02:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=167s) Pre-training proprietary foundation models from scratch is virtually never necessary for enterprise applications and usually signals a strategic mistake. - [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) Overengineering architectures with complex multi-agent frameworks too early often causes project failure; teams should validate simpler setups first. - [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) Data quality strictly trumps volume: hand-curating a compact set of roughly 400 golden samples is vastly superior to training on massive, noisy datasets. - [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) Parameter-efficient fine-tuning (PEFT / LoRA) modifying as little as 1% of model weights can drastically reduce errors without full-model retraining overhead. - [06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) Evaluation must begin on day one with a lean benchmark—even as few as five test cases—that every prompt or configuration change must pass. - [06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) Small Language Models (SLMs) can be leveraged as automated judges to achieve scalable quality evaluation at a fraction of frontier model costs. - [06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) Deterministic, programmatic rules should be used for structural and formatting guardrails rather than unnecessarily relying on additional AI calls. - [06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) End-to-end versioning of prompts, validation schemas, and model checkpoints is essential for diagnosing and resolving production anomalies.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) The Prototype Trap and Why LLMs Fail in Production

The generative AI landscape moves so rapidly that technical presentations require near-constant rewriting just to stay relevant. Across the industry, engineering teams repeatedly encounter the same frustrating pattern: building a "cute LLM prototype" that looks magical in internal testing, only to watch it completely crumble the moment real users interact with it. Industry data indicates that the vast majority of enterprise LLM initiatives stall out and never make it to production. When deployments meet reality, they get derailed by unexpected user inputs, sudden API hiccups, and runaway inference costs.

When dissecting why these projects fail to see the light of day, four primary obstacles emerge:

- **Hallucinations:** While factually fabricated answers might be laughed off during an internal demo, they become disastrous in front of paying users with strict expectations; you cannot realistically tell customers, "it's going to work like 10% of the time, and you figure out which 10%." - **Domain Mismatch:** Off-the-shelf foundation models produce generic responses; an enterprise financial application cannot speak generic GPT-4 when the use case demands the rigor and terminology of Goldman Sachs, particularly in regulated environments like banking and the public sector. - **Safety and Compliance Risks:** If an LLM blurts out offensive content, leaks confidential information, or executes destructive actions like deleting database records, corporate legal teams will immediately pull the plug. - **Operational Complexity:** Teams frequently underestimate the operational burden of production machine learning, including architectural downtime, traffic scaling, application integration, and latency management.

---

[02:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=167s) The Hierarchy of LLM Customization

A central mistake teams make is assuming that addressing domain gaps requires training an expensive, custom model from scratch. LLM customization should instead be approached as a ladder, where teams begin at the lowest rung and only ascend when simpler methods fail to meet concrete requirements. If an organization concludes that it must train a proprietary foundation model from scratch, it has almost certainly made a strategic misstep along the way by skipping simpler, more cost-effective layers.

```mermaid flowchart TD accTitle: Hierarchy of LLM Customization Ladder subgraph Frontier["Frontier Zone"] L5["RLHF & Pre-training from Scratch"] end subgraph Intermediate["Intermediate Adaptation"] L4["Full-Model Fine-Tuning"] L3["PEFT / LoRA (Train ~1% of Weights)"] end subgraph Baseline["Baseline Foundations"] L2["Retrieval-Augmented Generation (RAG)"] L1["Prompt Engineering & Few-Shot Examples"] end

L1 --> L2 L2 --> L3 L3 --> L4 L4 --> L5 ```

- **Prompt Engineering & Few-Shot Prompting:** The baseline layer of the hierarchy involves prompt tweaks and providing high-quality contextual examples, which solves a surprisingly large percentage of enterprise use cases. - **Retrieval-Augmented Generation (RAG):** When an LLM lacks up-to-date facts or proprietary knowledge, bolting on a retrieval system to fetch reference documents on the fly is far more effective than retraining. - **Parameter-Efficient Fine-Tuning (PEFT / LoRA):** When models require new behavioral patterns or stylistic adaptations, Low-Rank Adaptation allows teams to update as little as 1% of the model parameters cheaply and quickly. - **Full Fine-Tuning and Pre-Training:** Full-parameter updates and building ground-up foundation models are rarely necessary for enterprise applications. - **Reinforcement Learning from Human Feedback (RLHF):** The top tier of the hierarchy remains the domain of frontier AI research labs and is unnecessary for typical product teams.

---

[04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) Best Practices: Avoiding Overengineering and Prioritizing Data Quality

Drawing from lessons across top industry practitioners, teams must resist the urge to overengineer their architectures before validating the basics. While complex multi-agent frameworks and deeply chained tool pipelines are popular, jumping straight to high complexity often leads directly to project failure. Starting with robust prompt engineering and adding tooling incrementally as real requirements dictate is the most reliable path to deployment.

The timeless principle of "garbage in, garbage out" applies acutely to generative models: a compact, meticulously curated dataset outperforms a massive, noisy dataset 99% of the time. Eugene Yan from Amazon advocates for hand-curating at least 400 "golden samples" before beginning evaluation work. While manually inspecting and labeling data feels tedious compared to outsourcing or synthetic generation, direct contact with the data often separates successful production deployments from failed experiments.

When fine-tuning is truly required, teams should leverage parameter-efficient methods. In one Microsoft implementation for an on-device educator tool, engineers plugged LoRA adapters into the architecture, modifying only 1% of the model weights. This lightweight intervention reduced rejected answers by 75% and improved quality ratings nearly fivefold, avoiding the immense computational cost and latency of retraining the entire model.

---

[06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) Evaluation, Hybrid Feedback Loops, and Guardrails

Evaluation must be treated as a first-class citizen from day one rather than an afterthought. The evaluation team at OpenAI recommends starting lean by establishing a benchmark of just five representative test cases, ensuring that every proposed prompt modification or configuration change passes this baseline before reaching staging or production. Financial institutions like Morgan Stanley have realized significant performance gains simply by embedding strict automated evaluation checkpoints into their deployment cycles.

```mermaid flowchart LR accTitle: LLM Evaluation and Guardrail Pipeline subgraph InputStage["Input & Generation"] P["Prompt + Context"] --> M["Model Inference"] end subgraph SafetyStage["Verification & Guardrails"] M --> G["Deterministic Guardrails\n(Format & Regex Validation)"] G --> J["Judge Layer\n(SLM / LLM as a Judge)"] end subgraph FeedbackStage["Feedback & Observability"] J --> U["User Interface\n(Thumbs Up / Down)"] U --> V["Versioned Artifacts\n(Prompts, Datasets, Models)"] end ```

To maintain performance over time, systems should blend automated evaluation with human-in-the-loop mechanisms:

- **Human-in-the-Loop Feedback:** Simple binary interfaces (thumbs up and thumbs down) help distinguish between foundational model failures and broader application-level bugs, grounding system iteration in real user experience. - **AI-as-a-Judge and SLMs:** Automated evaluation using LLMs as judges (such as practices developed at AWS) provides scalable quality checks; however, teams can drastically lower operational costs by deploying Small Language Models (SLMs) as judges instead of high-cost frontier models. - **Deterministic Guardrails:** Protective measures do not always require another AI call; Microsoft implemented hardcoded programmatic rules in an educational quiz generator to immediately toss out outputs that failed structural or formatting constraints. - **End-to-End Traceability:** Production robustness relies on rigorous versioning of prompts, validation schemas, and model checkpoints, ensuring engineers can pinpoint the exact cause of any production anomaly.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) building a "cute LLM prototype" that looks magical in internal testing, only to watch it completely crumble the moment real users interact with it. — Dean Pleban

> [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) it's going to work like 10% of the time, and you figure out which 10%. — Dean Pleban

> [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) an enterprise financial application cannot speak generic GPT-4 when the use case demands the rigor and terminology of Goldman Sachs — Dean Pleban

> [02:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=167s) LLM customization should instead be approached as a ladder, where teams begin at the lowest rung and only ascend when simpler methods fail to meet concrete requirements. — Dean Pleban

> [02:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=167s) If an organization concludes that it must train a proprietary foundation model from scratch, it has almost certainly made a strategic misstep along the way by skipping simpler, more cost-effective layers. — Dean Pleban

> [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) While complex multi-agent frameworks and deeply chained tool pipelines are popular, jumping straight to high complexity often leads directly to project failure. — Dean Pleban

> [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) a compact, meticulously curated dataset outperforms a massive, noisy dataset 99% of the time. — Dean Pleban

> [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) direct contact with the data often separates successful production deployments from failed experiments. — Dean Pleban

> [06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) Evaluation must be treated as a first-class citizen from day one rather than an afterthought. — Dean Pleban

> [06:47](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=407s) teams can drastically lower operational costs by deploying Small Language Models (SLMs) as judges instead of high-cost frontier models. — Dean Pleban

## Mentioned Dean Pleban, DagsHub, Ai4, Goldman Sachs, GPT-4, Eugene Yan, Amazon, AWS, Microsoft, OpenAI, Morgan Stanley, LoRA, PEFT, RAG, RLHF

## Video description Dean Pleban, Co-Founder & CEO, DagsHub

Abstracts

With LLMs making an impact in every industry and product, almost every company is racing to create their LLM application, whether it's an internal tool to boost productivity, or a chat interface for their product. Most of these will fail to make any impact!

If your product or domain isn't fully generic, you'll probably hit a lot of challenges that make deploying your LLM application likely to fail. meaningful risk. In this talk, I'll share some of the most important best practices to customize and evaluate LLMs, so your application makes it to the real world.

[Source](https://www.youtube.com/watch?v=CXWtyxz0Uso)

中文简报

DagsHub警告：过早模型训练致企业AI应用陷入停滞 企业LLM部署在生产环境中频频崩溃，原因在于工程团队过度设计复杂架构，而非依托精选数据集、提示词工程与轻量级参数微调。 许多工程团队陷入了“原型陷阱”，眼看着内部正常运行的Demo在幻觉、领域不匹配、合规风险及运维复杂度的重压下在生产环境中瓦解。企业不应直接转向昂贵的全模型重训或复杂的多智能体框架，而应遵循规范的定制阶梯，从提示词工程、RAG到轻量级参数高效微调（PEFT/LoRA）循序推进。成功的关键在于优先选用精简、高质量的黄金数据集，而非嘈杂的海量数据。最后，团队必须在第一天就将评估列为重中之重，结合确定性护栏、小语言模型裁判和人工反馈闭环，以确保生产环境的可靠性。 大多数企业级大模型原型在面对真实用户时走向失败，主要受阻于幻觉、垂直领域不匹配、安全与合规隐患以及沉重的运维复杂度。 (0:04) 模型定制应遵循阶梯演进原则，应优先穷尽 Prompt 工程、Few-shot 示例与 RAG 检索增强，而从零预训练专属基座模型通常是严重的战略误判。 (2:47) 避免过早搭建复杂的多智能体架构，高质量数据远胜过海量杂乱数据，人工精选约 400 条黄金样本并结合仅修改 1% 权重的 PEFT/LoRA 微调即可大幅降低错误率。 (4:19) 模型评估必须从第一天起步，团队可先从 5 个核心测试用例构建基准，并结合确定性规则与轻量化小语言模型（SLM）作为自动裁判以降低评测成本。 (6:47)

Reading summary

According to Ai4, most enterprise LLM projects fail to survive deployment because teams stumble over hallucinations, operational complexity, and architectural overengineering rather than adopting an incremental customization hierarchy and early evaluation. Production Obstacles Beyond Internal Demos (0:04) Engineering teams frequently build functional internal prototypes that fail upon exposure to real-world traffic, unpredictable user inputs, unexpected API interruptions, and spiraling inference costs. Industry data shows that the vast majority of enterprise LLM initiatives stall out before deployment. Four primary barriers prevent systems from reaching production: factual hallucinations that paying users cannot tolerate, domain mismatches where generic foundation models lack industry-specific rigor, safety and compliance violations that trigger legal interventions, and underestimated operational complexity regarding latency, downtime, and scaling. An Incremental Customization Ladder (2:47) A critical strategic error is assuming domain gaps require training a foundation model from scratch. Teams should treat customization as a step-by-step ladder, ascending only when simpler, less costly options prove inadequate. The foundation begins with prompt engineering and few-shot examples, which resolve a substantial portion of enterprise needs. When models lack factual or private information, Retrieval-Augmented Generation (RAG) supplies reference context dynamically. For behavioral or stylistic adjustments, Parameter-Efficient Fine-Tuning (PEFT/LoRA) allows teams to adapt roughly 1% of model weights cheaply. In contrast, full fine-tuning, ground-up pre-training, and Reinforcement Learning from Human Feedback (RLHF) remain largely unnecessary for typical enterprise products. Curated Data Over Architectural Complexity (4:19) Teams often rush into complex multi-agent frameworks and deeply chained pipelines, leading directly to project breakdown. Successful implementations prioritize data curation over architectural complexity, acknowledging that small, high-quality datasets routinely outperform massive, noisy ones. Eugene Yan from Amazon recommends hand-curating at least 400 "golden samples" before initiating evaluation. In practice, focused adjustments yield outsized results: Microsoft implemented LoRA adapters on an on-device educator tool, modifying only 1% of model weights, which cut rejected answers by 75% and raised quality ratings nearly fivefold without the expense of full retraining. Lean Benchmarking, Guardrails, and Observability (6:47) Evaluation must be built in from project inception rather than added at the end. OpenAI's evaluation team advises starting lean with a benchmark of just five representative test cases to vet any prompt or configuration change. Financial institutions such as Morgan Stanley have achieved substantial performance gains by integrating automated evaluation checkpoints into deployment cycles. Long-term reliability depends on a hybrid defense: simple binary user feedback (thumbs up and down) to isolate model issues from app bugs, AI-as-a-judge systems utilizing Small Language Models (SLMs) to contain costs, deterministic programmatic rules to enforce formatting without secondary model calls, and strict versioning of prompts, schemas, and checkpoints to ensure end-to-end traceability. Moving generative AI from prototype to production requires abandoning premature architectural complexity in favor of baseline prompt improvements, curated datasets, deterministic safety rules, and continuous, multi-tiered evaluation.

精炼阅读摘要

行业数据显示绝大多数企业大语言模型（LLM）项目止步于原型阶段，未能在生产环境中成功落地。演讲指出，要跨越从内部演示到实际投产的鸿沟，工程团队应遵循自底向上的模型定制阶梯，坚守数据质量优先并抵制过度工程，同时构建包含硬性规则与自动化判别的综合评估体系。 原型陷阱与投产核心障碍 (0:04) 工程团队常常遭遇“原型陷阱”：在内部测试中表现惊艳的实验性LLM原型，在面对真实用户输入、意外的API故障以及不可控的推理成本时往往迅速崩溃。演讲分析认为，企业级LLM项目难以走入生产环境主要受阻于四大瓶颈：首先是幻觉问题，内部测试中尚可容忍的事实捏造在付费用户面前是灾难性的，无法要求客户自行鉴别输出真伪；其次是领域不匹配，通用基础模型的回答过于笼统，无法满足银行等强监管领域对高盛级严谨术语的要求。 此外，安全与合规风险也是导致项目被法务部门叫停的直接诱因，这包括模型生成攻击性言论、泄露私密数据或执行删除数据库等破坏性指令；最后，团队普遍低估了生产级系统在停机风险、流量扩展、系统集成以及延迟管理等方面的运维复杂性。 自底向上的模型定制阶梯 (2:47) 许多团队陷入的战略误区在于，一旦发现领域匹配不足，就误以为必须从头预训练专属基础模型。演讲提出，模型定制应被视作一个必须自底向上攀爬的阶梯，只有当低层级手段无法满足明确需求时才应逐级上升；直接跳过前置阶段去构建全新模型，通常是巨大的战略失误。 该阶梯的基础层级是提示词工程（Prompt Engineering）与少样本提示（Few-Shot Prompting），仅凭高质量上下文示例即可解决大量企业需求；第二层是检索增强生成（RAG），通过动态检索参考文档补充最新事实与专有知识，效率远高于重新训练；第三层是参数高效微调（PEFT / LoRA），仅需调整约1%的模型权重即可低成本适配行为与风格；而全参数微调、从头预训练以及人类反馈强化学习（RLHF），属于前沿AI研究实验室的范畴，对常规产品团队而言基本是不必要的。 抵制过度工程与深耕数据质量 (4:19) 顶尖实践者的核心经验表明，团队必须克制在验证基础可行性之前就堆砌架构的冲动。尽管多智能体框架（Multi-Agent）和长链条工具调用流水线备受关注，但过早引入高复杂度架构往往直接导致失败；可靠的做法是从扎实的提示词工程起步，并仅在真实需求明确时逐步添加外部工具。 在数据层面，“垃圾进，垃圾出”的规律在大模型领域更为严苛。来自Amazon的Eugene Yan主张在启动评估前人工精选至少400个“黄金样本”；规模紧凑且经过严密清洗的数据集，在99%的情况下都优于庞大嘈杂的数据集。在确实需要微调的场景中，参数高效方法收益显著：Microsoft在一款端侧教育工具中仅采用LoRA调整了1%的权重，便将不合格回答减少了75%，回答质量评分提升近五倍，彻底规避了全模型重训的高昂算力与延迟代价。 混合评估闭环与确定性防护 (6:47) 评估工作必须从项目启动第一天起就确立为核心机制。OpenAI评估团队建议从轻量级起步，确立由5个代表性测试用例构成的基准，确保任何提示词或配置改动在发布前均能通过验证；Morgan Stanley等金融机构仅通过在部署链路中嵌入严格的自动化评估检查点，便实现了显著的性能提升。 维护长期稳定性需要构建自动化与人工介入结合的混合闭环：一方面，在界面端部署简单的点赞与点踩二元反馈，有助于区分模型基础缺陷与外部应用层漏洞；另一方面，利用AI作为裁判（AI-as-a-Judge）能提供可扩展的质量检查，而AWS等实践表明，使用小型语言模型（SLM）替代高成本前沿模型担任裁判可大幅压低运营支出。此外，安全防线并不完全依赖AI，Microsoft在教育题目生成器中引入了硬编码的正则表达式与格式验证规则作为确定性防护栏，即时过滤结构异常。最终，配合对提示词、校验模式与检查点的严格版本控制，工程团队才能在发生生产异常时实现端到端精确定位。 将大语言模型推进至生产环境并非依赖盲目追求复杂前沿模型，而取决于遵循渐进定制阶梯、人工精调高质数据以及前置自动化评估闭环。然而，面对真实用户交互中不可预测的边界情况，系统能否长期稳定运行，仍考验着团队对确定性规则防护栏与全流程版本追溯的执行力度。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2025 年 8 月 24 日发布，主讲人为 DagsHub 联合创始人兼 CEO Dean Pleban。根据页面介绍，演讲聚焦当前多数企业在构建 LLM 应用时面临的落地瓶颈，并针对非通用业务场景分享了定制化与评估大模型的核心最佳实践，旨在帮助团队跨越原型走向生产环境。 ## TL;DR 演讲深入剖析了绝大多数 LLM 原型无法走入生产环境的四大痛点：幻觉、领域不匹配、安全合规与工程运维障碍。Dean 提出了循序渐进的“定制化阶梯”，主张企业优先从 Prompt 工程、少样本和 RAG 等低复杂度路径切入，避免盲目从头训练模型或过早引入复杂智能体。同时，他结合亚马逊、微软与 OpenAI 的实际经验，强调了亲自把控高质量黄金数据集、将系统化评测前置、采用小模型裁判（SLM as a Judge）以及硬编码护栏与版本追溯的工程落地准则。 ## Atmosphere & style 整个分享务实而接地气，开场以大模型飞速迭代迫使演讲者频繁重写 PPT 的自嘲拉近距离，随后直击业界“可爱原型一上生产就崩溃”的骨感现实。Dean 的讲述风格干练凝练、不讲虚招，毫无追逐大模型概念的狂热浮躁，而是处处流淌着资深 MLOps 工程师对数据质量、成本控制和确定性规则的敬畏。听众能清晰感受到一种从“盲目堆复杂度”向“极简工程兜底”回归的冷静定力与实战指南氛围。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) 大多数 LLM 项目止步于“可爱但脆弱”的原型阶段，在面对真实生产环境的高期望用户和复杂输入时极易崩溃。 - [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) 阻碍 LLM 走向生产的四大核心挑战包括：不可接受的幻觉、通用模型与专业领域的风格/监管不匹配、安全与合规风险，以及 MLOps 运维和成本失控。 - [02:51](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=171s) 企业应对 LLM 定制化采取循序渐进的“阶梯”策略，优先从底层方案做起，而非一上来就尝试从头预训练或全量微调。 - [02:51](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=171s) 定制化阶梯从低复杂度到高依次为：Prompt 工程与少样本示例、RAG 检索增强、参数高效微调（PEFT/LoRA）、全量微调，最顶层才是 RLHF 等前沿实验室专属领域。 - [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) 避免系统过度设计，能通过单条 Prompt 解决的就不要过早串联脆弱的多 LLM 调用链或复杂 Agent 智能体框架。 - [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) 数据质量比数据量更关键，亲自手工筛选和标注极小但高质量的“黄金样本”（如 400 个样本），在 99% 的情况下都优于海量粗糙的数据。 - [06:00](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=360s) 通过 LoRA 等参数高效微调技术仅训练模型约 1% 的权重，即可以极低成本显著降低拒绝回答率并成倍提升回答质量。 - [06:00](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=360s) 评估必须被当作“一等公民”而非后期补丁，每次提示词变更或上线发布前，都应严格跑通经过精心设计的黄金评测集（可从 5 个用例起步）。 - [06:00](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=360s) 构建人机结合的评估反馈机制，前端采集用户点赞点踩，规模化评测场景下可优先采用性价比更高的小语言模型作为裁判（SLM as a Judge）。 - [08:33](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=513s) 推向生产环境必须设置多层护栏（Guardrails），初期即便采用简单的硬编码 if-else 规则做格式校验也能提供关键的工程兜底。 - [08:33](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=513s) 生产级 LLM 应用必须对所有提示词与模型进行严格的版本控制与调用追溯，以确保每次异常均能精确复现与排查因由。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) 为什么大多数 LLM 原型无法走入生产环境 在技术演讲开场时，DagsHub 的联合创始人兼 CEO Dean Pleban 调侃道，由于大模型行业迭代节奏飞快，几乎每次做分享都不得不重写大部分幻灯片。当前几乎所有团队都在尝试构建 LLM 应用，但行业数据表明，绝大多数项目都止步于“可爱但脆弱的 LLM 原型”（cute LLM prototype）阶段，根本无法真正触达真实用户并上线运行。一旦面对实际生产环境中的真实输入，原型便会迅速崩溃。 ```mermaid flowchart TD accTitle: LLM 走向生产面临的核心挑战 subgraph Challenges["LLM 上线面临的核心挑战"] A["幻觉问题 (Hallucinations)"] B["领域不匹配 (Domain Mismatch)"] C["安全合规 (Safety & Compliance)"] D["工程运维 (MLOps & Scale)"] end subgraph Result["结果"] E["原型无法上线 / 被业务叫停"] end Challenges --> Result ``` 造成这一现象的核心阻碍主要体现在四个方面： 1. **幻觉（Hallucinations）**：在 Demo 演示中偶尔出错或许能蒙混过关，但面对抱有高期望的真实用户时，若告诉他们“系统只有 10% 的概率能正常工作，请自行分辨哪次是对的”，对业务而言无异于灾难。 2. **领域不匹配（Domain Mismatch）**：若系统面向金融场景，用户期待的是高盛（Goldman Sachs）级别的专业对话，而模型给出的却是泛化的通用 GPT-4 风格，且难以适应政府、公共部门等高度监管行业的严格标准。 3. **安全与合规（Safety & Compliance）**：模型一旦输出不当言论、泄露数据甚至是误删数据，法务部门便会立刻介入叫停项目。 4. **运维与工程挑战（MLOps / Operations）**：除了模型本身，还必须应对意外输入、API 故障、失控的调用成本，以及系统的扩展性与宕机恢复。 --- ### [02:51](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=171s) LLM 定制化的层级天梯 针对 LLM 定制化路径，Dean 提出不应盲目从头训练模型，而应将其视为一个循序渐进的“定制化阶梯”（Hierarchy of LLM Customization）。绝大多数企业团队应当从阶梯的最底层做起，逐步提升复杂度，而不是一上来就挑战最高难度的方案。 ```mermaid flowchart TD accTitle: LLM 定制化阶梯 subgraph Ladder["LLM 定制化天梯"] L1["1. 提示词调优与少样本示例 (Prompt Engineering / Few-Shot)"] L2["2. 检索增强生成 (RAG / Retrieval Systems)"] L3["3. 参数高效微调 (PEFT / LoRA)"] L4["4. 全量微调与从头预训练 (Full Fine-Tuning / From Scratch)"] L5["5. RLHF 与前沿危险区 (RLHF & Danger Zone)"] end L1 --> L2 --> L3 --> L4 --> L5 ``` - **第一层（阶梯底层）：Prompt 调整与 Few-shot 示例**。无需修改模型，通过精心设计的提示词和少样本示例，就能解决行业内大量的实际问题。 - **第二层：检索系统（RAG）**。当模型缺乏特定事实或知识时，不必急于重训，挂载检索增强系统按需动态拉取数据即可。 - **第三层：参数高效微调（PEFT / LoRA）**。若需定制特定技能或输出风格，只需微调模型极小比例（约 1%）的权重，低成本实现风格与技能注入。 - **第四层及以上：全量微调与从头预训练**。Dean 强调，对绝大多数企业而言，如果觉得自己必须“从头训练模型”，大概率是在前面较低层级的方案选择上出现了偏差；而最顶层的 RLHF 等操作基本属于前沿实验室（Frontier Labs）的领域。 --- ### [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) 成功落地的行业实践经验：从低复杂度到高质量数据 根据工业界的成功实践，Dean 分享了前两条关键法则：**切忌过度设计**与**严把数据质量关**。 很多团队过早引入复杂的智能体框架（Agent frameworks）或将多个 LLM 工具链式串联，往往导致系统脆弱不堪。更稳健的做法是：**能用一条 Prompt 解决的，就不要上复杂的调用链**，只有当业务需求增加且项目被验证有效时，再逐步叠加复杂度。 在数据层面，“垃圾进，垃圾出”（Garbage in, garbage out）依然是铁律。极小但高质量的数据集，在 99% 的情况下都优于庞大而粗糙的数据集。来自亚马逊的 Eugene Yan 曾分享过一个方法：在启动评测前，亲自手工精选和标注至少 400 个黄金样本（Golden Samples）。尽管很多团队倾向于外包标注以避免脏活累活，但亲自审视和清洗数据才是拉开应用成败差距的核心。 --- ### [06:00](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=360s) 轻量微调、系统化评测与人机结合的反馈机制 在模型定制与评估环节，大型机构的落地案例提供了极具价值的参考路径： - **利用 LoRA 高效微调（微软经验）**：微软团队在面向教育场景的端侧设备模型中应用了 LoRA 适配器，仅训练了 1% 的权重，便实现了“拒绝回答率下降 75%、回答质量评分提升近 5 倍”的成绩，极大节省了重新训练全量模型的算力与资金成本。 - **将评估视为一等公民（OpenAI 经验）**：评估绝不能作为项目收尾时的补丁。OpenAI 评测团队的推荐做法是从 5 个黄金测试用例起步；每次提示词变更或准备发布生产前，都必须严格跑通这些核心评测集。摩根士丹利（Morgan Stanley）等金融机构也通过此方法获得了显著的质量收益。 - **引入人类反馈与大小模型裁判（Human-in-the-Loop & LLM/SLM as a Judge）**： - 在用户端放置简单的点赞/点踩（Thumbs Up / Down）按钮，能快速区分出是底层模型缺陷还是上层应用交互逻辑的问题。 - 在规模化质检场景下，可采用“AI as a Judge”。但考虑到调用成本与延迟，使用**小型语言模型作为裁判（SLM as a Judge）**往往是性价比更高的方案。结合人类经验与 AI 自动化打分，才能确保模型持续受控。 --- ### [08:33](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=513s) 生产护栏、版本管理与工程兜底 没有成熟企业会在缺乏防护网的情况下将模型直接推向生产环境。LLM 的安全与合规必须依靠多层护栏（Guardrails）来进行兜底保障。 在实际实现上，护栏并不一定需要依赖复杂的 AI 模型。例如在微软的一个测验生成器（Quiz Generator）流水线中，直接使用了硬编码的 if-else 规则，一旦模型生成的输出不符合预定格式规范就直接丢弃并重试。从单一场景切入实现首个护栏，能够快速积累落地经验并横向复用到其他系统。 Dean 最后总结道，构建生产级 LLM 应用不仅需要做好格式验证，还要对所有的 **提示词（Prompts）与模型版本进行严格的版本控制（Versioning）**。只有确保每次生产异常都能明确追溯“哪次调用失败、因何失败、对应哪个版本”，系统才具备真正的生产可用性。DagsHub 正是通过提供这些工具链与基础设施，帮助企业跨越从实验原型到稳定生产之间的鸿沟。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) 绝大多数项目都止步于“可爱但脆弱的 LLM 原型”（cute LLM prototype）阶段，根本无法真正触达真实用户并上线运行。 — Dean Pleban > [00:04](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=4s) 系统只有 10% 的概率能正常工作，请自行分辨哪次是对的。 — Dean Pleban > [02:51](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=171s) 绝大多数企业团队应当从阶梯的最底层做起，逐步提升复杂度，而不是一上来就挑战最高难度的方案。 — Dean Pleban > [02:51](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=171s) 如果觉得自己必须“从头训练模型”，大概率是在前面较低层级的方案选择上出现了偏差；而最顶层的 RLHF 等操作基本属于前沿实验室（Frontier Labs）的领域。 — Dean Pleban > [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) 能用一条 Prompt 解决的，就不要上复杂的调用链，只有当业务需求增加且项目被验证有效时，再逐步叠加复杂度。 — Dean Pleban > [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) “垃圾进，垃圾出”（Garbage in, garbage out）依然是铁律。极小但高质量的数据集，在 99% 的情况下都优于庞大而粗糙的数据集。 — Dean Pleban > [04:19](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=259s) 在启动评测前，亲自手工精选和标注至少 400 个黄金样本（Golden Samples）。 — Eugene Yan > [06:00](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=360s) 评估绝不能作为项目收尾时的补丁。OpenAI 评测团队的推荐做法是从 5 个黄金测试用例起步；每次提示词变更或准备发布生产前，都必须严格跑通这些核心评测集。 — Dean Pleban > [06:00](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=360s) 在规模化质检场景下，可采用“AI as a Judge”。但考虑到调用成本与延迟，使用小型语言模型作为裁判（SLM as a Judge）往往是性价比更高的方案。 — Dean Pleban > [08:33](https://www.youtube.com/watch?v=CXWtyxz0Uso&t=513s) 构建生产级 LLM 应用不仅需要做好格式验证，还要对所有的提示词（Prompts）与模型版本进行严格的版本控制（Versioning）。 — Dean Pleban ## Mentioned Dean Pleban, DagsHub, Eugene Yan, OpenAI, 微软 (Microsoft), 高盛 (Goldman Sachs), 摩根士丹利 (Morgan Stanley), 亚马逊 (Amazon), GPT-4, LoRA / PEFT, RAG, RLHF ## Video description Dean Pleban, Co-Founder & CEO, DagsHub Abstracts With LLMs making an impact in every industry and product, almost every company is racing to create their LLM application, whether it's an internal tool to boost productivity, or a chat interface for their product. Most of these will fail to make any impact! If your product or domain isn't fully generic, you'll probably hit a lot of challenges that make deploying your LLM application likely to fail. meaningful risk. In this talk, I'll share some of the most important best practices to customize and evaluate LLMs, so your application makes it to the real world. [Source](https://www.youtube.com/watch?v=CXWtyxz0Uso)

Source quotes

[0:04] Dean Pleban: 绝大多数项目都止步于“可爱但脆弱的 LLM 原型”（cute LLM prototype）阶段，根本无法真正触达真实用户并上线运行。 [0:04] Dean Pleban: 系统只有 10% 的概率能正常工作，请自行分辨哪次是对的。 [2:51] Dean Pleban: 绝大多数企业团队应当从阶梯的最底层做起，逐步提升复杂度，而不是一上来就挑战最高难度的方案。 [2:51] Dean Pleban: 如果觉得自己必须“从头训练模型”，大概率是在前面较低层级的方案选择上出现了偏差；而最顶层的 RLHF 等操作基本属于前沿实验室（Frontier Labs）的领域。 [4:19] Dean Pleban: 能用一条 Prompt 解决的，就不要上复杂的调用链，只有当业务需求增加且项目被验证有效时，再逐步叠加复杂度。 [4:19] Dean Pleban: “垃圾进，垃圾出”（Garbage in, garbage out）依然是铁律。极小但高质量的数据集，在 99% 的情况下都优于庞大而粗糙的数据集。 [4:19] Eugene Yan: 在启动评测前，亲自手工精选和标注至少 400 个黄金样本（Golden Samples）。 [6:00] Dean Pleban: 评估绝不能作为项目收尾时的补丁。OpenAI 评测团队的推荐做法是从 5 个黄金测试用例起步；每次提示词变更或准备发布生产前，都必须严格跑通这些核心评测集。 [6:00] Dean Pleban: 在规模化质检场景下，可采用“AI as a Judge”。但考虑到调用成本与延迟，使用小型语言模型作为裁判（SLM as a Judge）往往是性价比更高的方案。 [8:33] Dean Pleban: 构建生产级 LLM 应用不仅需要做好格式验证，还要对所有的提示词（Prompts）与模型版本进行严格的版本控制（Versioning）。

Source quotes

[0:04] Dean Pleban: building a "cute LLM prototype" that looks magical in internal testing, only to watch it completely crumble the moment real users interact with it. [0:04] Dean Pleban: it's going to work like 10% of the time, and you figure out which 10%. [0:04] Dean Pleban: an enterprise financial application cannot speak generic GPT-4 when the use case demands the rigor and terminology of Goldman Sachs [2:47] Dean Pleban: LLM customization should instead be approached as a ladder, where teams begin at the lowest rung and only ascend when simpler methods fail to meet concrete requirements. [2:47] Dean Pleban: If an organization concludes that it must train a proprietary foundation model from scratch, it has almost certainly made a strategic misstep along the way by skipping simpler, more cost-effective layers. [4:19] Dean Pleban: While complex multi-agent frameworks and deeply chained tool pipelines are popular, jumping straight to high complexity often leads directly to project failure. [4:19] Dean Pleban: a compact, meticulously curated dataset outperforms a massive, noisy dataset 99% of the time. [4:19] Dean Pleban: direct contact with the data often separates successful production deployments from failed experiments. [6:47] Dean Pleban: Evaluation must be treated as a first-class citizen from day one rather than an afterthought. [6:47] Dean Pleban: teams can drastically lower operational costs by deploying Small Language Models (SLMs) as judges instead of high-cost frontier models.

Original transcript

[0:04] Hi everyone. I'm uh excited to be here. [0:07] Um hope no one was uh scared away by the [0:10] talk title. I'm Dean from DAGSHub and [0:12] this is going to be a short and updated [0:15] version of a talk uh I gave a while [0:17] back. It's basically impossible now not [0:19] to rewrite most of the talks you give [0:21] because of the pace of the industry. Uh [0:23] if you find it interesting and you want [0:25] to chat more, you can find me afterwards [0:26] at the DAGA booth. Uh, but this is going [0:29] like I'm going to talk a tiny bit about [0:31] DAGsa, but it's not the main point of [0:33] the the talk. So, uh, if you're here, [0:36] I'm guessing that you're either working [0:38] or want to work on LLMs. And I feel like [0:41] uh we've all been there. You know, [0:42] there's a cute LLM prototype uh that [0:45] crumbles and uh uh doesn't work when [0:48] real users show up. And um most LLM [0:52] projects, this is based on a bunch of [0:53] data, uh but most LLM projects don't [0:55] actually make it to the point where [0:57] users are seeing them and using them, [0:59] which is kind of a shame. Um some of the [1:02] reasons are like unexpected inputs. [1:04] There's a lot of jokes about that I can [1:06] tell you afterwards. API hiccups, uh co [1:09] costs that just go crazy. Uh reality [1:12] tends to hit hard. And in this talk, I [1:15] will try to help you bridge that gap so [1:18] your LLM baby prototype survives in the [1:21] wild. Um, so let's start by unpacking [1:25] why a lot of these initiatives never see [1:27] the light of day. Uh, first [1:29] hallucinations, right? Everyone knows [1:31] what that is at this point. Um, you [1:34] probably could get away with it in your [1:36] demo, but uh, it's kind of disastrous [1:38] when it's gets to real users that have [1:40] real expectations. Uh, and then you tell [1:42] them, oh, it's going to work like 10% of [1:44] the time, and you figure out which 10%. [1:47] Um, next part is domain mismatch. So, if [1:51] you're building a LLM that's oriented [1:53] towards finance, it won't look good if [1:56] it speaks generic GBT4 when you need uh [1:59] Goldman Sachs. Uh, and this point is, I [2:02] think, especially true when there are [2:04] concrete standards and regulations to [2:06] meet uh, for example, in government, [2:08] public sector, and other regulated [2:10] industries. [2:11] uh which leads me to the third point uh [2:13] which is safety and compliance. So we [2:15] heard about a lot about that uh during [2:16] the talks and the keynotes today but if [2:19] your model is going to blurt out [2:21] something inappropriate um or leak data [2:23] or delete data god forbid um suddenly [2:27] legal might be pulling the plug and [2:29] these are some of the dragons that we [2:31] need to slay. Uh and the last thing [2:33] right is the op stuff the dirty stuff. [2:35] So um you know you need to plan for [2:38] scale, you need to plan for downtime, [2:39] you need to plan for an app. So the LLM [2:41] is a part of a bigger story and that [2:44] story needs to work uh uh well with that [2:47] part. So what I'm going to claim and [2:51] this is like an important slide at least [2:53] in my opinion is that uh not every [2:55] project needs a fancy new model from [2:58] scratch. Uh often we sort of can climb [3:01] this hierarchy of LM customization. Um [3:04] and and you can think of it kind of like [3:05] a ladder. So at the base you have simple [3:07] prompt tweaks, few shot examples. Uh and [3:10] those will get you surprisingly far from [3:13] the conversations that we've had in [3:14] industry. Uh next, if the model is [3:16] missing facts, uh you don't need to [3:18] retrain yet. Uh just bolt on some [3:20] retrieval system. So rag is obviously [3:23] the buzzword, but it can be whatever [3:25] works for your use case. Um so then it [3:28] can fetch the right info uh on the fly. [3:30] Uh if you need more tuning then you can [3:32] use a parameter efficient fine-tuning [3:34] aka uh Laura which basically means [3:36] you're fine-tuning but just a tiny tiny [3:38] portion of your model um so that you can [3:41] imprint new skills or a new style [3:43] cheaply. Um and then if that's not [3:46] enough then you go into the fine-tuning [3:47] and the whole model stuff or even train [3:49] a new one. I'm guessing that for most [3:51] organizations here in this uh audience [3:53] that's never going to be relevant. Like [3:55] if you're getting to the point where you [3:57] say we have to build a model from [3:58] scratch, you're more likely making a [4:00] mistake along the way and not realizing [4:02] what's the lower level in the ladder [4:04] that's going to fit your your problem. [4:06] That's not 100% of the cases, but that's [4:09] sort of a a good sort of a rule of [4:12] thumb. And then you have RHF and the [4:14] danger zone, which is I think only the [4:15] Frontier Labs basically um get there. So [4:19] um let's go over a few industry tips to [4:21] improve your chances of success. All [4:23] these are based on uh stuff that I got [4:25] from other much smarter people than I [4:28] am. So uh don't take my word for it. Um [4:32] so the first tip is basically to not [4:34] overengineer it. Right? It might seem [4:36] obvious but I think that a lot of teams [4:38] even now you know talking uh here are [4:41] sort of aiming for the higher levels of [4:42] the ladder before going through the [4:44] lower levels. Um and so that usually [4:47] tends to result in uh crashing and [4:49] burning. So agent frameworks are great [4:51] but you should start with prompt [4:52] engineering. Um successful projects keep [4:55] it simple and then add complexity as a [4:57] project is more successful requirements [4:59] add up uh etc. And uh so yeah so before [5:03] you uh build a really complex uh system [5:05] of chained LLM calls with a bunch of [5:07] tools can a prompt just do it. Um [5:11] another universal law that you probably [5:13] have heard if you're long enough in this [5:14] business is garbage in garbage out. your [5:16] model is only going to be as good as [5:18] your data. Um, a smaller uh highquality [5:22] data set outperforms a bigger sloppy one [5:25] 99% of the time. So, invest in cleaning [5:28] your data. Um, Eugene uh Yan from Amazon [5:32] once told me that he starts most [5:33] evaluation work by hand curating at [5:36] least 400 golden samples. I know that's [5:38] like a dirty word. People don't like to [5:40] actually build the data sets themselves. [5:42] It's easier to pay someone to do it. But [5:44] many times looking at the data and [5:46] actually curating and crafting that data [5:48] set is going to make a huge different in [5:50] the success of the project. Um so yeah I [5:52] tal I spoke about this um but I really [5:54] think this separates the successful from [5:57] unsuccessful uh applications. Um [6:00] sometimes you do need to fine-tune but [6:02] you don't have to fine-tune everything. [6:04] So this is a tip from uh Microsoft uh [6:07] shy there showed that you can use Laura [6:09] adapters uh and basically just train 1% [6:12] of the model weights and that can work [6:13] wonders. Um this specific story sort of [6:16] they plugged in Laura into an ondevice [6:18] model for educators and they got 75% [6:21] fewer uh rejected answers and almost [6:24] five times better uh quality rating. So [6:27] and all of this without retraining the [6:29] entire uh model which would be much more [6:32] expensive. Microsoft probably has the [6:34] money to to fund that, but you you [6:36] probably don't want to do that if you [6:37] don't have to. So, try to be efficient. [6:39] Uh, and then do the the Laura parameter [6:42] efficient fine-tuning. If you're not [6:44] familiar with those terms, those are [6:45] really important terms to know in my [6:47] opinion. Um, tip four is OpenAI, right? [6:51] So, you need to treat evaluation as a [6:54] first class citizen. And this is also [6:55] something that comes up in a lot of the [6:56] conversations we've had uh today, like [6:59] when people are walking through our [7:00] booth. Um it's not an afterthought. I [7:03] think uh the eval team at OpenAI likes [7:05] to start with just a handful of test [7:06] cases and so I would recommend like I [7:08] think their number is five. So just pick [7:11] five examples uh and then make sure that [7:13] every prompt everything that you're [7:15] actually considering getting to [7:16] production goes through that uh um those [7:19] sort of evals. Um and yeah, there's like [7:22] really huge gains that that companies [7:24] like Morgan Stanley got from this, but [7:26] um I'll run ahead. Um, so the last thing [7:31] is how do we make sure that our LLM is [7:33] actually useful to people and I think [7:34] that the answer there is that we want to [7:36] involve humans in the feedback cycle and [7:38] this is another tip from OpenAI but just [7:40] like putting humans in the loop with a [7:42] thumbs up and thumbs down uh button [7:44] which you've seen if you've ever used uh [7:45] chatgpt um and then you discover some of [7:48] the problems are model problems but [7:50] other like application problems but in [7:52] the end this is what's going to [7:53] determine the user experience um in a [7:56] different uh case the scale quality [7:58] checks. Folks at AWS used another AI as [8:02] a judge. So LLM as a judge is like very [8:04] popular. We can talk about that later. [8:06] Um just take into consideration the cost [8:08] and performance into this. Many times [8:10] you want to do like SLM as a judge which [8:12] I think is uh something that you hear [8:14] less of but is very relevant for a lot [8:16] of organizations that want to do these [8:18] evaluations and not spend like millions [8:20] of dollars on them. Um, and so mixing a [8:23] human insights and AI focused stuff and [8:25] AI based feedback, sorry, uh, is the [8:27] best way to keep your, uh, model on [8:29] track. Um, last one, we're almost at the [8:33] end. So, no company in their right mind [8:35] deploys an ML model without safeguards. [8:37] Uh, and LLMs are um, no different. So, [8:40] building guardrails as a safety net, [8:43] right? Um, in one case, um, Microsoft [8:46] was using like hardcoded rules into a [8:49] quiz generator's pipeline and then if [8:51] the model answer didn't meet the, um, [8:53] the format guidelines, it got tossed [8:55] out, right? So, it doesn't always have [8:56] to be like LLM as a judge. It can be [8:58] simple if else uh, statements. Um, but [9:01] that's sort of uh, uh, something I think [9:03] we can all aspire to do. And I think [9:05] again here if you choose one case and [9:08] implement that guardrail that's going to [9:09] take you a huge portion of the way to [9:12] having implemented uh production [9:14] guardrails at scale because you'll learn [9:17] a ton on doing this the first time and [9:19] then replicating it is uh much easier. [9:22] Um so yeah the other port parts here are [9:27] also you know important. So you want to [9:29] format validate and then version all of [9:32] the prompts and the model versions that [9:34] you're using so that you know when [9:35] something fails in production what [9:37] failed why it failed and all of that. Um [9:40] so yeah um I'm happy to talk more about [9:43] how DAGsub helps all these organizations [9:45] and using these tools uh to um you know [9:49] build better models and get them to [9:51] production. Um but yeah, I'll end there. [9:54] Thank you for listening and I'm looking [9:56] forward to seeing you at our booth.