Empromptu AI Converts Agent Traces Into Custom Models to Cut API Costs

Using Your Existing Agents to Train Custom Models for a Fraction of a Cost of Frontier Models — Ai4

Ai4

Enterprises can eliminate reliance on commercial frontier APIs and convert operational spending into balance-sheet assets by fine-tuning open-weight models on their existing agents' execution traces.

AI-assisted summary of Using Your Existing Agents to Train Custom Models for a Fraction of a Cost of Frontier Models — original source. Check the original for full context.

Source published: 2026-08-15

Detailed notes

## Background Published by Ai4 on YouTube on August 15, 2026, this talk features Shanea Leven, CEO and co-founder of Empromptu AI. The presentation outlines a practical framework for capturing execution traces from deployed AI agents to build rigorous evaluation sets and fine-tune task-specific open-weight models at a fraction of frontier model costs.

## TL;DR Shanea Leven explains that relying exclusively on commercial frontier model APIs creates prohibitive costs, vendor lock-in, and compliance vulnerabilities in high-stakes regulated sectors. Rather than training monolithic models from scratch, enterprises can capture 30 days of execution traces and edge cases from high-volume automated agents to construct golden evaluation suites. These operational traces are then used to fine-tune compact, specialized open-weight models orchestrated via RAG, leaving frontier models solely as fallback escalation paths. This modular flywheel transforms recurring API operating expenses into proprietary, on-premises intellectual property assets that improve valuation and security.

## Atmosphere & style The presentation carries a pragmatic, authoritative, and direct tone that cuts through broad AI hype to address the financial and operational realities of enterprise deployment. Leven balances sober warnings about hallucinations in regulated healthcare and shadow IT leaks with approachable humor regarding unmaintained internal tools. The delivery feels like an empowering, actionable masterclass for technical and business leaders seeking independence from centralized frontier model providers.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) Relying exclusively on rented frontier model APIs introduces severe operational, cost, and outage vulnerabilities for enterprises. - [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) In regulated, high-stakes environments like healthcare, probabilistic approximations and 'mostly right' model outputs create unacceptable liabilities. - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) Long-term enterprise defensibility requires owning custom model intelligence rather than depending entirely on third-party AI vendors. - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) Owning custom AI assets does not require pretraining a foundation model from scratch; targeted fine-tuning on open-weight models achieves domain-specific parity. - [04:07](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=247s) Existing autonomous agents and automations continuously generate valuable training data through runtime execution traces, human overrides, and failure logs. - [04:07](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=247s) Building robust evaluation benchmark suites (evals) from real-world agent traces is critical for codifying subject-matter expertise and benchmarking models. - [05:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=304s) The four-step framework involves selecting a high-volume agent, capturing 30 days of decision traces, creating golden eval sets, and fine-tuning narrow task models. - [05:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=304s) A modular architecture that runs routine tasks on small, specialized open-weight models and reserves frontier models for complex edge cases drastically cuts API costs. - [06:28](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=388s) Rampant shadow IT—such as employees using personal subscriptions like Claude Code—bypasses enterprise retention policies and leaks proprietary knowledge. - [06:28](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=388s) Retaining proprietary model weights transforms ongoing API operating expenses into capitalized intellectual property that enhances company valuation. - [06:28](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=388s) On-premises deployment with dynamic guardrails and automated self-monitoring prevents pipeline drift and unmaintained internal automations in regulated sectors.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) The Fragile Reality of Renting Frontier AI

Every major technology revolution crowns its defining darlings: the cloud era produced Salesforce, the gig economy launched Uber, and the generative AI wave has catapulted frontier model providers into some of the most valuable corporations on the planet. Yet for enterprise leaders striving to build the next breakout company, leaning exclusively on frontier models introduces existential risks. Shanea Leven introduces Dr. Haley, co-founder and CEO of Ascent Autism, whose company delivers online group therapy for children on the autism spectrum. Dr. Haley sits at the crossroads of deep domain expertise, a vulnerable customer base, and strict healthcare regulations. Like many founders, she was sold the pervasive narrative that AI is a transformative "messiah" poised to revolutionize unit economics and multiply margins.

The central challenge in regulated industries is that probabilistic approximations are intolerable: when a model makes an assessment, it directly influences whether a patient receives care, how parents perceive their child, or how that child interprets the surrounding world. In high-stakes environments, "mostly right isn't good enough." Even outside life-or-death scenarios, organizations spending millions of dollars on experimental AI pipelines are struggling to realize defensible returns.

That fragility was thrown into sharp relief when a major service disruption took key systems offline, prompting Leven to ask how many in the room remembered the sudden blackout—a prompt met with hands raised across the room. Relying entirely on hosted third-party infrastructure exposes enterprises to a stark reality: pouring millions into intelligence they neither own nor fully control.

- **Historical parallels**: Cloud computing birthed Salesforce; the gig economy established Uber; the AI boom is currently concentrating enterprise value inside frontier model creators. - **High-stakes use case**: Ascent Autism (led by Dr. Haley) conducts digital group therapy for autistic children, where standard LLM hallucination rates and "mostly right" outputs create direct clinical and reputational liabilities. - **Enterprise dilemma**: Massive capital allocations into third-party AI APIs leave companies stranded in a "sea of endless uncertainty" during vendor outages and policy shifts.

---

[02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) The Imperative to Own Intelligence

The compounding pain of ballooning API costs, lack of operational ownership, and zero infrastructure control all trace back to a single architectural error: organizations are renting their core intelligence. Industry leaders across tech—Satya Nadella of Microsoft, Alex Karp of Palantir, and Clément Delangue of Hugging Face—have publicly argued that sustainable enterprises must train and control their own custom models rather than outsourced dependencies.

Leven introduces herself as the CEO and co-founder of Empromptu AI, alongside co-founder Dr. Sean Robinson, a PhD in computational astrophysics who previously worked on deploying satellites into orbit. For years, their mission has centered on helping companies in regulated sectors achieve measurable returns without the financial drain of frontier API billing.

Leven clarifies a common misconception among overwhelmed executives: owning a custom model does not require training a frontier foundation model or reinventing ChatGPT from scratch. The legacy route of attempting to fine-tune a monolithic custom model across an entire enterprise catalog is obsolete. Instead, companies can establish an operational flywheel using specialized open-weight models trained on real operational workflows.

- **Key industry voices cited**: Satya Nadella (Microsoft), Alex Karp (Palantir), and Clément Delangue (Hugging Face) all advocate for enterprise-owned custom intelligence. - **Empromptu AI leadership**: Shanea Leven (CEO) and Dr. Sean Robinson (computational astrophysicist with background in satellite deployment). - **Core philosophy**: Owning proprietary AI assets should not require pretraining a foundation model from zero; targeted fine-tuning on open-weight architectures yields equivalent or superior performance on domain-specific enterprise tasks.

---

[04:07](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=247s) Turning Agent Traces and Evals into Custom Models

Rather than building massive datasets manually, deployed autonomous agents and workflow automations naturally generate high-value training material every day. As agents interact with internal workflows, they output actionable traces that document real-world enterprise reasoning. By wrapping these agents with proper data governance, companies can capture executions, parse domain corrections, isolate edge cases, and harvest concrete user feedback.

These real-world execution traces form the foundation of internal evaluation suites ("evals"). Without rigorous evals, companies cannot systematically benchmark model capabilities or capture institutional domain expertise. By pairing an open-weight base model with targeted training data extracted from agent failures, corrections, and verified preferences, organizations convert operational runtime exhaust into proprietary fine-tuned assets.

```mermaid flowchart TD accTitle: Agent Training Flywheel subgraph Capture["1. Capture Traces"] A["Deployed High-Volume Agent"] --> B["Capture Execution Traces (30 Days)"] B --> C["Filter Edge Cases, Corrections & Failures"] end subgraph Evals["2. Build Evaluation Benchmarks"] C --> D["Define Ground Truth & Evaluation Sets"] end subgraph FineTune["3. Fine-Tune & Orchestrate"] D --> E["Fine-Tune Open-Weight Task Model"] E --> F["Aggregate Task Models via RAG Architecture"] F --> G["Hybrid Routing: Small Custom Models + Frontier Escalation"] end ```

- **Data collection targets**: Routine agent logs, manual human overrides, system failures, user preferences, and anomalous edge cases. - **Role of evals**: Rigorous evaluation benchmarks codify enterprise subject-matter expertise into repeatable tests that validate model iterations. - **Architecture shift**: Replacing general-purpose frontier API prompts with targeted open-weight models fine-tuned solely on verified operational traces.

---

[05:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=304s) The Four-Step Implementation Framework

Leven breaks the transition down into a practical, repeatable four-step process:

1. **Select a high-volume agent**: Identify an agent or automated task executing repetitive decisions at scale. 2. **Capture outcomes over 30 days**: Record every decision trace, noting environmental inputs and final outputs across normal operations. 3. **Isolate repetitive decisions and construct an eval set**: Define clear boundaries for what ideal execution looks like, catalogue failures, identify edge cases, and formalize explicit operational rubrics for handling anomalies. 4. **Fine-tune a focused, task-specific model**: Train a compact, open-weight model dedicated entirely to that individual task, repeating the process across other functional workflows.

Once multiple narrow models are trained, an enterprise aggregates them using retrieval-augmented generation (RAG) and intelligent orchestration. This modular setup gives teams total operational elasticity: routine, narrow tasks run locally on lightning-fast, ultra-low-cost specialized models, while resource-intensive, open-ended reasoning tasks can still be escalated to frontier models when strictly necessary—saving companies "a boatload of money." To address lingering skepticism, Leven offers an open challenge to attendees: bring Empromptu AI a high-volume agent, and they will prove the measurable latency, cost, and accuracy improvements hands-on.

- **Step 1**: Target an existing, high-frequency workflow agent. - **Step 2**: Log performance and decision streams continuously for a 30-day window. - **Step 3**: Extract recurring operational paths to construct golden eval datasets. - **Step 4**: Fine-tune compact open-weight models for individual micro-tasks, combining them modularly via RAG. - **Cost advantage**: Drastically lowers API token expenditure by reserving expensive commercial frontier models purely for edge-case exceptions.

---

[06:28](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=388s) Q&A: Shadow IT, On-Prem Security, and Enterprise Viability

During the audience discussion, an attendee asks about enterprise data safety, conceding that while leaking data to frontier providers poses serious corporate risks, their company chose to accept that exposure to maintain competitive relevance. Leven responds that even under strict enterprise zero-data-retention agreements, unmonitored employee practices undermine safeguards. Staff frequently rely on unsanctioned personal subscriptions—such as individual Claude Code or OpenAI accounts—inadvertently piping internal intellectual property into commercial training pipelines. More critically, renting intelligence from external providers fails to accrue enterprise value, whereas packaging institutional knowledge into owned model weights builds a tangible, balance-sheet asset that elevates corporate valuation.

An audience member asks about deployment topology, and Leven confirms that Empromptu AI runs on-premises by default to meet stringent regulatory mandates. The platform is HIPAA and SOC 2 compliant, featuring integrated audit logs, dynamic guardrails, and centralized policy enforcement to rein in shadow IT without requiring constant manual policing. Leven jokes that this allows engineering and compliance leads to actually go on vacation rather than worrying about whether "Debbie from accounting decided that she wanted to build an invoicing app" that she has no intention of maintaining.

To handle orphaned or broken automations, the platform includes automated self-monitoring and maintenance loops. Powered by patented technology developed by Dr. Robinson, Empromptu AI achieves up to 98% output accuracy across underlying foundation models, eliminating low-quality "AI slop." Leven closes by outlining their track record across deeply scrutinized sectors: deploying organized crime tracking models in collaboration with the former head of the DEA, clinical workflows with healthcare leaders like Dr. Haley, long-term care management, physical security networks, institutional finance, and private equity.

- **Shadow IT risks**: Rogue employee use of personal tools (e.g., Claude Code, public consumer accounts) bypasses enterprise agreements and leaks proprietary workflow knowledge. - **Valuation upside**: Retaining proprietary model weights converts operational spending into capitalized IP assets rather than ongoing operating expenses paid to external API vendors. - **Security and infrastructure**: On-prem by default, HIPAA and SOC 2 certified, with pre-configured audit logging, automated guardrails, and compliance policy enforcement. - **Autonomous maintenance**: Built-in monitoring resolves pipeline drift and unmaintained internal tools (the "Debbie from accounting" scenario). - **Performance & validation**: Delivers up to 98% accuracy on complex domain tasks; deployed in federal law enforcement (organized crime tracking with a former DEA head), clinical healthcare, senior living, physical security, and private equity.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) mostly right isn't good enough — Shanea Leven

> [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) messiah — Shanea Leven

> [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) sea of endless uncertainty — Shanea Leven

> [04:07](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=247s) evals — Shanea Leven

> [05:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=304s) a boatload of money — Shanea Leven

> [06:28](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=388s) Debbie from accounting decided that she wanted to build an invoicing app — Shanea Leven

> [06:28](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=388s) AI slop — Shanea Leven

> [05:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=304s) Select a high-volume agent — Shanea Leven

## Mentioned Shanea Leven, Dr. Haley, Ascent Autism, Salesforce, Uber, Satya Nadella, Microsoft, Alex Karp, Palantir, Clément Delangue, Hugging Face, Empromptu AI, Dr. Sean Robinson, ChatGPT, Claude Code, OpenAI, Drug Enforcement Administration (DEA), Ai4

## Video description Shanea Leven - CEO & Co-Founder, Empromptu AI

Most teams don't realize their deployed agents are already generating the training data they need. This talk walks through how to capture agent traces, filter for high-quality examples, and fine-tune smaller models that match frontier performance on your specific tasks — at a fraction of the cost. You'll leave with a practical framework for turning your existing AI infrastructure into a model training flywheel.

[Source](https://www.youtube.com/watch?v=c39bjpPOW_Q)

中文简报

Empromptu AI 将智能体轨迹转化为定制模型以削减 API 成本企业可以通过在现有智能体的执行轨迹上微调开放权重模型，摆脱对商业前沿 API 的依赖，并将运营支出转化为资产负债表资产。 Shanea Leven 解释称，在高风险受监管行业中，完全依赖商业前沿模型 API 会带来高昂的成本、供应商锁定以及合规漏洞。企业无需从零开始训练单体大模型，而是可以从高业务量的自动化智能体中捕获 30 天的执行轨迹和边缘案例，以此构建黄金评估集。随后，这些运行轨迹将用于微调通过 RAG 编排的紧凑型专业开放权重模型，仅将前沿模型用作兜底升级路径。这种模块化飞轮将持续的 API 运营支出转化为专有的本地知识产权资产，从而提升估值并增强安全性。企业完全依赖租赁前沿模型 API 面临高昂成本、中断停机与合规风险，而拥有自主 AI 资产无需从零预训练基础模型，对开源权重模型进行针对性微调即可达到同等水准。 (2:25) 已部署的自主智能体（Agents）在运行中会持续产生执行轨迹、人工覆写记录与故障日志，这些实战数据正是搭建企业专属评估基准（evals）与微调飞轮的高价值养料。 (4:07) 四步落地框架包括：挑选高频智能体、捕获30天决策轨迹、提炼黄金评估集，以及微调特定微任务模型；结合 RAG 架构让常规任务跑在小型开源模型上，能大幅削减前沿模型调用开销。 (5:04) 针对员工使用非合规个人工具带来的影子 IT 与数据泄露隐患，系统采用支持 HIPAA 和 SOC 2 的本地化部署方案，通过动态护栏与自动维护闭环将业务专有知识沉淀为企业自有的权重资产。 (6:28)

Reading summary

Shanea Leven, CEO of Empromptu AI, argues that relying exclusively on commercial frontier AI models creates unsustainable expenses, operational instability, and compliance risks for enterprises. Instead of renting external intelligence, organizations can extract runtime traces from existing automated agents to fine-tune specialized, owned open-weight models at a fraction of the cost. Vulnerabilities of Renting Frontier AI Models (0:04) Leven asserts that enterprises relying entirely on third-party frontier APIs face ballooning costs, infrastructure blackouts, and critical vendor dependencies. In highly regulated sectors—such as clinical therapy for children on the autism spectrum, exemplified by Dr. Haley of Ascent Autism—probabilistic errors and standard model hallucinations pose direct clinical and reputational liabilities where "mostly right" outputs are unacceptable. Citing industry executives such as Satya Nadella of Microsoft, Alex Karp of Palantir, and Clément Delangue of Hugging Face, Leven contends that enduring enterprise value requires proprietary control over core intelligence. Relying on external model endpoints drains capital into rented infrastructure without establishing defensible corporate assets. Turning Agent Execution Traces into Evaluation Suites (4:07) Organizations do not need to pretrain monolithic foundation models from scratch to own their AI stack. Instead, Leven explains that active automations and autonomous agents continuously generate valuable reasoning exhaust through everyday internal workflows. By capturing these runtime interactions over time, enterprises isolate human corrections, operational edge cases, and systemic failures. These real-world logs form the basis for rigorous evaluation suites ("evals"), which codify institutional domain expertise into repeatable testing benchmarks. Organizations then use these verified datasets to fine-tune compact open-weight models dedicated to specific functional tasks. A Four-Step Flywheel for Hybrid Deployment (5:04) Empromptu AI outlines a four-step framework for transition: identify an existing high-volume agent, log operational decision traces across a 30-day window, isolate recurring workflows to build a golden evaluation set, and fine-tune a compact open-weight model on that specific task. Once multiple narrow models are trained, an enterprise aggregates them using retrieval-augmented generation (RAG) and hybrid routing. Under this modular architecture, high-frequency, narrow tasks execute locally on low-cost, low-latency models. Calls to commercial frontier models are reserved exclusively as an escalation path for ambiguous or highly complex edge cases, sharply reducing overall API expenditure. Mitigating Shadow IT and Capitalizing Enterprise Assets (6:28) Enterprise data security is frequently compromised when employees bypass corporate policies to use unsanctioned personal accounts, such as individual Claude Code or OpenAI subscriptions, inadvertently exposing internal intellectual property. Leven states that deploying custom models on-premises with HIPAA and SOC 2 compliance provides automated audit trails, guardrails, and automated maintenance loops to manage orphaned workflows. Beyond risk mitigation, Leven argues that training custom weights converts operating expenses into balance-sheet intellectual property that enhances enterprise valuation. Supported by patented technology from co-founder Dr. Sean Robinson, Empromptu AI claims up to 98% accuracy across production deployments in healthcare, institutional finance, physical security, and federal organized crime tracking. The presentation positions specialized open-weight models as a more secure, cost-effective alternative to frontier APIs, though the proposed flywheel assumes that an organization already runs high-volume agents generating structured traces and possesses the infrastructure to orchestrate a modular, multi-model architecture.

精炼阅读摘要

Empromptu AI 首席执行官 Shanea Leven 指出，企业过度依赖租用商业前沿模型不仅面临高昂的 API 成本与停机断供风险，在严监管的高风险场景中更无法承受概率性幻觉；企业应当利用现有智能体在日常运行中积累的决策轨迹构建评估集，针对性微调开源权重模型，以极低成本将组织知识沉淀为自主可控的资产。租用商业模型的脆弱性与高风险挑战 (0:04) 在历史技术周期中，云时代催生了 Salesforce，零工经济建立了 Uber，而当前的生成式 AI 浪潮正将巨额企业价值集中在少数前沿基础模型提供商手中。然而，对于受严格监管且容错率极低的行业而言，完全依赖第三方前沿模型蕴含重大风险。例如，Dr. Haley 创立的 Ascent Autism 为自闭症儿童提供数字化团体治疗，其模型输出直接影响临床决策与患儿家庭，在该领域中“基本正确”的概率性回答被视为不可接受的临床与声誉隐患。大规模采购第三方 API 容易让企业在遭遇供应商宕机或策略调整时陷入被动局面。包括微软的 Satya Nadella、Palantir 的 Alex Karp 以及 Hugging Face 的 Clément Delangue 在内的多位行业领袖均公开提出，可持续发展的企业必须掌握并训练自身的定制模型，而非将核心智能持续外包给基础设施供应商。将智能体运行轨迹转化为自有评估集 (4:07) Leven 澄清了一个广泛存在的认知误区：拥有自主模型并不需要从零预训练一个前沿基础模型，也不再需要对整套企业业务目录进行庞大且单体式的微调。企业在日常生产环境中部署的高频自主智能体与自动化工作流，每天都在自然生成极具价值的真实决策轨迹。通过建立完善的数据治理机制，企业能够记录运行数据、人工介入修正、边缘异常以及用户真实反馈。这些日常运行轨迹构成了企业内部评估套件（evals）的基石。缺乏严格的评估集，企业便无法系统化衡量模型能力或固化专有领域知识。通过将经过验证的边缘案例、纠错记录和偏好数据与开源权重（open-weight）基础模型结合，企业能将原本被视为系统消耗的运行日志转化为高价值的微调数据资产。四步实施框架与混合编排架构 (5:04) Empromptu AI 提出了一个标准化的四步落地路径：第一，选定一个承担高频重复性决策的现有业务智能体；第二，在 30 天周期内持续记录其业务环境输入与最终输出轨迹；第三，剥离出高频决策路径并整理异常与失败场景，建立清晰的判定准则与黄金评估数据集；第四，针对该特定任务微调轻量级开源模型，并在此后向其他业务流程逐步推广。完成多个特定任务模型的微调后，企业可通过检索增强生成（RAG）与智能调度架构进行模块化集成。常规的高频细分任务由本地部署的低成本、低延迟专用模型处理；当遇到极为复杂的开放式推理需求时，系统再将任务升级转发给商业前沿模型。这种混合架构在保证准确率的同时大幅削减了 API Token 采购支出。防范影子IT、本地合规与模型资产化 (6:28) 针对企业数据合规，Leven 提示即便签署了企业级零数据保留协议，员工私下使用未经授权的个人账号（如个人端 Claude Code 或 OpenAI 服务）仍会形成严重的影子 IT 风险，导致专有工作流外泄。从财务与估值角度看，将资金用于租用外部 API 仅属于日常运营费用，而将专有业务知识训练并固化在自有模型权重中，则能转化为资产负债表上的无形知识产权，从而直接提高企业估值。在基础设施架构上，Empromptu AI 默认采用本地化（on-prem）部署以满足严苛监管，具备 HIPAA 与 SOC 2 合规认证，并内置审计日志与动态护栏，同时配备自动监测回路以防止无人维护的内部工具出现管道漂移。据 Leven 介绍，由计算天体物理学博士 Sean Robinson 研发的技术能使底层模型输出准确率达到最高 98%，目前已在有组织犯罪追踪（与前 DEA 负责人合作）、临床医疗、养老照护、实体安防以及私募股权等高要求业务场景中落地。该方案的核心逻辑在于利用企业已有的智能体运行足迹替代昂贵的外部模型调用，从而将运营数据转化为专有模型权重资产；不过该路径的有效实施高度依赖企业现存的工作流体量，且要求团队具备高质量提炼评估基准与处理复杂边缘用例的工程能力。

Detailed notes — zh

## Background 该内容由 Ai4 于 2026 年 8 月 15 日发布在 YouTube 平台。视频记录了 Empromptu AI 联合创始人兼 CEO Shanea Leven 的主题演讲，主要阐述企业如何捕获已部署 AI Agent 的运行轨迹，将其转化为高质量训练数据并微调开源小模型，以极低成本达到前沿大模型性能并构建模型飞轮。 ## TL;DR 演讲指出强监管行业若过度依赖第三方闭源大模型，将面临高昂成本、服务中断以及“租赁智能”带来的失控风险。为此，演讲者提出了一套四步实操框架：持续捕获高频 Agent 运行轨迹、提炼决策模式构建评测集（Eval）、针对性微调开源小模型，最后通过 RAG 架构聚合为宏观系统。该方案支持原生本地化私有部署并满足 HIPAA/SOC 2 等合规要求，不仅能大幅削减推理成本、杜绝影子 IT 隐患，还能在医疗、金融及跨国犯罪追踪等复杂严苛场景中保障高达 98% 的输出准确率。 ## Atmosphere & style 整场演讲务实利落且极具行业共鸣感。讲者开篇借助自闭症诊疗的高容错门槛与第三方模型突遭停服的现实危机，瞬间引发全场观众的深度认同；随后语调转为清晰自信的工程解构，穿插幽默生动的业务场景比喻化解复杂的治理难题，最终在硬核政企案例的佐证下，营造出兼具危机警醒与技术自主掌控力的专业现场氛围。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) 在医疗等强监管垂直领域，“大部分正确”的 AI 输出无法达标，极低的容错率要求模型具备绝对可靠的精确度。 - [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) 过度依赖第三方闭源大模型存在严重脆弱性，服务突遭下线或接口变更会直接威胁企业的业务连续性与数百万美元投入。 - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 依赖外部商业大模型本质上是“租赁智能”，不仅带来高昂成本，也导致企业无法真正沉淀自身的核心数据资产与技术壁垒。 - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 企业无需从零构建大模型，更实用的策略是将宏观业务拆解为垂直小任务，并在开源权重基座上进行针对性微调。 - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 企业现有的 AI Agent 是天然的数据生成器，通过捕获其连续 30 天的高频决策轨迹即可构建模型训练飞轮。 - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 梳理重复决策模式并构建包含优质标准、失败样本与边缘案例（Edge Cases）的企业专有 Eval 评测集，是成功微调的关键步骤。 - [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 通过 RAG 架构聚合多个专用小模型承接常规高频任务，仅在少数复杂场景调用前沿大模型，能以极低成本匹配顶尖模型性能。 - [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 员工使用个人账号等“影子 IT”行为会导致核心业务数据外流，建立私有数据资产与专属模型可消除合规隐患并提升企业估值。 - [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 企业级落地需要默认支持本地化私有部署（On-prem），并原生满足 HIPAA、SOC 2 等强监管行业的合规审计与安全护栏要求。 - [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 系统内置的自我监控与自维护机制，有效解决了业务人员自主搭建 AI 应用后缺乏专业技术人员负责后续运维的难题。 - [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 任务级微调方案可将模型输出准确率提升至 98%，已在跨国犯罪追踪、医疗长期护理及金融等极端复杂场景中得到实战验证。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) 科技浪潮的宠儿与高门槛监管行业的 AI 困境每一次技术革命都会诞生属于那个时代的“行业宠儿”——云计算时代诞生了 Salesforce，零工经济带来了 Uber，而如今则是人工智能的天下，前沿大模型公司迅速跃升为全球最具价值的企业。然而，对于身处特定垂直领域的创业者而言，AI 的落地却伴随着截然不同的现实考验。以 Ascent Autism 的联合创始人兼 CEO 兼医生 Dr. Haley 为例，其团队致力于为自闭症儿童提供线上团体治疗。在医疗健康这种强监管行业中，专家经验与客户基础固然完备，外界也反复宣传“AI 是拯救一切的弥赛亚、能重塑利润率并打造下一家明星企业”，但这里的容错率极低。AI 的一次决策不仅关乎医疗服务获取，更直接影响家长对孩子的认知以及孩子对世界的看法，在关乎人类生命的场景下，“大部分正确”（"mostly right isn't good enough"）根本无法达标。数周前 Fable 5 突遭下线停服的事件，令现场几乎所有从业者感同身受地举手共鸣。这一突发状况揭示了当前 AI 生态的脆弱性：企业正将数百万美元的预算投入到自己既不拥有、也无法完全掌控的技术之中，在充满不确定性的浪潮中艰难求存。 - **行业容错率反差**：普通消费级应用可以接受概率性错误，但在医疗与强监管领域，模型输出的微小偏差可能造成不可逆的现实后果。 - **依赖外部大模型的风险**：核心业务若完全寄生在第三方闭源模型上，一旦遭遇停机、接口变更或服务下线，企业的业务连续性将面临巨大威胁。 --- ### [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 摆脱“租赁智能”：构建企业专属模型的飞轮框架企业面临的高昂成本、缺乏掌控权与缺乏数据所有权，本质上都指向同一个根源——企业在“租赁智能”（renting intelligence）。包括微软的 Satya Nadella、Palantir 的 Alex Karp 以及 Hugging Face 的 Clem Delangue 在内的诸多行业领袖，均指出企业必须训练专属自定义模型。Empromptu AI 团队（由 Shanea Leven 与曾参与卫星发射的计算天体物理学博士 Sean Robinson 创立）长期主张：企业无需从零构建一个庞大的 ChatGPT，而是应当将大任务拆解为小任务，基于开源权重模型（open-weight models）进行高效微调。企业不必推倒重来，其现有工作流中部署的 AI Agent 本身就是最好的数据生成器。通过捕获 Agent 的运行轨迹并加以治理，即可构建起高效的模型微调飞轮。具体实施可归纳为清晰的四步框架： ```mermaid flowchart TD accTitle: 从现有 Agent 到自定义模型的训练飞轮 subgraph Capture["1. 数据捕获与观察"] A["选定高频业务 Agent"] --> B["连续 30 天捕获运行输出与轨迹"] end subgraph Eval["2. 评测集与规律提炼"] B --> C["识别重复性决策模式"] C --> D["构建评测集 (Eval Set)<br/>• 优质标准 / 劣质样本<br/>• 边缘案例 (Edge Cases)"] end subgraph FineTune["3. 针对性微调"] D --> E["基于开源模型微调任务级小模型"] end subgraph Scale["4. 系统聚合与协同"] E --> F["利用 RAG / 架构聚合多个小模型"] F --> G["复杂宏观任务调用 Frontier 模型<br/>常规高频任务由小模型承接"] end ``` - **第一步：锁定高频 Agent**：挑选业务中调用频次最高的 Agent，连续 30 天完整捕获其决策路径与产出。 - **第二步：提炼重复决策并构建 Eval 集**：梳理常见决策模式，精准定义“何为良好输出”、“何为劣质输出”，收集修正记录、失败样本、用户偏好以及边缘案例（edge cases），将其转化为企业专有的评估基准（Eval）。 - **第三步：微调专项小模型**：利用 Eval 数据集，在开源权重基座上训练仅负责特定单一任务的轻量化模型。 - **第四步：模块化聚合与成本优化**：持续针对不同任务重复该流程，借助 RAG 架构将多个专用小模型聚合为宏观系统。高频常规任务交由自建小模型处理，仅在极少数复杂场景调用前沿大模型，从而大幅削减推理成本并沉淀企业数字资产。 --- ### [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 现场问答：数据隐私、本地化部署与企业级治理在演讲后的互动问答中，现场观众就数据安全与架构落地提出了关键问题。有观众提到，虽然明知将数据暴露给前沿大模型存在风险，但很多企业为了紧跟技术步伐依然选择冒险。Shanea 指出，即便签署了企业级协议，若员工在日常工作中随意使用个人 Claude Code 或 OpenAI 账号（即影子 IT，Shadow IT），核心业务数据依然会源源不断地流入外部模型。拥有自主数据资产并自建模型，不仅能杜绝数据外流，还能直接提升企业的市场估值。针对自托管与企业级合规的需求，Shanea 详细阐述了平台的工程落地能力： - **本地化部署与合规认证**：系统默认支持本地化私有部署（On-prem by default），原生符合 HIPAA 与 SOC 2 合规要求，全方位适配医疗、金融及政府等强监管行业。 - **内置全套治理体系**：开箱即用配备审计日志（Audit logs）、安全护栏（Guardrails）与企业级 AI 策略管控，有效消除影子 IT 带来的合规隐患。 - **自我监控与自维护机制**：针对业务人员自主搭建应用后的维护难题，Shanea 幽默地举例道：“如果财务部的 Debbie 自己搭建了一个发票处理应用，她显然不会负责后续的技术运维”，而平台内置的自我监控与自维护机制可以确保应用长期稳定运行。 - **高精度与复杂场景验证**：联合创始人研发的核心技术可使模型输出准确率达到 98%，彻底规避低质无用的“AI 垃圾”（AI slop）。该方案已在诸多极端复杂场景中落地，包括与前美国缉毒局（DEA）负责人合作构建的政府级跨国犯罪追踪模型，以及长期护理机构、医疗健康组织、金融证券与私募股权（PE）机构的核心业务系统。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) 在关乎人类生命的场景下，'大部分正确'（mostly right isn't good enough）根本无法达标。 — Shanea Leven > [00:04](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=4s) 企业正将数百万美元的预算投入到自己既不拥有、也无法完全掌控的技术之中，在充满不确定性的浪潮中艰难求存。 — Shanea Leven > [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 企业面临的高昂成本、缺乏掌控权与缺乏数据所有权，本质上都指向同一个根源——企业在'租赁智能'（renting intelligence）。 — Shanea Leven > [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 企业无需从零构建一个庞大的 ChatGPT，而是应当将大任务拆解为小任务，基于开源权重模型（open-weight models）进行高效微调。 — Shanea Leven > [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 企业不必推倒重来，其现有工作流中部署的 AI Agent 本身就是最好的数据生成器。 — Shanea Leven > [02:25](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=145s) 高频常规任务交由自建小模型处理，仅在极少数复杂场景调用前沿大模型，从而大幅削减推理成本并沉淀企业数字资产。 — Shanea Leven > [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 即便签署了企业级协议，若员工在日常工作中随意使用个人 Claude Code 或 OpenAI 账号（即影子 IT，Shadow IT），核心业务数据依然会源源不断地流入外部模型。 — Shanea Leven > [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 拥有自主数据资产并自建模型，不仅能杜绝数据外流，还能直接提升企业的市场估值。 — Shanea Leven > [06:27](https://www.youtube.com/watch?v=c39bjpPOW_Q&t=387s) 如果财务部的 Debbie 自己搭建了一个发票处理应用，她显然不会负责后续的技术运维。 — Shanea Leven ## Mentioned Shanea Leven, Empromptu AI, Sean Robinson, Dr. Haley, Ascent Autism, Satya Nadella, Microsoft, Alex Karp, Palantir, Clem Delangue, Hugging Face, Fable 5, OpenAI, Claude Code, ChatGPT, Salesforce, Uber, DEA (美国缉毒局), HIPAA, SOC 2, Ai4 ## Video description Shanea Leven - CEO & Co-Founder, Empromptu AI Most teams don't realize their deployed agents are already generating the training data they need. This talk walks through how to capture agent traces, filter for high-quality examples, and fine-tune smaller models that match frontier performance on your specific tasks — at a fraction of the cost. You'll leave with a practical framework for turning your existing AI infrastructure into a model training flywheel. [Source](https://www.youtube.com/watch?v=c39bjpPOW_Q)

Source quotes

[0:04] Shanea Leven: 在关乎人类生命的场景下，'大部分正确'（mostly right isn't good enough）根本无法达标。 [0:04] Shanea Leven: 企业正将数百万美元的预算投入到自己既不拥有、也无法完全掌控的技术之中，在充满不确定性的浪潮中艰难求存。 [2:25] Shanea Leven: 企业面临的高昂成本、缺乏掌控权与缺乏数据所有权，本质上都指向同一个根源——企业在'租赁智能'（renting intelligence）。 [2:25] Shanea Leven: 企业无需从零构建一个庞大的 ChatGPT，而是应当将大任务拆解为小任务，基于开源权重模型（open-weight models）进行高效微调。 [2:25] Shanea Leven: 企业不必推倒重来，其现有工作流中部署的 AI Agent 本身就是最好的数据生成器。 [2:25] Shanea Leven: 高频常规任务交由自建小模型处理，仅在极少数复杂场景调用前沿大模型，从而大幅削减推理成本并沉淀企业数字资产。 [6:27] Shanea Leven: 即便签署了企业级协议，若员工在日常工作中随意使用个人 Claude Code 或 OpenAI 账号（即影子 IT，Shadow IT），核心业务数据依然会源源不断地流入外部模型。 [6:27] Shanea Leven: 拥有自主数据资产并自建模型，不仅能杜绝数据外流，还能直接提升企业的市场估值。 [6:27] Shanea Leven: 如果财务部的 Debbie 自己搭建了一个发票处理应用，她显然不会负责后续的技术运维。

Source quotes

[0:04] Shanea Leven: mostly right isn't good enough [0:04] Shanea Leven: messiah [0:04] Shanea Leven: sea of endless uncertainty [4:07] Shanea Leven: evals [5:04] Shanea Leven: a boatload of money [6:28] Shanea Leven: Debbie from accounting decided that she wanted to build an invoicing app [6:28] Shanea Leven: AI slop [5:04] Shanea Leven: Select a high-volume agent

Original transcript

[0:04] Um, so every uh tech revolution has its darlings. Uh, cloud gave us Salesforce, [0:11] the gig economy gave us Uber, and now we have AI. And the frontier model companies are already some [0:17] of the most valuable companies um in the world. But who else wants to create one of these darling [0:24] companies? I'm sure all of you do. Meet Dr. Haley. Uh she is the co-founder and CEO of Ascent Autism [0:32] where her team provides online group therapy for children with autism. This is a massive problem um [0:38] in healthcare and in regulated industries. She has the expertise, she has the customers um and she's [0:44] been told over and over again like all of us that AI is the future, not merely another technology, [0:51] but this messiah that's coming in uh to change the company. It's going to save all of us, [0:56] transform its margins, and be the next darling, or at least get the attention of one. And there's [1:02] only one problem. She operates in a regulated industry. Um, when her AI makes a decision, [1:08] that decision might affect whether someone receives health care, how a parent perceives [1:14] that child, or how that uh that child perceives the world. Um, mostly right isn't good enough. And [1:20] if you guys are building AI applications, mostly right is probably not good enough for you either. [1:26] Um, when human lives are not on the line, shouldn't we all expect our companies to get [1:33] the returns of the millions of dollars that we're putting in based on these experiments? [1:39] Now, a few weeks ago, we all had a glimpse of just how fragile this future can be. Fable 5 was taken [1:46] offline. Please raise your hand if you remember a few weeks ago Fable 5 was taken offline. [1:50] Basically everybody. It brought back obviously a glimmer of this reality that we are investing [1:59] millions and millions of dollars into a technology that we do not own. We cannot fully control. [2:04] And many of us um have uh don't have the access uh to make sure that we can make valuable decision [2:11] for the hope that you guys are working for or you're building a company who could be a darling. [2:17] Um and we are all trying to survive in this sea of endless uncertainty. But here's the secret. [2:25] The cost, the ownership, and the control is all stemming from the same root cause. [2:34] um we are renting intelligence um that we're building uh our companies on [2:40] and these guys actually told us what the solution is. Did everyone know who these people are? Yes, [2:44] I see everybody nodding. If you don't, that's uh Satia of uh Microsoft um Alan Karp from Palanteer [2:52] and Clem from Hugging Face. They're some of the biggest leaders in the world telling us that we [2:57] need to train our own custom models. Um, we need to have our own intelligence that we can own. [3:05] Um, and as it says, I'm also saying that too. So, who am I? I'm Shanea. I'm the CEO of a [3:11] company called Impromptu AI and my co-founder, Dr. Sean Robinson, um, has a PhD in computational [3:17] astrophysics um, put satellites into space. We've been saying this for many, many years, [3:22] even helping regulated industries, um, get real returns with AI without the cost burden. [3:29] Um, our belief is very very simple. Every company, everyone in this audience who works for a company [3:34] should have their own models. But if you are thinking, I'm just trying to survive. I don't have [3:40] the time or the infrastructure or the data space or anything like that to train a custom model, [3:46] the truth is that you do not need to recreate chat GBT to be able to use a custom model. Um, [3:54] there are ways that you can fine-tune a custom model in a very simple way [3:59] for small tasks that then add up to build up one overarching model. That's the old way. [4:07] The new way is you can take your agents uh that you already have or you can build [4:13] agentic uh applications in your workflow or out of your workflow so that you can produce the data, [4:20] take that data, clean it, make sure the agents have governance around it to make sure that you [4:27] put in the framework to collect that data to then use that to create fine-tune models. [4:34] Um you can use an openw weight model and then the data coming out of your apps use that to train. [4:39] Make sure that you can create um collect the corrections, the failures, the preferences and [4:46] any of those edge cases. Those are the things that you should be collecting and turn those into eval. [4:52] if you aren't making eval those are incredibly important and then use that to essentially [4:58] uh preserve your expertise in the company and use that to fi fine-tune your own custom models. [5:04] So how does this work? This sounds like a lot of work. It can be but there's very there's [5:09] four steps very very simple. You choose a high volume agent. You capture its outcome for 30 days. [5:16] You identify the repetitive decisions um and you build an eval set based on those repetitive [5:22] decisions. What good looks like, what not so good looks like, what are the edge cases and what do [5:27] you do um as a result of those edge cases. Um and then you test use that to build a smaller [5:34] model for just that task. Um, and if you uh have that smaller model, you just keep doing that [5:42] over and over and over again until you build up to a larger overarching model. You use a rag system [5:49] that you're able to aggregate those smaller models for one larger model. then you actually [5:54] have the flexibility to move things forward and backwards or control a smaller task for smaller [6:02] um outcomes and then use frontier models for larger outcomes saving you a boatload of money. [6:08] Um if you're still skeptical like Dr. Lee was um I invite you to bring impromptu [6:13] one of your high volume agents um and we can actually prove to you [6:17] how much time and money you can save. Um, we're giving away free time with with us, [6:22] but we also are able to take some qu I'm also able to take some questions right now. [6:27] Um, let's find out what you already know. Thank you. Does anyone have any questions? [6:33] Is anyone in the room also building custom models? Are you uh is everything perfect in your world? [6:40] You have a question? Are you excited to give your data to the frontier models? uh it's a risk uh but [6:47] always to expose the data uh to frontier model in order to tune them but uh I think the company has [6:52] decided to take that risk to go ahead uh to be more relevant. Yeah. So we we're on it for now. [6:58] Got it. Um well, typically what we see is that um if even if you're on an enterprise contract, [7:04] you're still collecting or if your employees are using their own agents or they're using [7:10] their own clog code accounts or uh their open AI accounts, those data points actually go right into [7:15] the model. Um so it's very helpful to have your own systems to be able to even do smaller things. [7:21] Also, it actually creates makes your company much more valuable if you own your own data asset [7:27] um as opposed to renting the intelligence from the frontier model. So, you can actually increase the [7:32] valuation and the value of the company. Okay. Since you mentioned the question, [7:36] is this solution going to be like a self-hosted somewhere? Yes, we are onrem by default. [7:41] Um we're HIPPA sock 2 compliant. Um, we work with a lot of regulated industries, healthcare, we have [7:47] everything built in, audit logs, guard rails, AI policy. So, if you're dealing with shadow IT, um, [7:53] you can actually embed the policies that you want all of your teammates to use. You don't have to [7:58] worry about it. You can go on vacation. Okay. Um, we also have self-monitoring and self-maintenance [8:04] totally built in as well. So, if you're like, "Oh, my Debbie from accounting, she decided that [8:10] she wanted to build an invoicing app." Um, and Debbie's obviously not going to maintain that app. [8:16] That's okay because we built in self-maintenance as well. Um, uh, our technology, my co-founder, [8:23] uh, invented can get up to 98% accurate outputs out of any AI model. So, we definitely do not deal [8:30] in any AI slop. We've built such things that's as complicated as organized crime tracking models [8:36] for the government. We've been able to um, work with the former head of the DEA. We also work [8:41] with other healthcare organizations like Dr. Lee. Um we work with large long-term care facilities, [8:47] um security facilities, finance companies, uh PE, equity, uh all sorts of fun, good exciting things.