Headspace, Marriott Curb AI Agent Risks With Rigid Code, Small Models

Headspace与万豪通过严格代码与小模型遏制AI Agent风险 自主Agent在企业生产中往往失败，除非团队用针对特定任务的小模型替代前沿LLM 业内从业者深入剖析了自主AI Agent，强调Agent系统约90%的架构由确定性软件脚手架组成，而非原生基础模型。专家小组详细介绍了在心理健康、酒店管理和商业银行领域久经实战检验的工程策略，包括原生追踪的可观测性、分层模型路由以及旨在降低延迟的专用模型蒸馏。演讲者最后批评了毫无规范的“凭感觉编码”（vibe coding），倡导通过严谨的角色分解、合成评估流水线和严格的红队测试，以安全地规模化扩展生产级Agent。 AI Agent 本质上类似于操作系统，其整体架构中约 90% 由确定性软件脚手架、记忆模块和 API 集成构成，核心基础大模型仅占约 10%。 (3:35) 生产环境下的多 Agent 系统需要覆盖中间 Prompt、思维链和工具调用的全链路原生可观测性，并借助群体自对弈红队测试提前暴露异常行为与回归缺陷。 (8:37) Headspace 构建了包含数千个合成患者画像的模拟库（占测试流量的 95%），结合临床心理学量表与 LLM 评审员对 AI 治疗师进行高频、严苛的安全基准评测。 (10:37) 面对安全校验链路的延迟瓶颈，通过将特定领域的分类任务蒸馏至微型模型（如 DistilBERT），在不牺牲准确率的前提下将审查延迟从 900 毫秒骤降至 60 毫秒。 (16:49) 串联的 Agent 工作流会成倍放大单步误差，而在实时语音等低延迟场景下，系统可通过前置生成对话填充语来掩盖后台批判 Agent 的多轮验证停顿。 (20:46)

Deploying autonomous AI agents into enterprise production requires treating foundation models as small reasoning components within deterministic software scaffolding, while actively managing compounding latency, evaluation fidelity, and governance boundaries. Architectural Scaffolding Over Raw Foundation Models (3:35) Panelists demystify autonomous agents by framing them as operating systems that coordinate tools, memory, and asynchronous workflows around one or more language models. Alina Rivilis notes that roughly 90% of a production agentic system consists of traditional deterministic software scaffolding, API connectors, and state tracking, leaving only 10% to the raw foundation model. Nitin Kumar emphasizes that architectural choices must mirror operational risk and problem variability, ranging from rigid sequential pipelines—such as Marriott's property copy generation—to parallel worker swarms and generator-critic loops. For enterprise frameworks, the speakers caution against excessive framework lock-in or relying on visual, drag-and-drop builders. While GUI platforms offer fast onboarding, they quickly become brittle and unmaintainable for complex production execution graphs. Instead, code-first frameworks like LangGraph scale significantly better because they allow teams to maintain stateful scratchpads and pair with AI-native IDEs to configure testable, modular graph nodes directly in code. Hierarchical Routing and Distillation for Latency (16:49) Daisy-chaining frontier models across multi-step agent topologies generates severe operational bottlenecks in execution latency and token costs. To prevent compounding expenses, Kumar advocates for hierarchical routing: delegating routine, bounded classifications to lightweight models while reserving large foundation models strictly for ambiguous reasoning nodes. Specialized high-speed inference providers, such as Groq and Baseten, further optimize execution at the kernel level using techniques like PagedAttention and quantization. Where off-the-shelf guardrails introduce unacceptable delay, domain-specific distillation provides a viable alternative. Akhil Chaturvedi explains that cloud moderation services initially introduced around 900 milliseconds of latency in Headspace's clinical mental health workflows. By labeling domain data with frontier LLMs and distilling that intelligence into a compact DistilBERT model—roughly 1,000 times smaller than GPT-4—Headspace slashed safety verification latency to 60 milliseconds. For interactive, real-time voice applications, teams combine these optimizations with speculative decoding techniques, streaming conversational fillers to mask background verification passes. Trace Observability, Simulation, and Adversarial Testing (8:37) Because chained agent interactions compound individual failure rates exponentially, monitoring end-to-end execution requires granular, trace-native observability. Dr. Yizheng Wang highlights that engineering teams must capture complete execution graphs—including prompt versions, intermediate scratchpads, and external tool payloads—to identify precisely where reasoning loops derail. In addition, defending against semantic threats like indirect prompt injection requires continuous red teaming, pitting autonomous agents against one another in simulated sandbox playgrounds to detect edge-case behavioral regressions prior to release. In safety-critical applications, evaluation relies heavily on synthetic populations. Chaturvedi outlines Headspace's benchmarking methodology for its AI therapist, which converts clinical psychology rubrics—such as empathetic listening, therapeutic alliance, and client resistance—into automated scorecards evaluated by LLM judges like Anthropic's Claude. To achieve continuous testing velocity without risking end-user safety, 95% of Headspace's evaluation pipeline uses synthetic patient personas, while the remaining 5% benchmarks gold-standard human therapy sessions alongside licensed clinicians. Enterprise Security, Human Oversight, and Clinical Ethics (37:25) Deploying agents into regulated environments requires strict enforcement of the principle of least privilege, ensuring models operate within bounded database and tool permissions. Rivilis points out that while human intermediaries cannot be placed inside rapid internal agent transactions without degrading system performance, human-in-the-loop checkpoints remain mandatory at outward-facing enterprise gateways before customer communications are released. Healthcare and banking pipelines also enforce data sanitization; for instance, Headspace routes all telemetry through a local VPC de-identification layer to strip protected health information before data reaches third-party observability platforms. Addressing the ethical implications of customer-facing autonomy, Chaturvedi argues that conversational agents provide a necessary paradigm shift to address massive structural healthcare shortages, noting that fewer than 2% to 3% of individuals needing clinical mental health support receive it. Always-available agents bridge acute, intermittent vulnerability moments that traditional, costly biweekly therapy models cannot serve, framing safe autonomous systems as an access expander rather than a replacement for human practitioners. The panel concludes that moving agents from exploratory prototyping to production requires abandoning unconstrained 'vibe coding' in favor of disciplined unit testing, strict role decomposition to reduce policy entropy, and starting with simple sequential topologies focused on high-frequency business bottlenecks before expanding agent complexity.

在Ai4研讨会上，来自Headspace、Straiker、万豪国际与Home Trust Company的技术主管指出，自主AI智能体的稳定运行本质上依赖占系统90%的确定性软件工程脚手架而非纯基础模型；企业若要在生产环境中落地智能体，必须在延迟、成本、复合幻觉与安全边界之间建立严格的工程约束。 智能体架构：确定性软件与模型中枢 (3:35) Akhil Chaturvedi指出，AI智能体本质上是一个以LLM为核心的“操作系统”，负责调用外部API、检索结构化与非结构化数据库、运行异步子任务并协调多步推理循环。Alina Rivilis强调，生产级智能体系统中约90%是由确定性软件脚手架、指令、状态记忆模块和接口构成的，基础模型本身仅占约10%。针对实际业务需求，Nitin Kumar总结了三种常见工程模式：执行明确规则的顺序工作流（如万豪抓取属性数据并生成网页文案）、并发拆解任务后汇总的并行蜂群工作流，以及由独立批判智能体审查安全合规的自我批判循环。 Yizheng Wang提醒，硅谷当前存在过度炒作“智能体”的倾向，工程团队应避免为了自治性而过度设计那些普通确定性流程即可解决的问题。系统必须维护显式的状态控制循环以管理草稿记忆与工具输入输出，从而确保生产调试时的细粒度可观测性。 全链路追踪、模拟评估与对抗红队测试 (8:37) 当多个智能体产生依赖并协同运行时，排查推理中断与行为异常变得极度复杂。Yizheng Wang提出必须采用原生追踪可观测性（trace-native observability），全程记录初始提示词、思维链过程、工具输入、API响应及最终输出。为防范上线后的意外动作，Straiker利用基于种群的自我博弈构建红队模拟靶场，让自主智能体相互对抗攻击，在部署前暴露边缘失效模式与潜在越权风险。 在心理健康等高风险垂直领域，评估体系直接关乎安全底线。Akhil Chaturvedi介绍了Headspace为AI心理咨询师建立的评测基准：团队将建立共情、识别根因、反思性提问等临床心理学维度转化为量化细则，利用Anthropic的Claude作为自动评审员（LLM-as-a-Judge）。在测试流量中，约95%由包含数千种虚拟画像的合成患者模型承担以实现快速验证，剩余5%采用真实人类咨询记录，由持牌临床医生与大模型按照相同评分标准实施双盲评测。 延迟与成本控制：分层路由与模型蒸馏 (16:49) 生产环境面临调用成本与响应延迟的双重压力。Nitin Kumar倡导“分层路由”策略：将简单的分类与常规任务分发给轻量、低成本模型，仅在复杂推理与模糊决策节点调用高参数前沿大模型。Akhil Chaturvedi进一步指出，多层安全审查往往会拖垮系统响应。针对自杀倾向、药物滥用等细分风险，通用云端防护栏难以精确识别且耗时过长，Headspace通过大模型标注垂直数据，将专业判别能力蒸馏到DistilBERT模型（体积仅为GPT-4的千分之一左右），将安全分类延迟从900毫秒大幅压缩至60毫秒。 针对实时交互场景，多层验证造成的停顿会严重破坏用户体验。工程团队一方面接入Groq、Baseten等专业推理平台，利用PagedAttention与量化技术实现硬件级加速；另一方面借鉴推测解码理念，在系统后台批判智能体校验内容期间，优先流式输出填充短语以掩蔽延迟。然而，多模型与多供应商调度也增加了敏感数据外泄的风险，系统必须在工具调用层面配置严格的Schema输入输出校验。 企业级安全治理：最小权限与代码优先 (37:25) 复合错误率是智能体链式调用的核心隐患，即使单个LLM调用仅有1%的错误率，串联多次交互后系统崩溃概率也会显著放大。因此，企业级落地必须建立深度防御。Nitin Kumar强调，智能体必须遵循最小权限原则，严格限制工具执行范围与数据库访问边界，并在输入端清洗个人身份信息（PII）。Alina Rivilis补充道，虽然智能体内部交互无法容忍人工介入的延迟，但在面向客户的外部边界上，高风险输出必须设置人类审批员（Human-in-the-Loop）。 在架构选型上，与会专家倾向于基于代码的框架（如LangGraph）而非无代码拖拽工具。面对复杂的状态机和业务逻辑，结合AI原生编程助手直接编写代码比在图形界面中配置连线更加灵活且便于维护。在合规层面，金融与医疗机构正逐步从高维护成本的本地纯私有部署，转向符合HIPAA标准并签署业务合作协议（BAA）的受控混合云或专有VPC架构。 伦理落地：从原型探索迈向严谨工程 (40:17) 面向终端用户的自动化应用引发了深刻伦理考量。Akhil Chaturvedi谈到，全球心理健康资源存在巨大供需缺口，即使在发达国家也有超过97%的需求未被满足，高昂费用与专业医师匮乏使得常态化支持无法普及；具备即时响应能力的AI辅助系统不会取代专业医生，而是开创了填补深夜失眠、考前焦虑等急性脆弱期的新范式。 针对工程部署路线，与会专家一致提醒企业警惕随意原型编码（vibe coding）的泛滥。Yizheng Wang强调必须通过角色解耦来降低智能体的“策略熵”，防止通用代理越权产生不可逆破坏；Nitin Kumar建议团队采取“由慢到快”策略，先从1至2个确定性顺序智能体起步，充分测试环境记忆、对齐机制与边界安全，切忌盲目构建复杂的拓扑网络。Alina Rivilis总结表示，企业应聚焦高频痛点以获取明确的业务回报，而非单纯为了追逐新技术而部署模型。 研讨会表明，自主AI智能体正从概念验证迈向严谨的软件工程落地阶段。尽管在垂直审查、流程自动化及心理支持等领域已展现明确效能，但面对链式调用引发的错误级联放大、间接提示注入攻击以及高阶推理的延迟瓶颈，行业仍需保持工程克制，通过角色解耦、专用蒸馏与严格的边界控制循序推进。

## Background 该视频由 YouTube 频道 Ai4 于 2025 年 8 月 27 日发布。内容为一场关于构建与部署自主 AI Agent（智能体）的技术研讨会，由来自 Headspace、Straiker、万豪国际（Marriott International）和加拿大住房信托公司（Home Trust Company）的四位资深专家共同探讨多智能体架构、推理决策、扩展挑战及落地伦理等现实议题。 ## TL;DR 多位来自医疗、金融、安全与酒店领域的工程专家指出，AI Agent 系统本质上是“90% 的软件工程加上 10% 的 AI”，绝不能脱离传统工程规范而盲目追求概念炒作。讨论系统梳理了串行、并行聚合与批判评估等核心架构模式，并针对多 Agent 落地中的三大现实阻碍——高昂成本、级联幻觉与延迟瓶颈，分享了模型蒸馏、分层路由和原生追踪等工程化解法。此外，专家们结合真实业务深入拆解了医疗场景中“LLM-as-a-Judge”的合成评测方案与银行按揭流水线的合规实践，一致呼吁开发者在生产环境中克制技术狂热，从简起步并严格落实红蓝对抗与单元测试。 ## Atmosphere & style 整场研讨会散发着硅谷一线工程团队特有的清醒与务实气息，彻底褪去了大模型炒作初期的狂热浮躁。专家们在台上的互动直率而默契，既有像“90% 是软件工程”或“别为了做 Agent 而做 Agent”这样引人会心一笑的行业金句，也有直面自残识别延迟、多智能体级联失控等高危场景时的严肃与严谨。聆听体验如同置身一场高质量的内部架构复盘会，在坦诚拆解生产踩坑经验的同时，也展现了对技术普惠社会向善的深层思考与坚定信念。 ## Key takeaways - [03:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=217s) 智能体系统本质上是“以大模型为内核的操作系统”，技术构成中约 90% 是软件工程，仅 10% 是 AI，核心在于构建任务导向的确定性软件架构。 - [05:59](https://www.youtube.com/watch?v=89UR-fjDA_I&t=359s) 企业级智能体常见三种核心模式：确定性规则的顺序执行模式、宽泛调研的并行与聚合模式，以及用于安全把关的自我批判模式。 - [07:29](https://www.youtube.com/watch?v=89UR-fjDA_I&t=449s) 工程实现应博采各框架所长并构建强控的“有状态循环”，避免单一框架绑定，且坚决不应在确定性规则已足够的高置信场景中为赶时髦而滥用 Agent。 - [08:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=517s) 面对多智能体复杂黑盒，需依赖原生链路追踪可观测性，并通过沙盒内的基于种群自我博弈（Self-play）提前挖掘极端边界故障。 - [11:50](https://www.youtube.com/watch?v=89UR-fjDA_I&t=710s) 垂类 Agent 质量评测可引入 LLM-as-a-Judge 裁判机制，结合大规模合成虚拟对象（如数千名虚拟患者）与少量真人对照实现高频自动化回归。 - [15:03](https://www.youtube.com/watch?v=89UR-fjDA_I&t=903s) 为防止多智能体调用导致成本失控与延迟膨胀，应采用分层路由架构，将清晰任务分派给轻量小模型，仅深度推理时调用前沿旗舰大模型。 - [16:49](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1009s) 通过领域知识蒸馏将通用大模型压缩为专有轻量模型（如 DistilBERT），可将关键安全判定延迟降低 90% 以上，有效破解端到端延迟瓶颈。 - [19:40](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1180s) 多智能体协同存在级联幻觉放大风险，架构上必须在输入输出两端建立严密的 Schema 约束、PII 过滤以及明确的系统问责机制。 - [22:39](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1359s) 在质量与延迟之间需做场景化权衡：离线异步任务可启用多层 Critic 模型深度校验，实时交互则应采取投机式快速返回缓冲时间。 - [25:28](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1528s) 上线前必须开展融合网络安全与 AI 提示词注入的自动化红队测试，防御隐藏在邮件等外部输入中的间接提示注入攻击。 - [33:45](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2025s) 拖拽式低代码工具难以实现规模化维护，结合 AI 编程助手（如 Cursor）的代码优先编排框架（如 LangGraph）更适合构建生产级应用。 - [36:01](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2161s) 企业正逐步摆脱笨重的纯本地部署（On-Prem），转向利用云厂商 Marketplace 与 BAA/HIPAA 合规契约的“本地数据 + 云端算力”混合模式。 - [38:18](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2298s) AI 心理治疗师等敏感智能体的伦理正当性在于填补极度匮乏的专业医疗供给鸿沟，提供高可及性、低成本的即时陪伴与支持新范式。 - [43:11](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2591s) 企业级落地严禁在安全治理上搞“氛围式编程（Vibe Coding）”，初期应从小规模串行模式切入，积累足够控制力后再扩展至复杂多智能体协作。 - [45:44](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2744s) 开发演进应遵循“原型期快速探索、生产期严谨测试”原则，紧密对齐业务战略，将 Agent 优先部署于高频且具备明确商业痛点的核心工作流。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=89UR-fjDA_I&t=4s) 研讨会开场与嘉宾背景：从心理健康、AI安全到酒店业的智能体实践 研讨会伊始，主持人 Alina Rivilis 热情开场，并抛出互动话题，询问现场有多少观众已经对 AI Agent（智能体）充满期待，或已在生产环境中部署了智能体。随后，四位深耕不同产业场景的技术专家依次介绍了各自的业务探索与研究重心。 首先发言的是来自知名冥想与心理健康平台 Headspace 的技术主管兼主任工程师 Akhil Chaturvedi。他领导的团队正全力研发“AI 心理治疗师（AI Therapist）”。这项工作兼具科研探索与生产级工程落地，既要将严谨的心理学理论拆解并注入大语言模型（LLM）进行后训练（Post-training），又要设计精细的记忆架构来捕捉患者的多维度历史信息，最终交付一套端到端、高度拟人且合规的数字疗法体验。 紧接着，AI 安全初创公司 Straiker 的 AI 负责人 Yizheng Wang 博士作了自我介绍。作为毕业于斯坦福大学的自主决策系统博士，他目前专注于为企业级 AI 应用与智能体构建全方位的安全防护平台。其核心业务涵盖自动化红队对抗评估（Automated Red Teaming Assessment）与运行时安全护栏（Runtime Guardrails），旨在全流程拦截智能体系统的潜在风险。 万豪国际集团（Marriott International）的数据科学总监 Nitin Kumar 随后介绍了酒店巨头在智能化转型中的探索。驻扎在达拉斯的他带领团队将传统机器学习（Machine Learning）、大模型以及 Agent 方案深度融合，全面赋能法律合规、市场销售、客户关怀等跨部门业务线。最后，主持人 Alina 介绍了自己身兼多职的双重身份——她既是加拿大住房信托公司（Home Trust Company）的数据科学与 AI 总监，带领团队落地生成式 AI 应用，同时也是 AI 教育机构 AI Future Leaders 的创始人兼 CEO。 - **Akhil Chaturvedi**：Headspace 技术主管 / 主任工程师，负责将心理科学知识融入 LLM 后训练，结合患者长期记忆构建“AI 心理治疗师”。 - **Yizheng Wang**：Straiker AI 负责人（斯坦福大学自主决策系统博士），专注于企业级智能体安全平台、自动化红队评估与运行时护栏。 - **Nitin Kumar**：万豪国际集团数据科学总监，统筹传统 ML、LLM 及智能体解决方案在法务、销售、客服等全业务场景的落地。 - **Alina Rivilis**：加拿大住房信托公司数据科学与 AI 总监、AI Future Leaders 创始人兼 CEO。 --- ### [03:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=217s) 重新定义 AI Agent：“90% 的软件工程与 10% 的 AI” 在探讨复杂的架构设计前，Alina 向现场调查了大家对多智能体框架的理解程度，并邀请嘉宾从宏观层面为“AI Agent”划定清晰的定义。Akhil 给出了一个形象的工程化比喻：智能体本质上是一个“以单模型或多模型为内核的操作系统（Operating System）”。 这个操作系统被赋予了类似人类的行动能力——它可以自主调用工具、请求外部 API、从长短期记忆与数据库中检索上下文、执行异步后台任务来丰富知识库，并最终以远超单一基础模型的能力与用户交互。Akhil 以 ChatGPT 的“深度研究（Deep Research）”功能为例指出，其底层正是一群智能体在自主决策下一步该检索什么、思考什么、输出什么，并最终整合成一份结构严谨的深度报告。这就像是“一群协同工作的自主计算集群（a swarm of computers that are doing stuff autonomously）”。 Alina 极为赞同并精辟地总结道，从技术构成来看，智能体系统“大约 90% 是软件工程，只有 10% 是 AI”。它本质上是一个具备指令逻辑、任务导向和确定性行为的复杂软件系统。在多智能体体系中，Agent 之间各司其职、协同配合：例如由一个智能体专门负责撰写邮件草稿，第二个智能体负责合规与准确性校验，第三个智能体则根据需要将其翻译为目标语言。 ```mermaid flowchart TD accTitle: Multi-Agent System Architecture subgraph Core ["Agent 核心架构"] LLM["LLM 决策核心"] Mem[("长期/短期记忆")] Tools["工具 & API 集成"] LLM <--> Mem LLM <--> Tools end subgraph MultiAgent ["多智能体协同流水线示例"] A1["撰写 Agent\n(生成初稿)"] --> A2["验证 Agent\n(合规/纠错)"] A2 --> A3["翻译 Agent\n(多语言输出)"] end Core -.-> MultiAgent ``` --- ### [05:59](https://www.youtube.com/watch?v=89UR-fjDA_I&t=359s) 智能体架构的核心模式：顺序、并行聚合与自我批判 面对复杂的业务场景，如何选择适合的智能体设计模式？Nitin 结合万豪的实际落地经验，将常见的算法模式划分为三大类型，并指出架构选择取决于业务场景的风险收益权衡： 第一种是**顺序执行模式（Sequential Pattern）**。该模式适用于具有高度确定性规则的流水线。例如万豪在为旗下酒店生成营销网页内容时，流程非常明确：智能体首先通过工具定向拉取指定酒店物业的基础数据与设施信息，随后交由大语言模型按照预设模板生成文案。整个链路步步为营，确定性极高。 第二种是**并行与聚合模式（Parallel & Aggregator Pattern）**。正如 Akhil 提到的深度研究场景，当面对一个宽泛的复杂课题时，系统会派发多个子智能体同时在不同维度展开并行调研，各自分头检索并挖掘信息，最后将多路数据汇聚到一个中心聚合器（Aggregator）进行去重、交叉验证与结构化总结。 第三种是**自我批判模式（Self-Critic / Evaluator Pattern）**。在该模式下，一个智能体负责生成内容，另一个扮演独立裁判或质检员角色的智能体则介入评估生成质量、核查事实并施加安全护栏。通过双向博弈与对抗性校验，系统能够有效控制生成风险，特别适合对输出安全性要求极高的企业级场景。 --- ### [07:29](https://www.youtube.com/watch?v=89UR-fjDA_I&t=449s) 框架选择、去伪存真与有状态控制循环 针对业界琳琅满目的开发工具（如 LangChain、LangGraph、CrewAI 等），Yizheng 结合 Straiker 的研发实践，分享了构建企业级智能体的三条关键避坑法则： 首先是**组合优于单一依赖**。团队发现最有效的工程做法绝不是把自己绑定在某个单一框架上，而是根据具体功能模块，将多个框架的优势组件融合到同一个 Agent 架构中，实现工具生态与执行效率的最大化。 其次是**警惕概念炒作，避免“为做 Agent 而做 Agent”**。Yizheng 直言智能体在硅谷已经成为炙手可热的流行词，但工程团队必须保持清醒。智能体系统不仅推理与运行成本高昂，其输出行为也具有极强的不可预测性（hard to predict）。如果传统确定性规则或单次模型调用能解决问题，就绝不该盲目引入多智能体机制。 最后是**必须构建强控的“有状态循环（Stateful Loop）”**。在设计智能体底层逻辑时，开发者必须完全掌控状态机，包括模型思考的草稿本（Scratchpad）、长期记忆读写、工具输入输出等所有中间态。只有实现精确的状态追踪，团队才能在任意环节灵活插入安全护栏，并建立全面的系统可观测性（Observability），确保在出现偏差时能迅速定位根因。 - **多框架融合**：博采 LangChain、LangGraph、CrewAI 等框架所长，打造模块化混合架构。 - **成本与确定性权衡**：智能体成本高且难以预测，避免在无必要的高确定性任务中滥用 Agent。 - **掌控有状态循环**：对 Scratchpad、上下文记忆与工具 I/O 保持完全可控，保障护栏插拔与故障排查。 --- ### [08:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=517s) 生产级可观测性与对抗评估：Straiker 与 Headspace 的落地实践 当系统从单个 LLM 调用演进到 5 个、10 个甚至数十个智能体协同运作时，监控、治理与可观测性的复杂度呈指数级上升。Alina 针对如何跨越“从原型（Prototype）到生产（Production）”的鸿沟向嘉宾提问。 Yizheng 指出，作为安全公司，Straiker 的核心优势之一在于打造**原生追踪可观测性（Trace-native Observability）**。系统能够无死角捕获智能体从 Prompt 输入、工具调用参数、I/O 交互到中间推理链（Reasoning Process）的完整轨迹。此外，Straiker 搭建了内部评测沙盒（Playground），利用**基于种群的自我博弈（Population-based Self-play）**机制，让海量智能体在闭环环境中持续互相对抗交锋。这种对抗测试生成了极其宝贵的边界测试数据集，能够提前暴露智能体在哪些极端情况下会出现逻辑崩溃或做出荒唐举动（"did something stupid"），从而在上线前筑牢防线。 ```mermaid flowchart LR accTitle: Production Observability & Quality Assurance subgraph Straiker ["Straiker: 安全与可观测性"] T1["Trace-native 全链路追踪\n(Prompt/工具/推理链)"] T2["Playground 自我博弈\n(Population-based Self-play)"] T1 --> T3["边界故障与荒唐行为拦截"] T2 --> T3 end subgraph Headspace ["Headspace: 心理学专业评估体系"] H1["多维度交互信号\n(共情/倾听/诱因识别)"] H2["引导反思与促进行动\n(深入探索/预约转化)"] H1 --> H3["临床基准 Benchmark 评测"] H2 --> H3 end ``` 在垂直业务场景侧，Alina 邀请 Akhil 分享 Headspace 如何验证其 AI 心理治疗师输出的专业度与准确性。Akhil 在兼顾公司知识产权（IP）保密的前提下，深入阐释了心理健康领域的基准评测（Benchmarks）逻辑。 他指出，评估心理咨询不能仅看语言流畅度，而是要拆解专业治疗中的多维度关键信号（Signals）： 1. **建立融洽关系（Rapport Building）**：模型是否能够有效倾听并取得来访者的信任； 2. **深度问题剖析**：能否准确识别来访者生活中的核心触发诱因（Triggers/Causes）； 3. **启发式认知反思**：能否引导来访者主动向内剖析自我认知并逐步深化； 4. **行动闭环**：最终能否共同推导出可行的解决方案，甚至促成下一次咨询的预约。通过将这些心理学临床标准转化为严格的评估基准，团队才能量化衡量对话质量，确保 AI 在高敏感的医疗健康领域安全落地。 ### [11:50](https://www.youtube.com/watch?v=89UR-fjDA_I&t=710s) 使用大语言模型作为裁判与合成患者评估体系 在心理健康 AI 助手的评估中，评估标准涵盖了问题解决能力、治疗联盟（therapeutic alliance）的建立，以及面对患者抗拒改变时的应对反应等多个复杂维度。基于对这些评估标准的明确界定，团队构建了一套量化评分细则（rubric），并引入大语言模型（LLM）作为裁判来审查每一场对话。具体而言，系统采用基于混合专家大模型（MoE LLM）构建的基准测试套件，由 Anthropic 的 Claude 等前沿模型担任裁判，为 AI 心理治疗师生成的每条对话打分，如同基础模型在软件工程评测或数学推理基准测试中所经历的评估流程一样。 ```mermaid flowchart TD accTitle: AI 心理咨询 Agent 评估与基准测试流程 subgraph S1["对话生成源 (95% 合成 + 5% 真实)"] A["合成患者库 (数千名 LLM 患者)"] -->|95% 模拟对话| C["AI 心理治疗 Agent"] B["真实人类患者对话"] -->|5% 真实抽样| C end subgraph S2["LLM-as-a-Judge 评估层"] C --> D["专家大模型裁判 (如 Claude)"] E["临床评估量表 (治疗联盟/抗拒应对等)"] --> D end subgraph S3["反馈与迭代优化"] D --> F["基准评分与诊断报告"] F -->|持续迭代| G["架构调整 / 知识检索 / 记忆调优"] G -.-> C end ``` 为了使这一套基准测试能够实现高频、快速的自动化运行，研发团队构建了一个包含数千名“合成心理健康患者”的虚拟患者库。每次工程师对 AI 治疗师的智能体架构、检索机制（RAG）或记忆系统进行迭代修改时，系统都会让这上千名由 LLM 模拟的虚拟患者与 AI 治疗师展开全面对话并自动打分。在整体评测数据中，约 95% 依靠这套合成患者体系完成自动化快速回归，其余 5% 则采用真实人类对话，由专业心理咨询师按照相同的量化标准进行对照评估。 - **评估维度**：问题解决能力、治疗联盟建立、对患者抗拒改变的反应能力。 - **LLM-as-a-Judge**：使用 Claude 等大模型依据专业量表为每次会话打分。 - **评测数据结构**：95% 为数千名合成患者的模拟对话，5% 为真人临床对话基准验证。 --- ### [13:23](https://www.youtube.com/watch?v=89UR-fjDA_I&t=803s) 生产化落地考量：避免盲目“Agent 化”与渐进式开发 在探讨智能体方案的实际生产化部署时，首要的前提是辨别业务场景是否真正需要 Agent。很多时候，一个简单直观的固定工作流（workflow）或传统的机器人流程自动化（RPA）就已经足以高效解决问题，盲目引入 Agent 反而会增加系统的脆弱性。只有当业务问题本身具备极高的不确定性和多变性、必须依赖自主推理与动态决策时，才应当考虑构建 Agent 架构。 在具体落地路径上，建议采取“由慢到快、从小到大”的渐进式策略。项目初期应先从一到两个 Agent 开始小规模验证，仔细评估其在上下文传递、安全防护栏（guardrails）以及伦理合规等多个维度的表现，权衡该用例的风险与收益。智能体系统的构建本质上是一个高度迭代的过程，绝不能简单地“开发完成便直接上线，任由真实用户去发现缺陷”。只有在各项指标全面经过验证后，才能逐步扩大规模并推进到生产环境。 --- ### [15:03](https://www.youtube.com/watch?v=89UR-fjDA_I&t=903s) 多智能体系统的成本治理、链路追踪与分层路由 随着系统规模扩大，当 10 个甚至更多 Agent 协同工作且每个 Agent 都在频繁调用昂贵的前沿大模型时，推理成本会急剧飙升。尽管从长远技术发展趋势看模型单价在持续下降，但生产系统依然必须引入严格的可观测性链路追踪（trace），精确记录全链路中每个 Agent 的输入与输出 Token 消耗量，避免在账单上出现不可预期的“意外惊吓”。 ```mermaid flowchart LR accTitle: 分层路由成本与延迟优化架构 Req["用户/上游任务请求"] --> Router{"任务复杂度分流器"} subgraph L1["低成本快速执行层"] Router -->|直截了当的单任务| Small["轻量级/专有模型 (如 DistilBERT/小模型)"] Small --> Out1["低延迟输出 (<100ms)"] end subgraph L2["深度推理决策层"] Router -->|高复杂度/深度推理| Big["前沿大语言模型 (如 GPT-4 / Claude)"] Big --> Out2["高可靠复杂决策"] end ``` 为了在架构层面对冲成本，多智能体系统应当遵循“分层设计模式（Hierarchical Pattern）”。对于逻辑清晰、任务单一的直截了当节点，优先分派给低成本的小型模型或专用模型；仅在涉及复杂推理、多步规划和关键决策的节点上才调用大型旗舰模型。这种分级调用策略不仅能显著控制 Token 支出，还能有效避免大模型推理耗时过长所引发的系统端到端延迟膨胀。 --- ### [16:49](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1009s) 破解延迟瓶颈：从大模型蒸馏到专有硬件加速 在多智能体架构的工程实践中，延迟（Latency）、高质量检索（Retrieval）以及长期记忆（Memory）被视为最核心的三大技术瓶颈。面对多 Agent 相互校验与质检带来的端到端延迟叠加问题，业内正在从多个层面展开优化： 一方面是通过知识蒸馏（Distillation）实现模型轻量化。例如在心理健康场景中，由于公有云平台（如 Amazon Bedrock 或通用大模型自带的安全护栏）现有的通用防护机制无法满足对自残自杀倾向、饮食失调、药物滥用、家庭暴力等极端敏感行为的严格识别要求，团队利用大规模标注数据集与大模型，将领域特定的安全分类知识蒸馏进一个比 GPT-4 小上千倍的轻量级模型（如 2020 年推出的 DistilBERT）。这一改造直接将单次安全风险判定的延迟从通用大模型的 900 毫秒骤降至 60 毫秒，在保证特定领域极高召回率的同时大幅削减了时延。 另一方面是在基础设施与推理引擎层面加速。通过与 Baseten、Groq 等专注于极致推理加速的服务商合作，在 GPU 调度、底层内核（Kernel）优化、PagedAttention 内存管理机制以及模型量化（Quantization）等多维度进行深度调优，从而从根本上压缩单次推理的物理耗时。 - **专有安全防护需求**：涵盖自杀倾向、进食障碍、药物滥用、家庭暴力等特殊场景。 - **蒸馏成效**：使用 DistilBERT 将安全检测延迟从 900ms 降低至 60ms（降低 93% 以上）。 - **底层加速手段**：Kernel 级调优、PagedAttention、模型量化以及专有推理加速硬件。 --- ### [19:40](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1180s) 数据隐私、Schema 约束与多智能体级联错误治理 当企业系统同时接入多个供应商（如 OpenAI、Anthropic）的不同模型时，数据主权、个人身份信息（PII）以及敏感数据的合规管理会面临极高的复杂度。为了确保系统行为完全可控，架构设计上必须建立极其严密的 Schema 约束： - **输入端校验**：严格拦截直接提示注入（Direct Prompt Injection）、越狱攻击（Jailbreaking）以及间接提示注入（Indirect Prompt Injection）； - **输出端审查**：严防数据外泄（Data Exfiltration）、PII 泄露以及自残、有害言论等高危内容输出。 与此同时，系统的伦理与治理体系必须在立项之初就全面确立，厘清问责机制（当 Agent 犯错时由谁承担法律与业务责任）、推理透明度（模型决策链路是否可解释可回溯）以及算法偏差控制。 针对多智能体协同中最具破坏性的“级联幻觉（Cascading Hallucinations）”问题——即单个 LLM 即使只有 1% 的幻觉率，在经过 10 个乃至 100 个 Agent 串联流转后，错误会被级联放大，导致整个系统输出完全失真——行业早期尝试通过“增设校验 Agent 互相把关（Agent-on-Agent QA）”来纠错。然而这种方式不仅造成成本与延迟成倍增加，更导致系统架构变得庞大臃肿、难以维护，如何在控制复杂度与保证准确率之间取得平衡仍是当前工程部署中最严峻的挑战之一。 ### [22:39](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1359s) 治理困境、批判者大模型与质量-延迟的权衡取舍 管理复杂的多 Agent 架构时，系统治理（Governance）与管理层（Admin layer）的缺位会带来极大的安全隐患。在多 Agent 协同运行时，Agent 彼此之间通过 API 相互调用并发起对话，内部的数据流转与交互过程极难被实时看清。随着系统复杂度的攀升，诸如越狱（Jailbreak）和提示词注入（Prompt Injection）等攻击风险成倍增加，必须部署更完善的安全护栏（Guardrails）。讲者开玩笑称“所有公司现在都需要一个更强大的法务团队或驻场优秀律师”，但核心技术痛点在于：引入 Actor-Critic 架构中的“批判者（Critic）大模型”虽然能增强校验，却会严重拉长响应延迟。 工程落地的核心在于根据业务场景在**质量（Quality）**与**延迟（Latency）**之间做权衡： 1. **异步/离线 Agent 任务**：无需实时响应的场景（例如在用户夜间睡觉时跑 10 到 15 分钟的复杂工作流），可以调用多层审查与工具链，换取极致的交付质量。 2. **实时语音交互场景**：语音交互对延迟极其敏感，用户无法忍受停顿。此时可借鉴类似投机解码（Speculative Decoding）的策略——先由轻量模型快速生成诸如“问得好！”等前几个信息密度低、容错率高的 Token 稳住用户注意力，同时在后台留出时间窗口让其他 Agent 异步校验和生成后续关键语义内容。类似 Cursor 等代码生成工具在工程实践中也采用了“首包极速返回、后续精细生成”的平滑延迟手段。 ```mermaid flowchart TD accTitle: 质量与延迟的权衡及投机式交互架构 subgraph OfflineWorkflow["离线/异步工作流 (注重极致质量)"] A["用户提交任务 (如夜间)"] --> B["多 Agent 协同流 (10-15 分钟)"] B --> C["Critic LLM 深度审查与校验"] C --> D["次日呈现高质量结果"] end subgraph RealTimeVoice["实时交互工作流 (注重低延迟体验)"] E["用户实时语音/文本输入"] --> F["轻量模型快速输出前缀 Token<br/>('太棒了，这个问题...')"] F --> G["争取缓冲时间窗口"] G --> H["后台 Critic/验证 Agent 异步并行校验"] H --> I["输出精准的核心语义内容"] end ``` --- ### [25:28](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1528s) Agent 架构拆解与红队测试（Red Teaming）实战 针对多 Agent 系统的安全管控与架构设计，Yizheng Wang 分享了 Straiker 的内部最佳实践：首先建立原生的链路追踪可观测性（Trace-based Observability），穿透多 Agent 黑盒；其次必须对 Agent 明确分工并进行角色解耦（Role Decomposition），严禁“单 Agent 试图包揽一切”，常见的拆解角色包括监督者（Supervisor）、验证者（Verifier）以及检索者（Retriever），同时严格克制系统膨胀，将活跃 Agent 数量精简至最低。 Yizheng Wang 强调：“Agent 的本质是具备智能（Intelligence），而智能就意味着它一定会犯错。”在部署上线（乃至内部使用）之前，必须执行严格的**红队测试（Red Teaming）**。 - **超越传统渗透测试（Pentest）**：Agent 红队测试不是传统的网络安全渗透，它融合了系统工程、网络安全与 AI 提示词安全多维度。 - **间接提示词注入（Indirect Prompt Injection）防御**：例如攻击者向目标发送一封恶意邮件，当用户要求 AI 总结近两天邮件时，大模型一旦解析恶意邮件中的隐藏指令，就会将用户的个人敏感数据外泄给黑客。 - **自动化红队测试体系**：由于不存在一套通用的防御模板，他们基于客户真实交互数据构建了攻击战术库（Bank of Attack Tactics），并开发了“一键攻击模拟（One-Click Attack Simulation）”试验场，用于自动化压力测试。 --- ### [28:11](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1691s) 银行场景的多 Agent 文档智能流水线 来自加拿大第七大银行（Home Trust Company）的 Alina Rivilis 介绍了金融机构核心业务中的 Agent 落地场景。除了语音与邮件生成，银行最核心的场景是**文档智能与处理（Document Intelligence and Processing）**，特别是在按揭贷款（Mortgage）与存款运营业务中： ```mermaid flowchart LR accTitle: 银行按揭贷款文档智能多 Agent 流水线 subgraph Ingestion["文档摄取与校验"] A["扫描/抽取 Agent<br/>(专用打标与数据抽取)"] --> B["校验 Agent<br/>(验证内容完整性)"] end subgraph Decision["协同与入库"] B --> C["LLM 协同 Agent<br/>(语义核对/意图对齐)"] C --> D["CRM 写入 Agent<br/>(入库与系统集成)"] end subgraph Exceptions["异常与复用模块"] C -.-> E["异常上报 Agent<br/>(数据缺失/不一致报警)"] C -.-> F["法语翻译 Agent<br/>(可跨业务复用)"] end ``` - **高可复用性设计**：构建 Agent 时尽量解耦具体业务逻辑。例如上述法语翻译 Agent，不仅服务于贷款流程，还能被其他跨业务系统无缝调用。 - **治理工具匮乏**：随着流水线中的 Agent 越加越多，整个系统变得极其臃肿庞大。当前市场上缺乏成熟、标准化的多 Agent 统一治理工具，迫使团队必须在“自建（Build）”与“外采（Buy）”之间作出抉择。 --- ### [30:31](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1831s) 自建 vs 外采、MCP 协议与 Headspace 的隐私可观测性实践 在多 Agent 基础设施的选型策略上，Nitin Kumar 与 Alina Rivilis 指出，这取决于团队技能栈、上线交付速度以及 Agent 的可复用潜力。如果只是特定概念验证（PoC）且后续可能废弃，外采商业现成方案（COTS）更为高效；如果具备通用价值，自建能积累长期资产。 同时，嘉宾们提到了生态体系的演进： - **MCP（Model Context Protocol）**：业界正广泛拥抱 MCP 与现成的工具生态仓库，结合 LangChain、LangGraph 内置的标准工具，开发者无需从零手写底层集成代码。 - **多模型与 Agent 治理体系**：企业需要清晰掌握“哪个 Agent 在何时调用了哪个底层基础模型”， observability 治理层已成为行业刚需。 Headspace 在心理健康与医疗场景下的可观测性治理实践极其严格： - **全链路调用追踪**：采用 Galileo AI 作为企业级可观测性分析平台，对每一次 LLM 调用进行监控。 - **严格去标识化（De-identification）与私有化部署**：为防止敏感医疗健康数据（Healthcare Data）泄漏，可观测性栈全部部署在 Headspace 自己的私有云基础设施中，且在数据流出核心系统进入监控管道前，必须先经过一层严格的脱敏与去标识化过滤。 - **代码级灵活度**：在构建底层时保留代码层级的细粒度控制，确保开发人员能高度定制化 Agent 行为。 ### [33:45](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2025s) 低代码拖拽与代码级编排（LangGraph）的规模化对比 小组讨论延续了开发与部署智能体（Agent）所使用的技术栈。Akhil Chaturvedi 谈到了市面上曾经风靡、且当下依然常见的“拖拽点击式”（drag-and-drop）低代码 Agent 构建工具。他旗帜鲜明地表示，这类图形化搭建方式长期来看根本无法规模化（scale）：原因在于现在的 AI 编程工具（如 Cursor 或 Windsurf）已经极其高效，“对我来说，直接向 Cursor 解释如何构建某些逻辑，远比在画布上点按钮、画连接线轻松得多”。 像 LangGraph 这类代码优先的编排框架展现出了巨大的规模化优势。开发者只需通过自然对话告诉代码助手：“我已经有了这个 Agent，你能不能给它加一个新节点（node），既能做这件事，还能顺便生成一张猫的图片？”工具便能迅速将逻辑转化为结构化的代码流。更重要的是，LangGraph 等框架能够无缝过渡到生产级系统，团队只需要在其之上挂载可观测性（Observability）监控层即可直接上线。 ```mermaid flowchart TD accTitle: Low-Code Visual Builder vs Code-First Orchestration subgraph LowCode["低代码 / 拖拽式构建"] A1["画布拖拽节点"] --> B1["手动连线配置"] B1 --> C1["难以维护与规模化"] end subgraph CodeFirst["代码优先 + AI 辅助（LangGraph + Cursor）"] A2["自然语言指令<br/>'添加新节点'"] --> B2["Cursor / Windsurf<br/>自动生成代码"] B2 --> C2["LangGraph 状态图架构"] C2 --> D2["挂载可观测性层<br/>直接进入生产环境"] end ``` --- ### [34:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2077s) 受监管金融机构的技术选型、治理与成本挑战 Alina Rivilis 从银行等强监管金融机构的视角对此作出了回应。在高度合规的银行环境中，工程师受到各种法规的硬性限制，很多热门的外部工具、尖端开源框架都无法直接使用。由于必须选用经企业合规审批的工具（enterprise-approved tools），银行在构建 Agent 系统时往往面临更大的阻碍与挑战。 为了在安全的前提下落地智能体，银行必须设计非常严格的治理架构与管理层（administration layers）： - **全链路行为观测**：必须对 Agent 的输入、内部决策路径以及最终输出建立全方位的监控机制。 - **治理与审查开销**：严格的审核机制与监控层不可避免地推高了构建和运行 Agent 系统的总体成本，但为了满足合规性，这是无法妥协的硬性支出。 --- ### [36:01](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2161s) 本地部署（On-Prem）的包袱与云端混合合规新范式 针对企业安全与合规需求，Akhil 深入剖析了架构演进的痛点。许多大企业面对安全要求的第一反应通常是搞“本地私有化部署”（On-Prem）。然而，On-Prem 往往会给组织带来沉重的技术与管理包袱：企业必须雇佣大量专职运维人员、IT 主管，还要在内部各个组件之间手动配置极其复杂的 IAM 安全访问策略。更致命的是，On-Prem 会严重拖慢研发迭代速度——“如果说速度是你的盟友，本地化部署会彻底将你拖垮，你每天都在耗费精力排查某个底层 Bug 是哪来的、Sentry 告警到底是哪个系统报出来的”。 作为替代，如今行业正在迎来一股基于合规契约与云原生市场的新趋势： - **BAA 与 HIPAA 合规契约**：越来越多的 SaaS 供应商提供具备签署商业合作协议（BAA，含 HIPAA 合规）资质的合同，将数据治理的法律责任（liability）直接转嫁给具备资质的合规供应商。 - **云厂商 Marketplace 的无缝合规**：AWS 等云服务商的 Marketplace 应用通常运行在相同的 AWS 基础设施上，能够自动继承 BAA 资质。 - **“本地数据 + 云端算力”混合模式**：数据留在客户本地，而高成本的模型计算放在云端完成，既保住了核心数据隐私，又免去了自建基础设施的痛苦。 Alina 赞同了这一观察。她指出，虽然某些传统金融机构确实将大模型完全锁死在本地私有服务器中，但以她所在的银行为例，他们选择深度使用微软 Azure 及其 Azure OpenAI 服务。在架构上，银行会在 API 接口层挂载密集的安全监控与拦截层；在涉及对外分发、外部交互的内容生成环节，必须强制引入人工介入（Human-in-the-loop）机制——尽管 Agent 之间的内部通信无法做到实时人工核验，但面向公众的输出必须经过把关。 --- ### [38:18](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2298s) 智能体伦理：数据隐私边界与 AI 心理治疗师的应用困境 讨论随后转向了构建智能体系统的伦理与隐私考量。Alina 提出，随着技术飞速演进，业界如今可以构建各类极具敏感性的系统——从用于合规审计、客户跟进、邮件撰写的 Agent，再到甚至能提供心理咨询的“AI 治疗师”（AI therapist）。这引发了关于数据安全与应用边界的深度思考。 Nitin Kumar 从软件系统与架构落地层面提出了伦理治理的核心准则： - **多维度问责制（Accountability）**：必须从架构设计之初就清晰界定系统的责任边界。 - **PII（个人身份信息）双向清洗**：必须建立严格的数据管道，在 Prompt 进入 LLM 之前剥离所有个人身份信息，同时对大模型输出的内容再次执行 PII 过滤与脱敏。 - **最小权限访问控制**：Agent 调用的资源和工具必须受到严格限制，绝不能授予其全局数据的无差别读取权限。 - **全流程透明度与系统文档化**：详细记录并向合规部门明确 Agent 接收什么格式的输入、产生何种输出，以及其背后的评估准则与核心指标。 随后，Alina 追问了一个更本质的社会学难题：“技术上我们确实可以构建一个既安全又合规的 AI 心理治疗师，但做这件事本身，在道德上究竟合不合伦理？我们当中到底有多少人真的愿意向一个 AI 敞开心扉？” Akhil Chaturvedi 结合自己在 Headspace 的实际工作经历对此作了坦诚的回应。他承认自己曾长期在这个问题上感到纠结与挣扎：引入 AI 辅助治疗是否会冲击现有专业治疗师的饭碗与整个职业生态？ 然而，当看到残酷的现实供需鸿沟时，看问题的视角就会发生转变： - **极度匮乏的服务覆盖率**：需要心理健康支持的人群中，甚至连 2% 到 3% 都无法获得专业帮助，美国本土如此，发展中国家更为严峻。Akhil 以老家印度为例：印度拥有 14 亿人口，但大众几乎完全享受不到心理治疗，并不是因为大家没有心理压力或不需要帮助，而是根本没有足够的专业治疗师供给。 - **昂贵的时间与金钱成本**：在北美，面对面的专业咨询通常两周才能约上一次，单次咨询对个人或保险公司而言费用高达 250 美元，“花 250 美元我都足够去做好多次专业按摩了”。 - **即时陪伴的新范式**：如果一个人能在深夜入睡前、登台演讲感到恐慌前、或者当女儿面临重大升学考试感到焦虑时，随时随地接入一个具有共情支持能力的助手，这种价值是传统低频面诊无法覆盖的。 Akhil 强调，当你不把 AI 看作“替代人类治疗师的工具”，而是将其视为一种提供持续可及性支持的“全新范式”（new paradigm）时，它的伦理正当性就变得清晰起来。他坦承自己是一个“技术加速主义者”（accelerationist），深信 AI 是推动社会向善的技术力量。 --- ### [43:11](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2591s) 最佳实践建议：规则分解、红蓝对抗与从简起步 在圆桌讨论接近尾声之际，专家们针对构建和部署自主 AI Agent 总结了最关键的实操经验与避坑建议： Yizheng Wang 博士提出了三点工程与安全守则： 1. **角色分解以降低策略熵（Policy Entropy）**：在系统架构上必须对智能体的职责进行精细拆分，收敛其行动空间，防止 Agent 在复杂决策中失控偏航（go rogue）。 2. **上线必经红蓝对抗（Red Teaming）**：每次模型或 Agent 部署上线前，必须进行严格的对抗性攻防测试，挖掘越狱与潜在恶意行为漏洞。 3. **警惕“氛围式编程”（Vibe Coding）渗透生产系统**：当下凭借直觉和 AI 随性写代码的“氛围编程”（Vibe Coding）在个人娱乐项目中很酷，但在严肃的企业级场景下绝对不能搞“氛围式安全治理”（Vibe Security Coding），哪怕出现微小的安全疏漏都会酿成灾难性后果。 Nitin Kumar 则给出了针对工程落地的实战路线图： - **严禁好大喜功，由浅入深**：“如果你刚刚踏上 Agent 之旅，请放慢脚步。不要试图一上来就用 Agent 解决世界饥饿级别的宏大问题。” - **先串行，后多智能体**：初期建议从单 Agent 或双 Agent 架构起步，优先采用执行步骤明确、变数可控的“串行流水线模式”（sequential pattern）。 - **稳步验证控制力**：尽管“一个系统指挥协调 10 个不同 Agent 互相通信协作”在概念上听起来极其酷炫，但多智能体交互的管理难度与系统发散度呈指数级上升；唯有在串行模式下积累了足够的置信度与控制经验，才适宜推进复杂的多 Agent 编排方案。 ### [44:36](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2676s) Agent 落地的三维权衡与从快速原型到严谨生产的演进 在多 Agent 协同与系统架构的落地讨论中，嘉宾指出了理论构想与现实落地之间的巨大落差。多 Agent 之间的通信机制在设计蓝图上听起来非常理想，但在实际运维中往往极难维护。面对复杂的落地场景，团队必须在三个核心维度上进行周密权衡：速度（Speed）、准确率（Accuracy）与安全性（Safety）。在设计或评估任何 Agent 架构时，团队都需要清晰地审视这三个维度，明确 Agent 究竟在哪些环节带来了真正的增益，而不是盲目引入复杂的系统链路。 针对开发模式的演进，嘉宾结合时下前沿的开发方式指出，像 Vibe Coding（基于大模型的氛围感编程/快速提示开发，录音中被转写为 VIP coding）这类模式在探索初期极具价值。当团队刚开始接触 Agent 或构思全新应用的原型验证（PoC）时，依靠这种快速编码方式能够以极高的效率将脑海中的构想迅速具象化。然而，一旦项目步入生产级别（Production-level）的系统构建阶段，开发策略就必须发生根本转变：工程师必须回归深思熟虑、节奏严谨且具备完备单元测试（Unit Testing）的代码工程规范。嘉宾一致强调，Agent 项目务必秉持“从小做起、验证先行”的原则，先通过小范围的 PoC 验证价值，只有在真正找到坚实的应用场景后，再逐步推进规模化部署。 ```mermaid flowchart TD accTitle: Agent 开发与落地的两阶段演进路径 subgraph Prototype["原型与概念验证阶段 (PoC)"] A["构思与创意"] --> B["快速编码与探索 (Vibe Coding)"] B --> C["快速验证可行性"] end subgraph Production["生产部署与工程化阶段"] D["三维权衡审查: 速度 / 准确率 / 安全性"] --> E["严谨的单元测试与工程构建"] E --> F["对齐真实场景，稳步规模化"] end C -->|"明确真实场景价值"| D ``` - **三维评估标准**：在审视 Agent 的实际成效时，必须始终平衡速度（Speed）、准确度（Accuracy）与系统安全性（Safety）。 - **开发范式分离**：初期原型构建可依靠大模型辅助快速出活，但对于线上生产系统，必须采取严谨、慢节奏、高覆盖率的单元测试工程方法。 - **规模化原则**：切忌盲目扩建系统架构，唯有在确凿的业务用例中验证成效后，才应推进规模化扩展。 --- ### [45:44](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2744s) 拒绝技术狂热：技术教育、商业对齐与高频业务闭环 在对谈的总结阶段，嘉宾给出了针对企业与开发者的关键建议：重视认知与教育的投资，保持对底层系统的深刻理解，并理性克制技术冲动。嘉宾特别强调了“能做不代表该做”（"Just because you can do something doesn't always mean you should do something"）的黄金法则。拥有大语言模型（LLM）的访问权限，绝不意味着现实中的所有问题都应该用 LLM 来解决。团队必须投入充分的时间进行前期调研，想清楚自己正在构建什么、为何而建，并让技术方案真正契合公司的整体业务战略。 真正能够为企业创造最高生产力价值的 Agent，必然是那些直击核心业务痛点、且能深度嵌入现有业务流程的系统。以具体场景为例，如果企业当前正面临严峻的客户流失（Churn）问题，那么围绕用户留存搭建 Agent 解决方案就是极具价值的投入；但如果这并非当下痛点，就不要为了套用新技术而强行生搬硬套。企业应当将目光聚焦于那些真正契合自身模式的业务节点，尤其是业务流转频繁、规则繁复的高频交互工作流（High-frequency flows），在这些场景下，Agent 的自动化与推理能力才能实现投资回报率的最大化。随着这一务实观点的落地，研讨会行至尾声，四位嘉宾向全场观众致谢，整场关于自主 AI Agent 落地挑战与机遇的圆桌对话圆满结束。 - **克制技术盲从**：大模型不是包治百病的万能工具，切忌在缺乏商业逻辑的情况下为了用技术而做技术。 - **深度业务对齐**：Agent 的生产力上限取决于其解决核心业务痛点、融入日常业务流程的深度。 - **优选高频场景**：优先选择高频、高交互密度的核心工作流（如客户流失预警与挽留、高频服务流程）作为落地切入点。 ## Notable quotes > [03:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=217s) a swarm of computers that are doing stuff autonomously — Akhil Chaturvedi > [03:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=217s) 大约 90% 是软件工程，只有 10% 是 AI — Alina Rivilis > [08:37](https://www.youtube.com/watch?v=89UR-fjDA_I&t=517s) did something stupid — Yizheng Wang > [25:28](https://www.youtube.com/watch?v=89UR-fjDA_I&t=1528s) Agent 的本质是具备智能（Intelligence），而智能就意味着它一定会犯错。 — Yizheng Wang > [33:45](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2025s) 对我来说，直接向 Cursor 解释如何构建某些逻辑，远比在画布上点按钮、画连接线轻松得多 — Akhil Chaturvedi > [36:01](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2161s) 如果说速度是你的盟友，本地化部署会彻底将你拖垮，你每天都在耗费精力排查某个底层 Bug 是哪来的、Sentry 告警到底是哪个系统报出来的 — Akhil Chaturvedi > [38:18](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2298s) 花 250 美元我都足够去做好多次专业按摩了 — Akhil Chaturvedi > [43:11](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2591s) 如果你刚刚踏上 Agent 之旅，请放慢脚步。不要试图一上来就用 Agent 解决世界饥饿级别的宏大问题。 — Nitin Kumar > [45:44](https://www.youtube.com/watch?v=89UR-fjDA_I&t=2744s) Just because you can do something doesn't always mean you should do something — Nitin Kumar ## Mentioned Akhil Chaturvedi, Alina Rivilis, Yizheng Wang, Nitin Kumar, Headspace, Straiker, Marriott International, Home Trust Company, AI Future Leaders, ChatGPT, Claude, Anthropic, OpenAI, DistilBERT, GPT-4, Amazon Bedrock, Baseten, Groq, Cursor, Windsurf, LangChain, LangGraph, CrewAI, Galileo AI, Model Context Protocol (MCP), Microsoft Azure, Azure OpenAI, Sentry ## Video description Akhil Chaturvedi, Staff Machine Learning Engineer, Headspace Alina Rivilis, Director, Data Science, Home Trust Company Yizheng Wang PhD, Head of AI Research Engineer, Straiker Nitin Kumar, Director Data Science, Marriott International Abstracts This session will dive into the technical complexities of building and deploying autonomous AI agents capable of performing complex tasks and making decisions in real-world environments. Panelists will discuss the underlying architectures, frameworks, and algorithms that power AI agents, as well as the challenges in scaling these systems for broader use. Topics will include multi-agent coordination, reasoning, and decision-making under uncertainty, as well as the ethical considerations of deploying autonomous agents. Learn from technical experts who are at the forefront of developing AI agents and discover the opportunities and future directions for this cutting-edge technology. [Source](https://www.youtube.com/watch?v=89UR-fjDA_I)

[3:37] Akhil Chaturvedi: a swarm of computers that are doing stuff autonomously [3:37] Alina Rivilis: 大约 90% 是软件工程，只有 10% 是 AI [8:37] Yizheng Wang: did something stupid [25:28] Yizheng Wang: Agent 的本质是具备智能（Intelligence），而智能就意味着它一定会犯错。 [33:45] Akhil Chaturvedi: 对我来说，直接向 Cursor 解释如何构建某些逻辑，远比在画布上点按钮、画连接线轻松得多 [36:01] Akhil Chaturvedi: 如果说速度是你的盟友，本地化部署会彻底将你拖垮，你每天都在耗费精力排查某个底层 Bug 是哪来的、Sentry 告警到底是哪个系统报出来的 [38:18] Akhil Chaturvedi: 花 250 美元我都足够去做好多次专业按摩了 [43:11] Nitin Kumar: 如果你刚刚踏上 Agent 之旅，请放慢脚步。不要试图一上来就用 Agent 解决世界饥饿级别的宏大问题。 [45:44] Nitin Kumar: Just because you can do something doesn't always mean you should do something

[3:35] Akhil Chaturvedi: an operating system with an LLM or multiple LLMs at the core. [5:59] Dr. Yizheng Wang: agent is such a buzzword, especially in Silicon Valley [18:57] one agent that's preparing the dish, another one garnishing it [20:46] Hey, great question [40:17] Akhil Chaturvedi: a lot of massages. [40:17] Akhil Chaturvedi: new paradigm [40:17] Alina Rivilis: accelerationist [42:54] Dr. Yizheng Wang: policy entropy [42:54] Dr. Yizheng Wang: vibe security coding [42:54] Nitin Kumar: go slow [42:54] Nitin Kumar: solve the world hunger problem [42:54] Alina Rivilis: just because you can do something doesn't always mean you should.

[0:04] Uh welcome everyone. Today we are going [0:06] to be talking about AI agents. And some [0:10] of you are excited, right? Some of you [0:12] may have AI agents in production. Uh so [0:17] we are going to start by introducing the [0:19] panel. Uh we're going to start at that [0:22] end perhaps [0:24] with uh Akil. [0:26] >> Yeah. Hi. Uh I'm Akil. I'm I'm a tech [0:29] lead, principal engineer at Headspace. [0:31] Uh I essentially lead their team that's [0:34] building an AI therapist. So everything [0:37] breaking down from the science of [0:39] psychology, how to put that into LLMs, [0:42] doing post training, but also work [0:44] through memory of different sorts of uh [0:48] information that you know about patients [0:50] and deliver that as an end-to-end [0:52] therapeutic experience. So I work for [0:55] that team. It's a mix of research and [0:56] production level systems. [0:59] >> Nice. Uh Yen. [1:02] >> Yes. Hello everyone. My name is Ejong [1:04] Wong and uh I'm a Stanford PhD and my [1:08] research lies in the autonomous decision [1:10] systems and now I'm a head of AI uh in a [1:13] startup called Striker. Uh Striker is a [1:16] AI security company and uh uh we [1:19] essentially build a platform for [1:20] enterprise uh AI applications and [1:23] agents. So we cover pretty much anything [1:26] related to the AI security. Uh we have [1:29] auto automated [1:31] you know right teaming assessment and [1:33] also runtime guard rails [1:36] >> and we have Nitan. [1:38] >> Hi everyone, my name is Nathan and I'm [1:40] based out of Dallas. Uh I work as a [1:43] director of data science at Marriott [1:45] International which uh most of you must [1:48] have stayed on. uh and as part of the [1:51] workwise I uh lead the data science and [1:55] jai um department there where we are [1:58] trying to see how we can actually use [2:01] the traditional machine learning as well [2:03] as the large language models and agentic [2:06] solutions to solve the business problems [2:08] and it can span across like legal team [2:11] sales team customer care so it's across [2:14] the board our team uh supports that [2:16] >> so um welcome and my name is Alen Ina, I [2:20] am here from Toronto, Canada. I am the [2:24] director of data science and AI at a [2:26] bank, First Home Bank of Canada. Um, [2:29] where I lead a team of ML engineers, [2:32] data scientists, and we build genai [2:35] applications. I'm also the CEO and [2:38] founder of an AI education company [2:39] called AI future leaders where we focus [2:42] on educating in AI. So this panel this [2:47] panel is about AI agents something that [2:50] uh we are um building experimenting with [2:54] deploying and just a show of hands how [2:57] many of you have got AI agentic [2:59] applications or AI agents in production [3:02] potentially [3:04] quite a bit and many of you are maybe [3:06] aware of all of the challenges that are [3:10] um presented when you're building AI [3:13] agents and applications. So my first [3:15] question to the panel is about [3:18] architectures the architecture of these [3:21] AI agents and um before me maybe we uh [3:25] how many of you maybe are aware of uh [3:28] what is an AI agent or when we say [3:30] multi- aent AI frameworks and [3:32] architectures so a couple of you so [3:35] maybe we should start by going there uh [3:37] what are AI agents at a high level um if [3:42] one of you would like to kick it up [3:43] >> I'd kick it So um [3:47] very simply an AI agent is it's a it's [3:52] an operating system. It's an operating [3:54] system with LLMs an LLM or multiple LLMs [3:58] at the core of it. And this system can [4:01] do stuff. What it can what does that [4:04] mean? It could use tools. It could use [4:06] APIs. It could have skills that you [4:10] might do as a human. It could p pull [4:13] information from memory, from databases, [4:15] run asynchronous tasks to enrich its [4:18] knowledge, but also communicate with [4:20] someone in a way where um the [4:24] information that you get uh is way [4:26] better than you might from a single or [4:29] even a combination of foundation models. [4:32] A great example is if you've used chat [4:34] GPD deep research, there are agents [4:36] behind the scenes that are making [4:38] decisions figuring out what to do next, [4:40] what should think next, what should they [4:42] think next, what should the output be, [4:45] and finally create a very well [4:47] structured report. This is like agents [4:49] in action. So it's it's think of it like [4:51] basically a swarm of computers that are [4:55] doing stuff autonomously. [4:57] >> So you would say 90% ballpark it's [5:00] software. you can think of it and 10% [5:03] maybe AI. So if you might think about it [5:06] well a software system with maybe [5:09] instructions maybe it has got some sort [5:11] of uh uh behavior or um they're often [5:15] task oriented and they will have access [5:18] to LLMs they will also have access maybe [5:21] to a memory maybe to some instructions [5:23] maybe access to databases APIs and they [5:27] might be speaking to other AI agents [5:29] that are maybe specialized. You might [5:31] have one AI agent to write an email, [5:33] another one to validate an email, and [5:36] the third agent might kick up to [5:38] potentially translate it to another [5:40] language. So these frameworks are [5:43] complex. So let's talk about the [5:45] architecture. If uh uh there is one [5:48] approach that you found maybe useful or [5:51] more interesting and which uh [5:53] architectures seem to to to work better [5:57] maybe than others. [5:59] >> I can start on that. So I think it kind [6:02] of depends on the use cases to use cases [6:04] basis like what kind of different um [6:07] algorithmic patterns that you have for [6:09] the agents. It could be a sequential one [6:12] where you have a pretty defined rule set [6:14] that okay uh take an example of [6:16] marketing content creation that okay you [6:19] you I I'll just give an example from a [6:21] married perspective that you have a [6:23] property you want to generate a website [6:25] content so you go and fetch the [6:28] necessary information using tools like [6:31] um uh he's mentioning about and you have [6:33] the data now you use the large language [6:35] model to actually generate it. So it's a [6:38] pretty defined pattern. So it's a [6:39] sequential one. Then you can have a [6:42] parallel one which like you know Akil [6:44] was talking about the research agent [6:45] where you say I want to do a research on [6:48] a specific topic. Multiple agents work [6:51] parallelly and then they can come to a [6:53] specific uh aggregator which kind of [6:56] consolidate all the information and be [6:58] able to uh provide the response. The [7:01] other could be about you know uh [7:03] self-critic like you know um like [7:06] someone actually one agent generate a [7:09] content and then the other agent will go [7:11] and evaluate whether that's right or not [7:13] or what kind of guardrails I have to put [7:15] on. So there it kind of depends on [7:17] specific use case uh and then what kind [7:21] of risk and reward you want to actually [7:23] go with to you know uh to pick that [7:26] specific pattern I would say. [7:29] >> Yeah. And uh there are many frameworks [7:31] out there for the agentic applications [7:33] like longchain, long graph, curi, you [7:37] know I can name so many. So uh what we [7:39] found the most effective way to build [7:41] agent in our work is that instead of [7:43] just leveraging on single framework, we [7:46] found like a combine multiple framework [7:49] into a single agent that we can leverage [7:52] all the tools and the pros from each [7:54] frameworks. That's number one. Number [7:57] two is that you know agent is such a [7:59] buzz word especially in Silicon Valley. [8:01] I feel like uh we don't need to build [8:04] agent for everything. You know agent is [8:06] very very costly sometimes and also hard [8:09] to predict. Uh that's another tip. Yeah. [8:13] And the third one I would say uh just uh [8:15] when you build the build AI agent make [8:17] sure you have this stateful loop uh in [8:20] there which means like you have control [8:23] on all the memories like a square pad [8:26] tool input output and then you can just [8:28] insert guard rails whatever then you can [8:30] have all the observability of the agent [8:33] and then you can see where went wrong. [8:35] Yeah. [8:37] So um you bring interesting points. Uh [8:41] how do you govern and regulate these [8:43] agents and how do you take these agents [8:47] from prototype to production and how do [8:50] you observe them? Because if you just [8:53] have one call to a large language model, [8:56] you can kind of put some monitoring [8:58] tools and you can get into some [9:00] observability. Now you might have five, [9:03] 10, even more agents playing together. [9:06] So how do we uh monitor and how do we [9:11] productionize these systems? [9:16] So you know one of the advantage like us [9:18] you know we are a security company and [9:20] one of our like strength is that we we [9:22] we are a we have the skill set to build [9:25] this trace native observability. So [9:28] essentially we can observe the entire [9:30] agentic workflow from prom to tool input [9:33] output from the you know reasoning [9:35] process. Uh and number one number two is [9:38] we have our in-house build like [9:40] playground and we can let the agent to [9:43] do this population based uh self play. [9:46] you know, we have multiple agents and [9:48] they just consistently battle to each [9:50] other and this is super critical data [9:52] set for us and we can see oh in which [9:55] cases you know the data the agent went [9:57] wrong and you know like did something [9:59] stupid. So so yeah that's very critical [10:03] uh for production to make sure you know [10:05] you have all the observability and have [10:08] you know know what agent can go wrong at [10:11] some point. It's interesting. Akila, I [10:13] wanted to ask you for a headsp space. [10:15] You are um working with agents right [10:18] now. Uh could you share maybe how you're [10:21] going about uh some of the um use cases [10:24] and applications and how you're making [10:26] sure that the agent is outputting the [10:30] right response and uh if you have got um [10:34] some guidelines for us. [10:37] >> Absolutely. So [10:39] um I'm going to be very careful about [10:43] the IP uh lines here but um to answer [10:48] your final question which is how do we [10:50] know that uh it's working well [10:52] essentially how the quality is good it's [10:55] uh it's we benchmarks so in psychology [11:00] uh and just you could think maybe some [11:02] of you have gone to therapy coaching uh [11:05] or even talked to CH GPT for your mental [11:08] health needs. You you can have a [11:11] conversation and based on the [11:13] conversation and by looking at it, you [11:16] could have certain signals that it's a [11:18] good conversation, right? The the [11:20] therapist build good rapport with you. [11:23] The therapist was able to understand [11:24] your problem better, hear you out. The [11:27] therapist was able to identify what are [11:29] the common causes or reasons in your [11:31] life. the therapist was able to mi make [11:34] you reflect on your own problems and go [11:36] deeper and deeper and then finally come [11:38] to some sort of solution and maybe even [11:41] schedule a new session. Right? So there [11:43] are a lot of different criterias that [11:46] have arisen from the success of this [11:48] conversation, right? Report generation, [11:50] problem solving, therapeutic alliance, [11:53] how they reacted to you resisting change [11:55] and so on. And the interesting thing is [11:57] because we understand this, you could [11:59] put this as a rubric and you could put a [12:02] man behind the rubric who judges every [12:04] single conversation and we call that man [12:07] anthropics cla. So what I'm trying to [12:09] say is that you could use LLMs as [12:11] judges. So what we essentially do is [12:13] like we have benchmarks based on a [12:16] mixture of expert LLMs that have judges [12:18] and every conversation that comes out of [12:21] a conversation with an AI therapist goes [12:24] through that benchmark and you get [12:25] scores just like you get scores for [12:27] software engineering evals math [12:29] reasoning benchmarks all you know the [12:31] kind of stuff that happens in the [12:33] foundation model layer. So that's how we [12:35] test it and but that kind of gives us a [12:38] very sort of good playground to work [12:41] with where we can make lot of tweaks to [12:42] our agentic architectures or different [12:44] models we might use or the use of [12:47] retrieval or memory or whatever but it [12:49] keeps coming back to that benchmark and [12:52] to make the benchmarks really fast what [12:54] we've actually ended up doing is we've [12:56] we've actually created a library of [12:58] thousands of synthetic mental health [13:00] patients. So it's almost like every time [13:02] you make a change to your AI therapist, [13:04] you have these thousand LLM patients [13:07] that chat with the AI therapist and [13:08] those conversations are benchmark. So [13:11] about 95% of the conversations are these [13:13] and 5% of them are real human [13:16] conversations that actual therapists [13:18] would be like you know a benchmarking [13:21] with the same rubric. [13:23] >> That is interesting. And um Newton um [13:26] you also have interesting applications. [13:28] >> Yes. But I think uh we may have to [13:31] understand that when we are talking [13:33] about productionizing uh any agentic [13:35] solution right so I'll probably go back [13:38] and you know understand or try to [13:41] explain that you have to see whether [13:43] that requires an agentic solution or not [13:45] because a lot of time what happens is [13:47] that a simple straight flow u workflow [13:50] or an RPA can actually solve that [13:52] problem. Now once you have identified [13:54] that yes this requires an agentic [13:55] because of you know uh kind of the [13:59] variability this solution needs. So you [14:02] kind of go with that. Now my [14:04] recommendation would be always go from [14:07] slow and then fast and then from you [14:09] know start with maybe like a one or two [14:12] agents and see how it works and then you [14:15] know then actually grow because once you [14:18] build the agent right you have so many [14:20] different aspects to it like you know [14:22] what about the context what about the [14:24] security guardrails what about the [14:26] ethical things right so all of those [14:28] things kind of are very uh strict and [14:31] and kind of depends on the the risk and [14:33] reward you kind of wants to have that in [14:36] that use case. So you you have to look [14:38] from a multiple uh dimensions I would [14:41] say and then based on that you kind of [14:43] say okay if things are good then only [14:45] you kind of uh think about [14:46] productionizing it otherwise go back and [14:48] iterate because building an agentic [14:51] solution it's it's more of a iterative [14:53] process than okay rather than just okay [14:56] I've built it and just deploy it and let [14:58] the the customers find it whether it's [15:00] good or bad. So it's kind of very [15:02] important on that. [15:03] >> Um also because you might have multiple [15:06] agents uh working there in your solution [15:10] um the costs the cost actually tend to [15:12] go up. You do need to observe uh what [15:15] they are doing but you also need to [15:17] monitor those cost because if you have [15:20] got now 10 agents each with access to [15:22] LLMs and they might be consuming very [15:26] expensive LLMs [15:28] um how do you balance it out? How do you [15:31] make sure that you're optimizing it? No, [15:34] >> I feel like uh cost is always a thing [15:37] you know like uh and from engineering [15:39] perspective I think cost is just going [15:41] down with you know the the AI [15:43] development uh that's why we need a we [15:47] really really need a uh the trace that [15:50] you know exactly how many input token [15:52] how many output token uh along the [15:55] entire agentic applications and you know [15:58] uh um otherwise it just sometimes just [16:01] can give you a lot of surprises this [16:05] >> and I think I just wanted to add on that [16:06] and I think when you're actually [16:08] building the agentic solution right you [16:10] may follow uh some sort of a [16:12] hierarchical pattern as well that if the [16:15] problems are very straightforward which [16:17] can be actually solved through a simple [16:19] uh you know LLM which is less expensive [16:22] probably go with that approach and then [16:25] where you know the decisioning is very [16:27] uh complex go for a you know bigger [16:31] model of LLM as as well. So that way you [16:33] kind of be able to optimize the cost [16:36] rather than for everything you just go [16:38] with a very big uh LLM as well because [16:40] the more bigger model uh probably you [16:43] use the the inference cost the inference [16:46] time all of those will actually go up as [16:48] well. So [16:49] >> yeah, as well as latency. This is [16:50] something that we just discussed. Uh uh [16:53] how do you control the the latency that [16:56] now you're introducing because if you're [16:58] doing multiple calls and you do need to [17:01] make sure for example in your example of [17:04] an AI therapy assistant, you want to [17:07] make sure that the AI therapist gives [17:09] you sound uh advice, right? So you might [17:12] put one agent on top of another one to [17:14] validate it and Q8 but that introduces [17:17] latency problem and how do you balance [17:20] it out with that customer experience [17:22] now? Um what kind of things have you [17:25] been trying? [17:26] >> Yeah, I can uh shed some light. So I [17:30] think latency along with great retrieval [17:33] and memory are like the three most [17:35] important problems at least in the agent [17:37] space. cost is important but every [17:40] quarter like cost halves so not really [17:43] optimizing for that uh I can tell you [17:46] like what we've tried so for example we [17:49] had to build our own safety guardrail [17:51] system because the uh safety guard the [17:54] custom safety guardrail systems that you [17:56] find on um bedrock guardrails which is [18:00] anthropic or open it just doesn't [18:02] suffice and so we had to essentially [18:04] have a separate model that does suicidal [18:06] ideiation [18:08] eating disorders, substance abuse, [18:09] domestic abuse, abuse of and other kinds [18:12] of things. And what we realized was that [18:15] um what we had was a big data set and [18:18] you could actually tag certain messages [18:20] using larger models with those things. [18:23] But then we what we did was we distilled [18:25] all of that information into a model [18:27] that's a thousand times smaller, let's [18:29] say, than GPD4, right? We used this [18:32] model called dist bird came out in 2020. [18:35] really popular dist from that time and [18:37] it brought our safety risk latency down [18:40] from like 900 milliseconds coming from [18:42] an LLM to 60 milliseconds. So it's it's [18:46] it's it's a way of fine-tuning but [18:48] essentially we distilled information in [18:50] one single domain which is great for [18:52] agents because in agents you have few [18:54] LLMs that just do one task and you kind [18:57] of distill them down into your custom [18:59] model. So that's one way we've done it. [19:01] Second way is we've started to this is [19:04] more of a application way but we've [19:06] started to like work with vendors that [19:08] just have super fast inference. You've [19:10] got base 10 you've got gro and uh they [19:13] they're essentially doing optimizations [19:15] at the GPU or at the kernel level or at [19:18] the page detention uh level and [19:21] quantization level. So we're spending [19:23] time making that those kind of workflows [19:26] faster. [19:27] >> Yeah. Being uh specialized. So, it's [19:29] almost like you have got one agent [19:32] that's uh preparing the dish, another [19:34] one garnishing it. They don't always [19:36] have to go after the most expensive [19:38] LLMs. [19:39] >> Yeah. [19:40] >> But you know what's interesting? Another [19:41] concern for us is the the data and the [19:45] privacy, the PII and the sensitive data. [19:48] Because if you're leveraging now [19:50] multiple models, you now start to look [19:53] at how your data is being used by let's [19:56] say you're sending your data to entropic [19:59] uh or GPT4 model how your data is being [20:03] used over there and if you are [20:06] leveraging multiple models from multiple [20:08] providers you're you're now dealing with [20:11] a very complex system. So managing that [20:14] complexity and the safety is is pretty [20:18] uh difficult. So [20:20] >> I think especially you need a you need [20:22] have very strict uh schema control for [20:26] like a tool input and output right for [20:28] the input you need to say oh whether [20:29] there's a direct prom injection [20:31] jailbreaking indirect prom injection [20:33] data expiltration output a PII harmful [20:37] suicide you know information all kind of [20:40] stuff then we can you know make sure the [20:42] agent is uh doing what we expected [20:46] >> and I would like to add on top of that [20:49] is that when you're talking about this [20:52] um you know ethical things right so it's [20:55] not only like the LLM agentic solution [20:59] is exciting but you know without the [21:01] proper guardrails it's very risky right [21:04] so right from the get- go you have to [21:06] think in a way that you'll be able to [21:08] build a system which have proper checks [21:10] and balances and when you're building it [21:13] you may have to look from multiple [21:16] angles like you know accountability [21:18] perspective who is going be responsible [21:20] when an agent make a mistake, right? So, [21:22] uh maybe uh from a transparency [21:25] perspective that how transparent your [21:27] model like the reasoning the LM is [21:30] doing, are you capturing on that? Uh [21:33] what about the privacy, right? What [21:34] about the biasness of that? So all of [21:37] those things you have to think it [21:38] through to make sure right from the get- [21:41] go to make sure the agentic solution [21:43] you're building is uh up to the mark and [21:45] be able to you know uh be able to do the [21:48] right thing for your use case. [21:51] >> LMS have gotten better but they still [21:52] hallucinate and they can still make [21:55] stuff up. So if you've got uh one agent [21:58] hallucin like one call to an LLM with [22:00] let's say 1% um hallucination and then [22:03] you have another one and another one by [22:05] the time you have got 10 or even a [22:07] hundred agents you're potentially [22:10] introducing error you are instead of [22:12] improving things you're maybe [22:14] introducing more risk uh one approach I [22:16] have seen was uh you use one agent to [22:20] control another one or you've put more [22:23] LLMs essentially on top but that also [22:26] introduces uh more cost more latency and [22:30] uh the system becomes becomes very bulky [22:33] and complex um to manage. So uh how have [22:37] you been dealing with uh this type of [22:39] thing um managing these complex [22:43] architectures? Now you also need to have [22:45] that governance layer, that admin layer [22:47] of what goes in and what comes out. And [22:50] it's hard to tell now what happens in [22:52] there when the agents are having [22:54] essentially conversations with each [22:56] other. That's what pretty much they are [22:58] doing. They're sending calls. they might [23:00] also have got access to APIs and you [23:02] just mentioned uh uh jailbreaks and uh [23:05] prompt injections and uh the more [23:08] complex your system is the more guard [23:11] rails you need which is why AI security [23:15] uh comes into play. I think we all need [23:19] a better legal team [23:22] or good good lawyers in the house. But [23:25] but jokes apart, uh I think I I think [23:28] all of what you've said is right like [23:30] these are important problems. They [23:33] happen all the time. Um I've seen that [23:36] as as you were talking about the critic [23:38] LLMs like you have an actor critic [23:40] layer. Critic LMS help in that but that [23:42] slows things down a lot. So latency [23:45] becomes a problem. So [23:48] I feel there's almost like a trade-off [23:50] that you're constantly making that for [23:52] this use case I'm willing to like [23:53] sacrifice quality or but this use case [23:56] I'm able to sacrifice latency and and [24:00] hence go for higher quality by better [24:02] checking and so on. So I think like you [24:05] kind of make these decisions um for like [24:08] things that require extremely high [24:10] quality. One thing that I think about is [24:12] like does this have to be real time? Can [24:15] this be an agent take task that can [24:17] happen while the user is sleeping and [24:20] then the user just sees it later on [24:22] tomorrow? Does this have to be [24:23] instantaneous? And that way you can like [24:26] run a almost like an agentic workflow [24:28] for 10 15 minutes and have beautiful [24:31] quality but you don't have to worry [24:32] about latency. But sometimes latency [24:35] becomes important like voice. Voice is [24:37] something that's exploding right now. [24:39] But the issue with voice is that people [24:41] don't want to wait. Like it needs to [24:43] feel human. So a lot of times their [24:46] quality is a little bit sacrificed or [24:48] what you can do with voice something [24:50] called speculative decoding is that the [24:52] first few tokens that you get through [24:54] voice can be fast but not that [24:57] inaccurate and typically the first few [24:59] tokens don't contain that much lexical [25:01] information anyway. It's just like, hey, [25:03] great question. [25:04] >> But then you kind of buy some time and [25:06] in that time that you've bought, you can [25:08] have other agents kind of checking and [25:10] validating the answer of the next [25:12] >> line that you're going to say or write. [25:14] So a lot of times you can also do that [25:16] like I know cursor uses this for code [25:19] generation sometimes that your first [25:21] response is fast but then your further [25:23] tokens are slower. So [25:26] >> that's interesting. I wanted to ask uh [25:28] Yuching here about uh how you guys are [25:31] dealing with as you know the security [25:34] part of uh managing all these uh you [25:37] know agentic applications and [25:39] >> yeah absolutely so that's why first we [25:42] build a uh you know native trace base [25:44] observability we the tool and we can see [25:47] everything going on there and second in [25:50] house we also build agents so what we [25:52] found important is that we need to give [25:54] very clear goal for each agent you know [25:57] one agent cannot do anything. Uh we [26:00] should do some like a row decomposition. [26:02] For example, we have a supervisor agent, [26:04] we have verifier, we have retriever, you [26:06] know, just make sure you know they are [26:08] not overlap to each other, you know, and [26:10] also once you have this big picture, [26:13] then you can optimize the framework and [26:15] we should minimize the number of agent, [26:17] you know, do not just compound this like [26:20] uh system uh and also do right teaming [26:24] before you release to production or you [26:27] know to anyone even internal use make [26:30] sure because agent What's the [26:32] definition? Agent is the intelligent. So [26:34] intelligent means make mistakes, right? [26:37] So uh sometimes the agent just do things [26:40] that out our expect our expectation. So [26:43] we really need to capture all the [26:44] scenarios before we put it in the world [26:47] like the world. Yeah. Maybe uh it's [26:50] important to kind of expand on what is [26:52] red teaming because I think [26:54] >> oh the red teaming so you know red [26:57] teaming red teaming aentic is not a like [27:00] a soft like traditional software like we [27:03] do at pentest or whatever it's more [27:05] complicated than that you know you need [27:08] a from a cyber security perspective to [27:10] do it you know from engineering [27:11] perspective to do it and also like AI uh [27:14] you know like whether there is indirect [27:16] prom injection for example If a like say [27:19] a malicious user send you email okay and [27:22] then you ask AI to hey can you summarize [27:25] the email you know from two days and [27:27] then suddenly the model saw that [27:29] malicious email and you know send your [27:32] personal information to the hiker to [27:34] whatever. So uh the yeah the right [27:37] teaming there is not a onesize fit all [27:40] right teaming. So uh that's why we have [27:42] this like a uh a starter you know like a [27:46] bank of uh attack tax you know to test [27:48] the model and also we have this like a [27:51] you know one click like a uh attack [27:54] simulation uh simulation we uh that we [27:58] we bu based on our playground you know [28:01] like all the data we collected on the [28:03] customer we saw yeah the scenario and to [28:06] you know pentest and red teaming the [28:08] agentic applications. Yeah. [28:11] >> Interesting. Um yeah, we have got [28:14] interesting applications with um agents [28:17] as well. I I work for a bank. Uh we are [28:21] um seventh largest banking in Canada. So [28:23] some of our applications uh are around [28:27] uh document intelligence and processing. [28:30] We have touched a bit on voice [28:31] applications, text to speech. We talked [28:34] about generation of content, email. Some [28:36] of our applications are around uh [28:38] document intelligence and processing [28:41] things like um I think uh deposits or [28:44] operations or uh when you apply for a [28:47] mortgage a great deal of documents come [28:49] in and you might have an agent that [28:52] scans the document. It's just maybe [28:54] specialized at scanning the document [28:55] tagging extracting the information. We [28:58] might have another one to validate it. [29:00] That might also go next to another agent [29:03] uh that talks to a large language model [29:06] to figure out what exactly that the [29:08] first guy saw and you know did you see [29:11] what you saw? What did you do what you [29:13] needs to be done and then another one [29:15] maybe picks it up and integrates it into [29:18] our CRM system. Another one might kick [29:21] up to say, you know what, something is [29:23] missing. Some data is missing or [29:25] something is inconsistent. [29:27] And we might even have another one that [29:29] translates it into French. Um, right. [29:33] And we are trying to build these agents [29:35] in a way that they are reusable. We are [29:37] trying to build them in a way that they [29:39] are not very specific to that particular [29:42] business use case, but could be applied [29:44] like that uh translation agent. We are [29:47] building it in a way that it could be [29:49] reused in other areas. So we are [29:51] optimizing it for sure and we have [29:54] another one to make sure the first guy [29:56] does what they are supposed to do. But [29:58] it it does become big and bulky. So we [30:01] need that governance system and right [30:03] now on the market you don't have that [30:05] many tools out there to govern these um [30:09] because this is relatively new where you [30:12] will have uh multiple agents and these [30:15] frameworks and you end up mostly [30:18] building I don't know uh are you more [30:21] building your own or buying certain [30:23] tools or how are how are you approaching [30:26] this this build versus buy in this [30:28] agenting base. [30:31] >> Um, so I think it kind of boils down to [30:35] the to the skill set that we have uh to [30:38] the speed like by which you know we want [30:41] to have this in production and then we [30:44] also have to think in a way that [30:48] the agent that we are going to be [30:49] building is it something which we can [30:51] actually repurpose for different areas [30:54] as well or not. Right? If it is [30:56] something which is very very uh specific [30:59] and maybe you know after a proof of [31:00] concept we may want to throw it away [31:03] then I think it may be possible to [31:05] actually don't spend that much of time [31:07] and energy to actually build it and see [31:08] if there's already a commercially uh [31:11] offtheshelf available that you can use [31:13] and because the the vendor can actually [31:16] learn from you and be able to build a [31:18] product which can actually or build an [31:20] agent which can be used for other [31:22] customers as well. So it it kind of [31:24] depends on on on the use case and the [31:27] appetite you have [31:29] >> and also leverage MCP like stores they [31:34] you know so many companies and you know [31:36] individual contributors have put out [31:38] there and also c you know definitely we [31:41] need some custom uh customized uh tools [31:44] know internally but just and also those [31:47] uh like framework we talked about [31:48] earlier like a longchain long graph they [31:50] also have built-in tools and you know [31:52] things that we can leverage on. So all [31:54] those sort of things we can we [31:56] definitely don't need to build ground up [31:58] everything but there are certain pieces [32:00] we need. Yeah. [32:03] >> Yeah. I'm definitely seeing MCP and lots [32:05] of companies offering that and there's a [32:07] couple other frameworks out there to [32:10] manage and govern your LLMs essentially [32:12] and and your agents. Um and many [32:16] companies are actually going there [32:17] because they're recognizing that you [32:19] need to govern this and you need to have [32:21] that observability. You need to know uh [32:24] which agent is leveraging which model. [32:27] Uh when what how essentially you need [32:30] that observability layer as well what [32:32] they're doing. Uh what kind of things [32:34] have you been doing at Headspace with [32:36] managing your agents? I'm sure you've [32:38] got many out there [32:40] >> for observability. [32:42] um [32:44] run-of-the-mill uh agentic tracing. Uh [32:47] we have every LLM call observed in [32:52] uh an observability company. We use one [32:55] of the major ones and um I can name them [32:59] Galileo uh Galileo AI. They've been uh [33:02] great. Um we have we always have a layer [33:07] of deidentification that goes from our [33:10] stack to um any observability stack [33:14] which too in our case is actually [33:16] deployed in our own cloud infrastructure [33:18] because we don't want any you know [33:20] healthcare data to leave our cloud [33:23] that's a big risk but we still uh [33:25] anonymize it [33:27] um in terms of uh this [33:31] um otherwise we often times times for [33:33] building I feel like um you want to use [33:37] things at a sort of granularity where [33:40] people can still build agents using code [33:43] and make them very customizable. [33:45] uh there was this whole and it's still [33:47] probably going on there's this like [33:48] whole phase of like agents that you can [33:50] build build with like drag and drops and [33:52] clicks and so on but I actually feel [33:55] that's not going to scale because it's [33:57] so easy to use cursor or wind surf like [34:00] it's easier for me to actually explain [34:02] to cursor how to build something than to [34:04] actually use buttons and clicks and [34:07] lines to actually build it and so it [34:09] actually happens to be that things like [34:11] langraph scale really well because you [34:14] can actually just have a conversation [34:15] that I have this agent but can you add a [34:17] new node that does this but that can [34:20] create a picture of a cat as well and [34:22] it's easier to do that and but the good [34:23] part is that langraph and all they then [34:26] scale to production level systems so you [34:28] just have to like kind of attach [34:30] observability on top of it but um [34:35] yeah does that answer that question [34:37] >> yeah it does uh those are excellent [34:39] examples um they they are great uh for [34:42] someone who like myself uh I work uh at [34:45] a bank. We are financially well [34:47] basically we are regulated institution [34:50] and some of those tools and techniques [34:52] we just cannot use because we are under [34:54] certain regulations. So for us [34:56] leveraging agents and building aenting [34:59] systems becomes um uh something that's a [35:03] bit more challenging. we cannot leverage [35:05] some of the tools that you just [35:06] mentioned and we tend to go with more [35:09] you know enterprise approved tools um [35:13] some of those opensource uh we we cannot [35:15] explore just because of the fact that we [35:17] do happen to be a bank and uh that means [35:20] we have to be a bit more creative and we [35:23] also make sure that we observe [35:25] everything the way the agents are [35:27] behaving and we are balancing things out [35:30] and we put administration layer I layers [35:34] on top of it. Um so we we do have that [35:38] governance uh as to what goes in and [35:41] what really comes out and what is [35:43] happening in there. So we have but that [35:46] means uh we are increasing the the cost [35:49] often when we are doing these things and [35:52] we have to do it because uh we have to [35:55] monitor these systems and yeah um but [35:58] yes it's a lot of tools are out there. [36:01] Um [36:01] >> yeah I I do want to say like for [36:03] security as you might have to the [36:06] initial reaction is on prem like you [36:09] basically want [36:10] >> but [36:12] is this is an opinion like it depends on [36:14] companies to companies but I'm seeing [36:15] like [36:16] >> it puts a lot of burden on your own [36:18] organization to manage stuff you have to [36:20] hire people there needs to be an IT [36:22] manager you need to create like uh you [36:25] know um security between different [36:27] things that like IM security policies [36:30] and all of that and more and more often [36:33] now come a lot of these vendors are [36:35] coming in with BAAS which is HIPPA [36:37] compliant contracts which basically [36:40] transfers the liability of uh data [36:42] governance to them. So they need to be [36:44] really sure that of their offering and [36:48] uh I feel that's great but that's also [36:50] being orchestrated by [36:52] uh sort of cloud offerings like AWS. Now [36:55] AWS has marketplace and marketplace apps [36:58] on marketplace can automatically have a [37:00] BA because their infrastructure too is [37:03] on AWS. So there are sort of this new [37:06] wave of like either safe but SASbased [37:10] offering but also like hybrid offerings [37:13] like your data could be on your prem but [37:15] the compute can be on their prem. these [37:17] kind of things that are sort of working [37:19] well because on-prem slows you down like [37:22] if your velocity is your ally. If you do [37:25] something onrem, you'll just like be [37:26] figuring out where does this bug come [37:28] from? Where's the sentry alert coming [37:30] from? And you know that's [37:32] >> you're right. Uh some financial [37:33] institutions locked it down and are [37:36] putting things on prem. they they put [37:39] LLMs that they host and others like uh [37:42] in my critical situation we tend to use [37:44] Azure and we leverage the you know uh [37:48] Azure OpenAI HPT type models and uh we [37:53] also often put APIs and monitoring on [37:55] that and we also tend to put human in [37:58] the loop when possible. Um obviously you [38:00] cannot have a human in the loop in [38:02] between the agents but we do have a [38:04] human in the loop when we are um [38:07] communicating and creating some sort of [38:10] content especially something that might [38:12] go externally. So we definitely have [38:15] that. So for my um last question [38:18] actually uh let's talk about the ethics [38:20] side of it. the ethics of these systems [38:23] and um some of you may be in regulated [38:27] industries may some of you not so much [38:30] but uh how ethical to do these things [38:32] because now we can build anything from [38:35] AI therapists to AI systems for [38:39] compliance reporting following up with [38:41] customers we can have agents to write [38:44] emails um what kind of uh what what have [38:48] you seen or what have you been um you [38:51] what approaches around the ethics and [38:53] the privacy with all the the data. [38:58] So I think from a ethical perspective [39:00] it's it's definitely one of the key [39:03] areas that you should always think when [39:05] you are talking about any any software I [39:07] would say right and I was like [39:09] mentioning earlier right you may have to [39:11] think multi-dimensional for example [39:14] accountability perspective uh PI [39:17] information because a lot of the data [39:20] that you have may not may contain [39:22] actually the PI information so you have [39:24] to make sure that the input which is [39:27] going to the LLM and the output that's [39:29] coming out from the LLM does not have [39:32] that. So you may have to process in a [39:33] way that it kind of removes those PII [39:36] information. You can also ensure that [39:40] you know the resources the agent have [39:42] access to is restricted. It should not [39:44] be able to have be able to access [39:47] everything. So it should be very very [39:49] restricted as well. And then also from a [39:52] documentation perspective you should be [39:54] able to talk about that okay what's your [39:56] agent input is what the output looks [39:59] like what are the you know uh key [40:01] criterias that your you know agent is [40:04] responsible for so all of those things I [40:06] think it kind of help on the [40:08] transparency as well so all of those [40:10] things you you have to take care and and [40:12] think about when you're trying to build [40:14] the agentic solution [40:17] >> and by ethics do you mean like uh in [40:19] terms of security of data but or also [40:22] about you mentioned AI therapist also [40:24] about like the [40:26] >> applications [40:26] >> yeah I was thinking uh more generally [40:28] the applications um I've struggled with [40:30] >> well how I was since since you came on [40:32] stage and since we we met I've been [40:35] thinking how ethical it is how many of [40:37] us are actually interested in speaking [40:39] with an AI therapist uh we we can [40:43] definitely build one it's going to be [40:45] nice and safe and secure but uh now how [40:48] how ethical [40:50] um to do such a thing. [40:51] >> Yeah. I I mean I have I have struggled [40:53] with this problem a lot because uh [40:55] coming from a company that already had [40:57] therapist, you know, there there is that [40:59] sort of hesitation that how does this [41:03] affect the the field the people's [41:06] careers and so on. But you you kind of [41:09] realize once you see the scale of the [41:11] problem and you'll see larger problems [41:13] of that size that like not even like [41:16] 2 3% of people who need this kind of [41:20] support are actually getting it let [41:22] alone in the United States away from the [41:25] United like I'm from India like nobody [41:27] has therapy it's not because they don't [41:29] want it it's because there are no [41:30] therapists right you've got population [41:32] of 1.4 4 billion people, they're still [41:35] living lives. They still have problems, [41:36] but it's just that it's not a thing [41:38] because it's not a thing. And I think in [41:41] that sense, when you start to think [41:42] about that as as almost like a day in [41:45] and day out piece, you start to see [41:47] that, oh, there's value in this. I can [41:49] only meet my therapist once every two [41:52] weeks. And it also costs me or the [41:54] insurance companies $250 per session. [41:57] Now, I can go for a lot of massages for [42:00] $250. Right? So you start to think of [42:02] other kind of things and but you're like [42:03] if I could have access to a therapist [42:05] before I'm sleeping before I'm going on [42:07] a talk before I'm doing something else [42:10] before my daughter has an important [42:12] school exam and starts to build that [42:14] connection. I think that's when you [42:16] start to see like maybe you don't think [42:18] of it as replacing someone you start to [42:21] think of these applications as a new [42:22] paradigm. And when you start to think of [42:24] these applications of new paradigm, then [42:26] you sort of might consider the ethics [42:29] more ethical. [42:31] >> I'm I'm an accelerationist. I I want to [42:34] put it out there. I think AI is a force [42:36] for good. But [42:38] >> um [42:38] >> for sure if you're doing it safely, um [42:41] there's definitely value in an AI [42:43] assistant for uh you know, for therapy. [42:46] And I can definitely see how the [42:49] application would be very useful and [42:51] beneficial. And uh now that we are [42:54] wrapping up the panel, I'm going to ask [42:56] um everyone here to actually maybe share [43:00] some of those best practices uh for for [43:03] the audience around building and [43:05] deploying autonomous AI agents. Uh [43:11] >> yeah. So first thing just uh recap you [43:14] know like uh like do this row [43:17] decomposition to lower the policy [43:20] entropy. So make sure the agent do not [43:23] go rrook and second just do like a right [43:26] teaming every time you deploy it. And [43:29] third one you know a lot of people do v [43:31] coding these days and uh v coding is [43:34] cool that you do it personally but you [43:35] know company if you do a lot of web [43:37] coding uh there are going to be a huge [43:39] issues you know we cannot say we do a [43:41] vibe like security coding right like uh [43:45] the security if it happens there going [43:47] to be a big issue uh so uh yeah security [43:50] I think is a is a is a huge very very uh [43:54] pivotal u like aspect for us at this [43:57] stage yeah And all I can say is like so [44:00] if you have just started the agentic [44:02] journey uh I would say go slow. Don't [44:05] try to solve the world hunger problem [44:06] through the agentic solution. Maybe [44:08] start with a one or two agents and see [44:10] how it behaves. Uh and then from a [44:12] pattern perspective maybe go with a [44:14] sequential one where you have a pretty [44:16] defined steps with a little bit of uh [44:19] you know uh variability. try to solve [44:21] that and then once you have more [44:23] confidence to it then you kind of build [44:26] through multi- aent solutions because [44:29] yes it does sounds good that you know [44:30] you build an agent which kind of [44:32] interacts with 10 different agents but [44:35] managing those agents their [44:36] communication it it's a little [44:38] challenging as well. So it sounds good [44:40] on paper but when you are actually [44:42] practically it's very hard to kind of [44:44] maintain and then once you have those [44:48] always think in a way that you have a [44:52] like speed, accuracy and safety are kind [44:55] of the three dimensions you have to [44:57] think it through and see you know um how [44:59] it's like the how the agent is actually [45:02] going to be helpful in that. So [45:05] >> yeah I think great great answers. I'll [45:08] I'll just uh kind of maybe just [45:12] piggyback on the VIP coding thing where [45:15] I I feel like if you're starting with [45:17] agents or any application for [45:19] prototyping and proof of concepts, I [45:21] think VIP coding is great because you [45:23] can like imagine and build really fast. [45:26] But yes, for production level systems, [45:28] you want to be do very deliberate slow [45:31] unit testy coding. uh but but yeah just [45:34] second all of that they said start with [45:37] a small proof of concept start small and [45:39] then scale but only if you find the use [45:42] case for it. So [45:44] >> yeah and I would want I would like to [45:45] add to this um you would want to invest [45:49] in some education. I would say uh do [45:52] learn about these systems and just [45:55] because you can do something doesn't [45:57] always mean you should do something. If [46:00] you have got access to large language [46:02] models doesn't mean that uh you know [46:05] anything can be sold with a large [46:07] language model and uh do invest that [46:11] time to research to understand what [46:14] you're building why you're building and [46:16] align it to your uh business strategy. [46:19] Those agents that are going to be the [46:21] most productive for you are going to be [46:23] the ones that are solving key business [46:26] challenges and are aligned to your [46:28] business processes. So if you have a [46:30] churn problem, sure, go for it. But if [46:33] you don't have one, go solve something [46:35] that makes sense for you and your [46:37] business, especially those high [46:39] frequency um uh flows. [46:44] So um so thank you everyone. We are at [46:47] time actually. Thank you so much for [46:49] attending our talk. [46:51] >> Thank you. [46:52] >> Appreciate your time. Thank you.