JPMorgan, US Bank Rein In AI Agents Over Unverified Planning Risks

因未经核实的规划风险，摩根大通与美国合众银行收紧AI Agent管控 企业级AI Agent的落地部署所遭遇的瓶颈并非模型智能本身，而是多步状态漂移与未经核实的规划轨迹，这迫使各类机构将自主系统约束在确定性代码护栏之后。 与会嘉宾深入探讨了如何构建可靠的Agent架构，对比了动态ReAct循环与传统基于规则的工作流，同时强调了Agent控制框架（harnesses）在状态管理和成本控制中的关键作用。来自强监管银行业和平台工程背景的发言者讨论了如何通过确定性护栏、隔离数据网格以及渐进式技能加载，来缓解长周期任务中的上下文退化问题。他们指出了生产环境中无约束自主性与身份委托所带来的严峻运维风险，并将企业级分阶段自主与人机协同验证，与开源开发中常见的快速实验进行了对比。最后，对话展望了世界模型的普及化，以及医疗健康和供应链物流等依赖纸质流程的传统线下行业的运维自动化。 Agent系统与传统基于规则的软件有着本质区别，它用受明确运维护栏约束的动态“推理-行动-观察”（ReAct）循环，取代了静态的DAG流程。 (1:42) Agent控制框架提供了关键的状态与上下文管理能力，使架构师能够通过将专用的轻量级模型分配给具体子任务，并将昂贵的前沿模型保留用于高自主性规划，从而优化成本。 (4:16) Agent架构可划分为三个级联层级：业务复杂度（定义工作流与角色）、模型复杂度（权衡延迟与推理深度）以及数据隔离（管控数据主权与网络边界）。 (7:40) 在生产环境中，团队很少仅仅为了追求基准测试分数的提升而更换底层LLM；系统一旦验证可靠，工程重点就会转向Token优化、成本控制和投资回报率。 (11:04) 结构化的问题分解降低了对高成本前沿模型的需求，因为定义明确的小型子任务通常可以由更便宜或较早期的模型以同等可靠性来执行。 (13:15)

Enterprise deployment of autonomous agents requires decomposing complex workflows into tightly scoped, deterministically bounded sub-tasks rather than relying on unconstrained frontier models. Engineering leaders from JPMorgan Chase, U.S. Bank, Apple, and CrewAI argue that long-horizon reliability and regulatory compliance depend on context management, strict identity isolation, and phased human oversight. Architectural Layering and Cost-Driven Model Selection (7:40) Traditional automation relies on static Directed Acyclic Graph (DAG) flows, whereas agentic systems employ dynamic Reason-Act-Observe (ReAct) loops. To deploy these loops effectively, Ashok Prakash outlines a three-layer architecture: defining business complexity and agent roles, balancing model intelligence against latency and cost, and enforcing data isolation boundaries. Panelists emphasize that once systems enter production, teams rarely swap models to chase benchmarks; instead, focus shifts toward token optimization and return on investment. Seetaram Rao Rayarao notes that system architects must structure problems into modular components. Dividing complex workflows into smaller, specialized sub-tasks enables enterprises to deploy lightweight or older models for routine execution, reserving expensive frontier models for high-autonomy planning steps. Rayarao also observes that aggressive provider-level safety filters on multi-trillion parameter models can inadvertently degrade reasoning on legitimate, complex operations, making structured problem framing essential. Context Drift and Long-Horizon Reliability Limits (23:04) A critical failure point in agent architecture is the performance degradation observed during long-horizon tasks exceeding one hour. Jason Bonvie notes that despite expanding model context windows, reliability remains flat because agents suffer from context and state drift as they navigate deep reasoning branches. Prakash adds that unconstrained agents can rapidly inflate context to a million tokens; when automated context compaction triggers, agents often lose their core objective and repeat completed tasks. To counter this drift, organizations implement context engineering and grounding hooks. Prakash explains that integrating structured, human-curated knowledge bases—tested to 99% reliability thresholds across permutations—constrains agent planning within approved business rules. At JPMorgan Chase, Rayarao explains that teams deploy tenant-isolated Retrieval-Augmented Generation (RAG) within a centralized data mesh, pairing it with progressive skill disclosure so that detailed API schemas and tool instructions load only when an agent explicitly triggers that capability. Deterministic Verification and Identity Delegation Risks (18:17) Autonomous planning introduces validation gaps because identical prompts can yield completely different intermediate reasoning traces. In regulated environments, Dhivya Nagasubramanian stresses that evaluations cannot rely solely on final outputs. U.S. Bank embeds deterministic Python checks directly into task-level agents to verify tool selection and argument specifications before errors can cascade, using explicit rubrics to route uncertain actions to human-in-the-loop (HITL) review. The panel also warns against privilege delegation pitfalls. When autonomous agents execute tasks using personal engineer credentials, any destructive operation—such as accidental database deletion—carries personal attribution in audit logs. Panelists argue that agents must be restricted to least-privilege, read-only personas, with institutions adopting phased governance pipelines that mandate 100% human supervision during pilot stages before granting graduated autonomy. Operational Realities Across High-Stakes and Open-Source Domains (42:11) The panel identifies substantial near-term value in applying agents to administrative bottlenecks in legacy offline sectors. In healthcare workflows, automating patient history and medication contraindication cross-checks allows nurses to review pre-screened orders rather than manual files, reducing administrative overhead by two to three hours per encounter while preserving human sign-off. Conversely, autonomous financial transactions face a sharp divide. Nagasubramanian expresses skepticism regarding near-term autonomous financial operations at scale, arguing that institutions require verifiable 100% correctness before allowing agents to transact. While traditional institutions remain cautious due to compliance mandates, panelists note that open-source developers and startups are actively building synthetic agent identities and alternative transaction rails outside corporate frameworks. The fundamental tension in agent design lies between unconstrained autonomy and verifiable determinism. While open-source ecosystems experiment with autonomous economic agents, enterprise adoption remains anchored to rigid guardrails, scoped contexts, and human oversight until long-horizon state drift and reasoning verification are solved.

来自金融、科技及智能体开发前沿的专家指出，构建企业级智能体（Agent）系统的核心挑战不在于底层前沿模型能力的单向堆叠，而在于通过运行框架、确定性护栏与上下文工程化，克服长周期任务的状态漂移与评估验证断层，在受控边界与业务投资回报之间取得平衡。 三层架构设计与成本效率权衡 (7:40) Ashok Prakash 将智能体架构划分为三个层级：界定具体企业业务场景与角色职责的“业务复杂度层”、平衡模型推理深度、延迟与调用开销的“模型决策层”，以及界定网络、状态与数据主权边界的“数据隔离层”。与传统依赖固定有向无环图的电话语音等规则系统不同，智能体依赖动态的 ReAct（推理-行动-观察）循环来处理非静态任务。Dhivya Nagasubramanian 强调，凡是标准确定性引擎能解决的场景均不属于智能体范畴，智能体模式专门用于在边界策略与护栏约束下处理动态工作流。 在工程落地中，Jason Bonvie 指出“智能体运行框架”（Harness）不可或缺，它提供细粒度的状态与上下文管理，防止大语言模型调用失控。面对频繁发布的模型版本，专家们认为工程重心已从追求极限性能转向严格的 Token 优化与投资回报率（ROI）。团队不应在复杂多智能体链条中全盘部署高昂的前沿模型，而应按子任务自主度进行分级：轻量检索或摘要分配给低成本模型，高自主度决策才调用前沿模型；通过结构化拆解问题，普通模型也能稳定完成既定目标。 规划评估盲区与确定性校验护栏 (18:17) Seetaram Rao Rayarao 指出当前行业存在显著的“验证断层”：生产评估框架普遍只测量最终输出，却忽视了推理规划本身的结构有效性与执行路径。两款前沿模型面对同一提示词往往会生成完全不同的推理步骤，这种未经校验的规划路径极易在长线运行中引发灾难性失败。与此同时，模型厂商施加的安全过滤机制虽然遏制了高危操作（如误删生产环境指令），但也频繁带来非线性能力衰减，意外削弱了复杂合法任务的推理表现。 为应对此类级联复杂性，金融等强监管行业拒绝依赖纯黑盒自主性。Dhivya Nagasubramanian 与 Jason Bonvie 提出，企业必须在任务层嵌入确定性 Python 检查与布尔断言，对调用的工具、参数格式及推理轨迹实施硬性验证，而非单纯依赖昂贵的大模型裁判（LLM-as-a-judge）。在银行业务中，系统通常采用分级评分机制：高置信度动作自动放行，处于不确定区间的操作路由至人工介入（Human-in-the-Loop）审核，低置信度操作则立即阻断并终止执行循环。 上下文状态漂移与长周期任务瓶颈 (23:04) Jason Bonvie 观察发现，智能体执行超过一小时的长周期任务时常出现严重的可靠性崩溃，即“一小时之墙”。该瓶颈并非源于基础模型的智能上限，而是由状态和上下文管理失效引起的“上下文失忆症”。当推理分支过深或上下文过度膨胀时，智能体会逐渐偏离核心目标。Ashok Prakash 举例指出，在某次未加约束的实验中，智能体为完成创建目录并保存文件的任务，上下文膨胀至 100 万 Token，触发记忆压缩后直接丢失了初始指令。 为遏制长周期任务中的混乱，工程团队通过收窄工具权限与上下文范围来降低决策分支维度。Prakash 建议引入结构化业务知识库作为行为锚点，将动态推理框定在已知操作规范内，并在投产前通过测试框架验证达到 99% 的一致性阈值。Rayarao 则分享了摩根大通的实践：利用基于集中式数据网格的租户隔离 RAG 管道过滤冗余信息，并推行“技能渐进式披露”，仅在明确需要调用某项能力时才动态载入完整 API 规范，从而显著减轻活动上下文的负荷。 身份特权风险与分阶段自主落地 (33:29) 当自主智能体直接接入生产基础设施时，权限代理带来了巨大的操作隐患。现场讨论指出，若直接将工程师个人的 IAM 权限授予智能体（例如用于查询 AWS CloudWatch 日志），智能体将继承该账号的所有高阶权限，甚至具备删除线上生产数据库的能力，且任何异常操作在审计日志中都将直接归咎于人类员工本身，因此必须严格实施只读与最小特权隔离。 在业务应用推进上，企业普遍采取保守的分阶段落地路径。即便系统在实验室测试中达到 99.999% 的可靠性，初版上线仍需保持 100% 的人工监督以建立业务信任，随后再根据评估标准逐步放宽自主权。针对部分观点预测智能体将在两年内大规模自主执行金融交易，Nagasubramanian 明确表示，受监管与问责约束，主流金融机构短期内不会采纳未经 100% 验证的自主交易系统。相比之下，在医疗临床医嘱审核（每位患者可节省 2 至 3 小时行政耗时）、邮件分流等领域，人机协同智能体已展现出明确的落地价值。 与会专家达成共识，智能体演进的技术重点正转向具备自我评估与递归优化能力的运行框架，以及模块化的世界模型。然而，真正的商业与架构挑战并非实现完全无约束的自主通用智能，而是在企业合规、严格审计与技术脆弱性之间建立严密的防护边界。未来智能体经济价值释放最显著的领域，在于渗透至仍依赖纸质与手动审批的传统实体业务流程中，通过人机协同消除官僚壁垒，而非完全取代人类监管。

## Background 该内容由 Ai4 于 2026 年 8 月 22 日发布在 YouTube 平台。视频记录了一场关于“智能体大脑（Agentic Brain）”架构与训练体系的行业圆桌研讨，来自 Apple、美国合众银行（U.S. Bank）、摩根大通（JP Morgan Chase）以及 CrewAI 的技术高管与工程师深入探讨了智能体的推理规划、记忆与工具调用、安全护栏以及在生产环境落地中的关键权衡与前沿趋势。 ## TL;DR 圆桌讨论围绕智能体系统从理论到工业落地的核心架构展开，指出智能体与传统自动化的根本区别在于具备 ReAct 闭环和自适应推理能力。专家们剖析了长时程任务中的上下文衰减、模型升级导致的能力回退等工程痛点，提出通过 Agent Harness 状态管理、技能渐进式加载、数据网格（Data Mesh）以及轻量确定性 Python 拦截来构建可靠防线。针对金融等强监管行业，对话强调了多租户隔离、推理路径可验证性与分级人机协同（HITL）的必要性，并展望了未来世界模型商品化与开源自主交易智能体的演进方向。 ## Atmosphere & style 对谈整体基调务实、敏锐且极具一线工程沉淀感，主持人的幽默穿插让硬核的技术探讨生动自然。现场展现了极为真实的观点交锋——一边是传统金融机构在严监管下对合规、确定性与零容错的审慎防守，另一边则是初创生态与开发者对赋予智能体独立身份和自主交易权的激进畅想。整场讨论既有剖析生产环境“踩坑”案例的直率与风趣，又兼具对技术边界与产业变革的深刻洞察。 ## Key takeaways - [01:42](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=102s) 智能体与传统自动化的核心分野在于动态自适应决策与 ReAct（Reason-Observe-Act）推理闭环，能够根据环境反馈自主调整路径，而非执行预定义的静态规则分支。 - [04:16](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=256s) Agent Harness 框架通过任务级状态与上下文管理保障长时程运行稳定性，而底层模型选型的核心是在任务认知要求与 Token 成本之间取得 ROI 平衡。 - [07:39](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=459s) 企业级智能体系统呈现业务复杂度、模型决策与延迟、数据隔离与安全的三层架构，自顶向下决定了是采用公有云 API 还是私有主权模型隔离。 - [11:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=664s) 生产环境中模型迭代的重心已从追求极致跑分转向结构化解耦与 Token 优化；只要子任务拆解清晰，低成本基础模型同样能稳定胜任。 - [14:30](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=870s) 超大模型引入的底层安全护栏（Guardrails）虽然封堵了破坏性指令漏洞，但也无形中限制了模型的推理灵活性，常导致特定任务发生性能回退。 - [17:00](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1020s) 外层 Agent 护栏落地应采用轻量级确定性 Python 脚本与布尔表达式核验输出，一旦出现异常立即重构上下文或熔断，防止错误扩散。 - [18:16](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1096s) 金融级场景追求推理的可验证性（Verifiability），通过确定性流程审计与量规（Rubric）分级，在未知模糊区间触发人机协同（HITL）。 - [23:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1384s) 超长时程任务的执行衰减并非模型基座智力不足，本质是 Context 与 State 的发散漂移；需通过收敛工具空间和专业化 Agent 压缩决策表面积。 - [25:08](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1508s) 结合知识库规则约束（如 OKDF）以及数据网格（Data Mesh）与技能渐进式加载（Progressive Disclosure），可避免上下文噪音并实现 99% 的准入合格率。 - [31:37](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1897s) 无边界 Agent 在长程交互中极易陷入上下文遗忘与混沌失控；高保真沙盒追踪在底层消耗高昂算力与内存，企业更侧重防越权与沙箱隔离。 - [33:29](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2009s) 智能体继承人类身份凭证存在极高误操作风险，受监管环境必须推行设计即合规（Design Phase Governance）与基于置信度的渐进式放权。 - [36:07](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2167s) 未来世界模型将走向通用商品化与即插即用，但在百万级自主交易智能体涌现与强监管合规之间，传统机构与开源初创生态存在着显著的演进分歧。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=4s) 嘉宾介绍与议题开场 圆桌主持人 Darrell Etherington 幽默地为本场讨论定下基调，表示今天的主题是构建“智能体大脑”（Agentic Brain）的核心架构考量；他打趣称自己其实完全没有提前做准备，打算和现场所有观众一起“实时探秘”。随后，四位深耕 AI 与大模型工程一线的重磅嘉宾依次介绍了各自的背景与业务范畴： * **Dhivya Nagasubramanian**：美国合众银行（U.S. Bank）AI 转型与创新副总裁（VP of AI Transformation and Innovation）。 * **Seetaram Rao Rayarao**：摩根大通（JP Morgan Chase）高级主任软件工程师（VP, Senior Lead Software Engineer），主要负责工程类智能体框架建设，该平台面向全公司约 20 万名员工提供智能体赋能。 * **Jason Bonvie**：CrewAI 前向部署工程师（Staff Forward Deployed Engineer），专注于企业级生产环境落地，迄今已主导并负责了跨大型企业的 1000 多个 Agent 生产部署。 * **Ashok Prakash**：苹果公司（Apple）主任机器学习工程师（Staff ML Engineer），供职于 Apple AI Platform 团队，该平台负责为包括 Siri、Apple Music 等在内的核心 AI 服务提供底层基础设施支撑。 --- ### [01:42](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=102s) 什么是“智能体大脑”：智能体与传统自动化的核心分野 讨论首先从底层概念的界定切入：究竟什么是“智能体大脑”？它与传统非智能体 AI 或传统自动化有何本质区别？来自 Apple 的 Ashok Prakash 首先给出了解释。他指出，智能体与传统系统的核心分界在于**是否需要做出非静态的智能决策**。在传统方案中（例如传统的交互式语音应答 IVR 系统），业务逻辑通常由固定的有向无环图（DAG flow）写死，用户只能按部就班地根据语音提示“按 1 或按 2”。而在智能体系统中，系统可以直接理解用户的自然语言意图，完成动态分类、自主决策并调用后续路径，彻底摆脱了机械死板的规则分支。 Dhivya Nagasubramanian 对此补充道，任何可以通过传统规则引擎（Rule-based Engine）穷尽的自动化流程都算不上智能体；只有当业务流具备动态性，且必须在“受控策略与边界护栏”（Bounded policies and bounded guardrails）内自适应演进时，引入智能体才真正具备价值。Seetaram Rao Rayarao 则从执行机制上进行了提炼：传统工作流是预定义的、不可更改的，而智能体工作流的核心在于引入了 **ReAct 循环（Reason-Observe-Act）**——它首先进行推理，识别需要完成的任务，进而观察环境反馈并采取行动，这种持续闭环的自适应推理正是智能体区别于以往技术的最大飞跃。 ```mermaid flowchart TD accTitle: ReAct Loop Diagram subgraph Traditional["传统非智能体系统"] A["预定义规则 / 静态 DAG"] --> B["机械执行 '按 1 / 按 2'"] end subgraph Agentic["智能体大脑 (ReAct 闭环)"] C["Reason 推理<br/>(识别任务与意图)"] --> D["Act 执行<br/>(调用工具 / 生成决策)"] D --> E["Observe 观察<br/>(获取环境与工具反馈)"] E --> C end ``` --- ### [04:16](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=256s) 架构落地的第一步：Agent Harness 框架与模型选型权衡 当工程师着手构建一个智能体系统时，应当如何选型和架构？来自 CrewAI 的 Jason Bonvie 从工程框架层面阐述了为什么不能仅仅依赖单次 LLM 调用。他指出，**Agent Harness（智能体运行底座/套件）** 的核心作用在于提供任务级别的颗粒度控制，使开发者能够精确注入专业知识与上下文，并实施状态管理（State Management）和上下文窗口控制（Context Management），防止大模型调用出现“野蛮失控”或死循环。这种 Harness 是让智能体能够长期、稳定执行长周期任务（Long-running tasks）不可或缺的抓手。 在底层模型（Foundation/Frontier Models）的选型策略上，Jason 坦言，当前企业在选择前沿模型时，最根本的权衡就是**成本与任务所需认知能力（Cognitive Output）的匹配**。对于文本摘要、简单数据检索等轻量级任务，完全可以选用便宜的轻量化模型；而当赋予智能体更高的自主权、涉及复杂决策逻辑时，才需要调用高成本的顶尖前沿模型。 Dhivya Nagasubramanian 结合实际落地经验进一步说明：在一个包含 5 到 6 个步骤的多智能体协同工作流（Multi-agent Orchestration）中，架构师必须拆解每个环节的目标与认知要求。例如，代码生成任务可以交给专有代码大模型，而信息汇总步骤则分流至轻量模型。Dhivya 特别强调，这种模型选型与成本优化的决定权通常不在业务业务方（Stakeholder），而属于设计技术架构的架构师与技术专家。 --- ### [07:39](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=459s) 智能体系统的三层架构复杂度与企业级定制实践 针对智能体架构的系统化设计，Ashok Prakash 提出了一个清晰的“三层复杂度”架构模型： 1. **业务复杂度层（Business Complexity Layer）**：整个架构的最顶层输入。定义具体的业务场景（如对话式语音助理 IVA、客服聊天机器人、后台异步智能体或特定功能智能体）。 2. **模型决策与复杂度层（Model Complexity & Latency Layer）**：根据业务需求权衡模型的认知深度、推理延迟与调用成本。大模型虽聪明但耗时长、花费高；小模型速度快、成本低但推理能力相对受限。 3. **数据隔离与安全层（Data Isolation Layer）**：底层基础设施与合规考量。决定企业数据是可以通过公有云 API 与第三方前沿模型交互，还是必须在企业私有边界内运行主权模型（Sovereign Model），实现严格的网络、状态和数据隔离。 Ashok 指出，这三层之间是自顶向下层层约束的：业务复杂度决定了模型复杂度，而模型决策又倒逼出数据与状态隔离的边界。 ```mermaid flowchart TD accTitle: Three-Layer Agent Complexity Architecture subgraph Layer1["1. 业务复杂度层 (Business Complexity)"] B1["功能型 / 后台智能体"] B2["语音 IVA / 交互式工作流"] end subgraph Layer2["2. 模型复杂度与延迟层 (Model Complexity & Latency)"] M1["前沿大模型 (高智力 / 高延迟 / 高成本)"] M2["轻量化小模型 (低延迟 / 经济实惠)"] end subgraph Layer3["3. 数据隔离与安全层 (Data Isolation Layer)"] D1["第三方 Cloud API 模式"] D2["企业私有主权模型 / 网络隔离"] end Layer1 ==> Layer2 Layer2 ==> Layer3 ``` Seetaram Rao Rayarao 随后分享了摩根大通的内部工程落地模式。为了应对不同模型在推理、代码生成等维度的能力差异，他们为全行员工提供了高度可定制的工作流界面与 SDK（类似于 CrewAI 的可视化拖拽与代码 SDK 结合体验）。用户可以根据具体的业务场景，自主配置不同的工作流节点、绑定最适配的特定模型并配置专属系统提示词（System Prompts），从而在企业受控框架下实现了最大程度的灵活性与定制化。 ### [11:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=664s) 架构迭代与模型替换的现实考量：ROI 与 Token 优化 在大模型技术飞速演进的背景下，架构师面临着设计决策“落地即过时”的困境。针对新技术与新专长模型层出不穷时该如何应对这一问题，Jason Bonvie 指出，在生产环境中目前并没有放之四海而皆准的大规模应对方案，大多仍需按具体情况逐案处理。引入 Agent 框架（Agentic Harness）的优势在于能够降低替换底层模型和做对照测试的门槛。然而在实际落地中，除非旧模型停止技术支持或性能严重劣化，否则很少有团队会主动回滚或频繁更换模型。一旦某套架构通过严苛测试并上线运行，团队通常会保持稳定，后续仅做成本优化。现阶段行业讨论的重心已从盲目的“追求极致性能”全面转向“Token 优化”与投资回报率（ROI）。 ```mermaid flowchart TD accTitle: Production Agent Model Lifecycle and Verification accDescription: Flow showing how task decomposition, deterministic checks, and multi-tier routing govern agent execution. subgraph ProblemDesign["1. 结构化解耦与模型分发"] A["复杂业务问题"] --> B["拆解为标准化子任务"] B --> C{"任务复杂度匹配"} C -->|"基础/常规任务"| D["低成本模型 (如 Claude Sonnet)"] C -->|"高复杂度任务"| E["前沿旗舰模型"] end subgraph ExecutionValidation["2. 执行与多重护栏校验"] D --> F["Agent 执行任务并生成 Trace"] E --> F F --> G["确定性 Python / 语义规则护栏"] G --> H{"单步逻辑与工具调用是否合规?"} H -->|"异常 / 越权"| I["中断循环 / 重新上下文重试"] H -->|"正常"| J["评分量规 (Rubric) 与 LLM 裁判"] end subgraph QualityRouting["3. 分级路由控制"] J --> K{"置信度评估区间"} K -->|"高于截断阈值"| L["放行进入下一阶段 Agent"] K -->|"处于未知/模糊区间"| M["触发 Human-in-the-Loop 介入"] K -->|"低于底线阈值"| N["立即熔断并阻断级联"] end ``` Darrell Etherington 对此深有共鸣，他调侃行业此前迅速经历了从“Token Maxing（无节制消耗 Token）”到“等等，千万别这么做”的戏剧性转变。在实际应用中，这种逻辑类似于企业中稳定运行了五年的 UiPath RPA 自动化脚本，只要没有彻底报错崩溃，开发者便倾向于让其持续服役。Seetaram Rao Rayarao 进一步补充，从 Claude 4.6、4.7 到各类前沿大模型，基准测试（Benchmarks）上的智能指标确实大幅跃升，但对于特定任务，高昂的旗舰模型相较于低成本模型并未展现出实质性差异。面对特定任务，没有必要为昂贵的模型支付高昂调用费，选用更经济的轻量模型完全足够。工程的核心在于将复杂问题拆解为定义清晰的子模块，只要问题结构化做得足够扎实，即使是基础模型也能稳定解决。 - 生产环境模型替换的核心触发条件通常是官方弃用（Deprecation）或支持降级，而非单纯追逐前沿基准跑分。 - 架构设计的核心原则是问题结构化（Problem Structuring）：将大问题拆分为标准化子任务，使低成本模型也能胜任。 --- ### [14:30](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=870s) 模型非线性升级与安全护栏（Guardrails）导致的性能回退 技术迭代并不总是带来线性收益，模型在版本跃迁中经常会在特定功能领域出现意外的性能回退（Regression）。Seetaram Rao Rayarao 指出，这种现象在很大程度上是由安全护栏（Guardrails）机制引起的。当参数规模达到 5 万亿至 6 万亿级别的超大模型问世时，其自主执行能力大幅提升，但也随之带来了严重的安全漏洞与网络攻击隐患。 如果模型在缺乏约束的情况下直接开放给用户，即便用户本身没有恶意，高智能模型也可能在后台做出超出预期的危险操作——例如在创建临时文件夹时生成并执行含有潜在恶意行为的 Shell 脚本，甚至静默运行类似 `rm -rf` 的破坏性系统指令。为了封堵这些漏洞，模型提供方必须在底层叠加极其严苛的 Guardrails。这种防御层虽然提升了整体安全性，却也无形中束缚了模型的推理灵活性，导致某些在上一代模型中表现优异的具体任务在升级后反而出现能力下滑。 --- ### [17:00](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1020s) Agent 安全护栏设计：确定性校验与多层拦截机制 当开发者在模型自带的原生护栏之上，再通过外层 Agent 框架（Harness）叠加额外的控制策略时，容易产生级联复杂性，甚至引发不可预测的系统交互。Jason Bonvie 阐述了构建企业级 Agent 工作流时的护栏落地实践：系统首先应具备广泛连接外部工具的能力，随后依据组织策略实施权限管控，包括分层身份认证、基于操作人员身份动态调整工具可用性、严格限制 Token 上限以及验证任务产出的合法性。 在具体工程实现上，护栏完全可以通过轻量化且高可靠的方式部署在单步任务层面： 1. **任务先行执行**：允许 Agent 先尝试执行目标任务，不对其推演过程做过多硬编码限制。 2. **确定性布尔判定**：在输出端接入轻量级的语义检查或 Python 脚本，以布尔表达式（Boolean Expression）对结果进行快速判别。 3. **熔断与重试策略**：检查输出是否满足预设标准、是否存在恶意攻击或滥用痕迹。一旦结果为 True 则放行；若为 False，系统将立即进行上下文重构（Recontextualize）、重新处理任务或直接退出循环，防止错误扩散。 --- ### [18:16](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1096s) 金融级 Agent 系统的推理可验证性（Verifiability）与人机协同分级 Dhivya Nagasubramanian 从金融行业的严苛监管视角切入，强调在金融业务流中，确保 Agent 准确执行既定目标具有最高优先级。当前金融场景所追求的核心是任务的“可验证性（Verifiability）”。过去系统通常只校验初始输入和最终输出，但在复杂的自主决策流中，两个 Agent 面对相同目标与任务时，可能采用完全不同的推理路径却偶然得出相同的正确结果。这种“侥幸成功”在生产环境中运行几十次后极易暴露出严重缺陷，因此系统必须具备验证“推理合理性（Reasonability）”的能力。 ``` 输入 Trace 评估 │ ▼ ┌──────────────┐ │ Rubric 评分 │ └──────┬───────┘ ├────────────────────────┬────────────────────────┐ ▼ ▼ ▼ [ 高于阈值区间 ] [ 模糊未知区间 ] [ 低于底线区间 ] 自动放行 (Pass) 人机协同 (Human-in-the-Loop) 立即熔断 (Halt) ``` 为防止错误在多 Agent 系统中悄无声息地级联蔓延，U.S. Bank 在架构中深度嵌入了确定性校验层： - **确定性流程审计**：严格比对工具调用的顺序、方式及参数是否正确，输出是否真实，以及推理轨迹（Reasoning Trace）是否符合预期业务范式。 - **确定性 Python 函数拦截**：通过轻量级代码直接核验执行轨迹，一旦偏离预设轨道即刻阻断动作，杜绝过大的自由发挥空间。 - **分级评估与成本控制**：针对多模型裁判（LLM Judges）带来的额外 Token 开销，团队制定了量规（Rubric）分级机制： - **高置信区间**：高于截断阈值的推理直接自动放行； - **模糊/未知区间（Unknown Zone）**：触发人机协同（Human-in-the-Loop），交由人工审核； - **低分区间**：直接阻断并熔断流程，杜绝潜在风险。 - 规则系统（Rules-based）依然是金融 Agent 架构中不可动摇的底座，必须贯穿于顶层编排器及执行具体任务的各个子 Agent 之中。 - 质量控制需兼顾良品率与 Token 成本：单纯依赖 LLM 裁判成本高昂，结合轻量确定性代码与分级量规是平衡安全与 ROI 的关键路径。 ### [21:28](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1288s) 金融强监管下的多租户隔离与 Agent 推理评估困境 在评估与校验 Agent 时，若引入多个 LLM 担任裁判（LLM-as-a-judge），其带来的 Token 成本开销将非常庞大。摩根大通（JP Morgan Chase）的 Seetaram Rao Rayarao 随即补充指出，银行属于极其严苛的受监管行业（strictly regulated environment），在架构上必须采用基于租户的多租户隔离设计（tenant-based setup）。每个租户只能访问特定且受限的数据源、数据集以及经过授权的模型，各租户之间设立了严格的规章与安全策略，绝不允许越界（cross a boundary）。 ```mermaid flowchart TD accTitle: 强监管环境下的多租户隔离架构 subgraph MultiTenant["多租户隔离环境 (Tenant Isolation)"] T1["租户 A 规则 / 数据集 / 模型"] T2["租户 B 规则 / 数据集 / 模型"] end subgraph Dilemma["Agent 核心矛盾"] D["确定性工作流<br/>(丢失自主性/失去规划能力)"] A["Agentic Harness 循环<br/>(具备自主推理/但不可控且难评估)"] end MultiTenant --> Dilemma ``` 紧接着 Dhivya 提出的观点，Seetaram 深入剖析了 Agent 设计中的核心矛盾：如果采用完全确定性的工作流（deterministic workflows），系统就会丧失自主性（autonomy），本质上变成了命令 Agent 按部就班地按 XYZ 序列执行任务；然而，Agent Harness 和循环机制的初衷是让模型能够自主推理、规划并执行任务。当前整个行业面临着巨大的治理空白：如果给两个不同的前沿模型输入相同的 Prompt，它们可能会生成截然不同的推理路径与规划步骤。目前行业普遍只验证最终输出（final outcome），却根本无法有效验证与把控 Agent 中间推理规划的合理性、执行的具体子任务数量以及每个步骤的响应质量。 - **多租户数据隔离**：金融场景下严格限制模型与数据调用边界，杜绝跨租户数据泄露。 - **行业评估缺失**：缺乏针对 Agent 中间规划逻辑（Planning）与任务分解过程的有效验证工具，评估大多停留在最终结果的端到端对比。 --- ### [23:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1384s) 规划合理性与长时程任务的上下文状态衰减 针对中间步骤难以验证的问题，CrewAI 的 Jason Bonvie 表示认同。他指出，Agent 非常擅长制定任务规划，但规划本身是否符合业务逻辑却极难验证。在构建 Agent 工作流时，最大的挑战在于如何建立一种结构，使 Agent 的自主规划既有章可循，又不会偏离预期目标。因此，业界目前的应对方案是构建“高度专业化的 Agent”（specialized agents）——通过极度收敛输入上下文、限制特定能力，并缩小其可调用的工具集合（scope down tools），以此压缩 Agent 的操作表面积（surface area），从而在有限的决策空间中保证逻辑收敛。 Jason 进一步剖析了任务衰减（task degradation）的深层机理：这种退化主要发生在执行周期超过 1 小时的长时程任务（longer-horizon tasks）中。过去六个月里，尽管各大前沿模型的上下文窗口不断扩大、单步推理能力持续增强，但在超长任务上的表现依然未见明显突破。Jason 强调，这并非前沿模型（Frontier models）自身能力不足，而本质上是一个“上下文管理与状态管理”（context management and state management）工程问题。随着推理分支不断发散蔓延，Agent 会逐渐丢失最原始的目标；唯有通过严密掌控长运行任务中的叙事与上下文链路，才能保证系统产出高质量的确定性结果。 - **范围约束（Scoping Down）**：通过缩小工具空间和上下文边界，将推理路径限制在可穷举的合理决策组合内。 - **长时程退化根因**：超长任务执行失败的核心在于 Context 与 State 的发散与漂移，而非模型的底层基座推理能力不足。 --- ### [25:08](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1508s) 知识库边界绑定与 99% 合格率测试机制 Apple 的 Ashok Prakash 结合自身的业务经验，分享了如何通过知识库（Knowledge Bases）来有效约束 Agent 的推理边界。以典型的“订单库存系统”（order inventory system）为例：一个订单处理 Agent 需要判断订单是否履约并流转到下一步。开发团队会将整套企业级的订单处理流程、履约中心规则、分拣打包节点以及客户交互标准等由人类长期验证沉淀的知识库完整输入。这些知识库充当了 Agent 推理过程中的“挂钩”（hooks），将其思维限定在人类既定业务规则内。 ```mermaid flowchart LR accTitle: 知识库约束与评测准入流程 KB["企业知识库 / 技能定义<br/>(OKDF / Human-coded rules)"] --> Agent["专业化 Agent<br/>(Bounded Reasoning)"] Agent --> Harness["评估 Harness 矩阵验证<br/>(N 组组合测试)"] Harness --> Pass{"合格率达到 99%?"} Pass -- "是" --> Prod["准入上线 Production"] Pass -- "否" --> Fix["调整约束 / 重新优化"] ``` 通过这种强边界约束，原本开放的生成式模型被转化为只有少数几种输出可能性的确定性系统，无论底座换成何种模型都不会越界。Ashok 提到了行业正在兴起的新范式，例如 Google 推出的开放知识文档格式（Open Knowledge Document Format, OKDF）以及模块化的 Agent Skills。在准入上线方面，团队会采用 Harness 对各类组合情况进行大规模自动化验证，设定严苛的置信度阈值（例如达到 99% 的输出符合预期）后才允许推向生产环境（Production）。 - **知识库边界约束**：利用 OKDF 或知识库构建人类业务规则闭环，将不确定的推理收敛为高确定性结果。 - **上线标准门禁**：通过评估 Harness 穷举排列组合输入，达到 99% 预期符合率后方可进入生产环境。 --- ### [27:51](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1671s) 数据网格（Data Mesh）、RAG 与技能渐进式加载 针对 Jason 提到的长时程任务中“大海捞针”（needle in a haystack）式的上下文丢失痛点，Seetaram 分享了摩根大通内部的工程实践方案。他们搭建了企业级的集中式数据网格（centralized data mesh），各个业务团队（租户）可将专属数据集接入中央平台并注入到 RAG（检索增强生成）管线中。通过这套基于租户的多租户 API 检索接口，Agent 仅需查询与当前任务强相关的信息碎片，无需将全部原始资料塞进 Prompt，从根本上实现了高效的上下文工程（Context Engineering）。 此外，Seetaram 与 Ashok 强调了“技能渐进式加载”（progressive disclosure of skills）的重要性。在定义 Agent Skills 时，系统在初始阶段仅暴露极简的元数据描述（Description），只有在实际任务流程触发该技能时，才会按需动态加载具体执行逻辑与上下文。这两项技术紧密配合，有效避免了超长上下文窗口被无效噪音淹没，大幅提升了长时程执行的稳定性。 - **集中式 Data Mesh + RAG**：结合多租户安全隔离，按需精准抽取业务数据，优化上下文负载。 - **技能渐进式公开（Progressive Disclosure）**：仅在用到特定 Skill 时动态加载详细指令，减轻常驻上下文开销。 --- ### [29:12](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1752s) 无边界 Agent 探索与企业落地的现实鸿沟 讨论转向了反向思路：业界是否存在完全不设边界（Unbounded Agents）的通用 AGI 路径？Jason 指出，在开源和极客社区中，像 NemoClaw 这类主打无拘无束、全自主探索的项目广受欢迎。但企业级应用与个人玩具存在着天壤之别：企业永远在思考“如何阻止 Agent 做出意料之外的行为”，而个人开发者则希望拥有一个无所不能的超级通才。 现场嘉宾展开了幽默而真实的讨论：主持人打趣道，工程师们总陷入一种无限循环——不断添加新的安全栅栏试图“关住野兽”，但内心深处又渴望拥有一只无拘无束的“超级神兽”。Jason 笑道，在个人生活中，他甚至愿意把借记卡直接交给 Agent，任其自主点外卖和购买生活杂货送货上门；然而在面对军方客户或核心企业系统时，安全红线极其严苛，首要任务是确保数据绝对不可窥探、各系统绝不交叉污染（cross-pollinate）。Ashok 补充分享了一个无边界 Agent 的实际翻车案例：他曾让一个 Agent 执行创建文件夹及文件的简单任务，然而在模型上下文因长期交互爆炸到 100 万 Token（1 million context）后，系统在尝试压缩上下文时瞬间丢失了原始记忆指令，直接导致执行逻辑崩溃。 > "In my personal life I definitely don't put any gates on my agents. I'm like here's my debit card. Enjoy. Please get the groceries to the front door... very much so a different story when I'm talking to the military." —— Jason Bonvie - **个人与企业诉求分野**：个人端追求完全放权的通用全能助手；企业与军工场景则要求严格的沙箱隔离与防越权保护。 - **超长上下文压缩缺陷**：在 1M 上下文急剧膨胀并进行自动压缩（Compact Context）时，模型极易遗失关键基准指令。 ### [31:37](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1897s) 上下文退化与“三体混沌”：无界环境中的智能体失控与沙盒成本 当智能体被置于缺乏严格边界的环境中时，极易陷入失控的混乱状态。随着执行轮次的增加，智能体会遗忘人类最初给出的具体指令，开始随意创建未经授权的文件夹与杂乱文件。这种行为不仅让人类大脑难以追踪，智能体自身也会遭遇严重的性能退化。即便拥有长达 100 万 token 的上下文窗口，在经历三到四轮迭代后，模型往往就会丢失最初存放文件的状态与路径信息，进而陷入机械重复已有动作的死循环。讲者将这种失控状态比作物理学中的“混沌理论”与“三体问题”——在无界环境中，智能体不仅失去了可信度，甚至由于上下文记忆机制的缺陷，根本无法意识到自己正在做什么。 针对这一问题，行业头部机构（如 OpenAI 和 Anthropic）所宣传的“沙盒环境（Sandbox）”实际上有着极高的技术与资源壁垒。所谓“允许智能体在沙盒内自由探索”，本质上是系统在底层持续维护并捕获智能体的运行状态，精细追踪其历史操作。这背后依赖于极其庞大的内存分配与高昂的计算预算，使得单纯的 100 万上下文窗口限制被工程化方案所弥补。对于缺乏顶级实验室资源与充裕资金预算的普通开发者或企业而言，要完全复现这种高保真的无界沙盒环境几乎是不可能的挑战。 --- ### [33:29](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2009s) 身份权限风险与受监管金融环境下的全生命周期治理 智能体的权限与身份（Identity）绑定带来了极其严峻的安全隐患。一旦智能体被赋予过高的操作权限，可能导致毁灭性后果——例如近期行业内曝出的自动化系统误删全部生产环境数据的事故。现场工程师分享了切身体验：在排查 AWS 故障时，出于省时考虑让智能体去调取 CloudWatch 日志排查原因；尽管智能体顺利完成了日志分析，但他随后感到后怕，因为该智能体继承了他本人的 IAM 身份凭证，这意味着它同样拥有删除生产数据库的全部权限，一旦发生误操作，所有责任都将直接记在工程师个人名下。 针对受监管行业（如银行业）中的合规治理，治理机制绝不能是事后补救，而必须从最初的设计阶段（Design Phase）深度嵌入整个系统架构中： ```mermaid flowchart TD accTitle: 受监管环境下的智能体分级治理与放权流程 subgraph Phase1["1. 试点设计阶段"] A["设计嵌入治理规则"] --> B["100% 人类在环 (HITL) 监督"] end subgraph Phase2["2. 评估与置信度验证"] B --> C{"指标是否达标且<br/>置信度 ≥ 99.999%?"} C -- "是" --> D["逐步放权: 自动执行特定子任务"] C -- "否" --> E["触发回退: 人工接管审核"] end subgraph Phase3["3. 审计与透明协作"] D --> F["持续全链路监控"] E --> G["呈现完整证据链与推理轨迹 (Traces)"] F --> G end ``` - **设计即合规**：在多智能体或单智能体工作流设计之初，就严格界定不同决策环节的自主权级别。 - **极端保守策略**：即便模型在统计层面达到 99.999% 的极高准确率，在试点初期依然维持 100% 的“人类在环（Human-in-the-loop）”全量监控。 - **渐进式放权与可解释性**：随着业务团队建立信任，系统根据量化指标逐步将特定子任务移交给 AI 独立完成；一旦指标跌破预设阈值，即刻回退至人工审批路径，并向人类专家完整展示推理轨迹（Traces）与决策证据链，让 AI 充当高效的起草者而非绝对黑盒。 --- ### [36:07](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2167s) 前沿展望：世界模型商品化与数百万现实交易智能体 谈及智能体领域的下一波技术浪潮，嘉宾们聚焦于架构创新与社会级影响： - **世界模型（World Models）的即插即用与商品化**：如同过去几年基础大语言模型（Frontier LLMs）迅速演变为通用基础设施一样，未来针对不同物理与虚拟场景的世界模型也将逐步走向商品化。开发者将能够像调用大模型一样，根据垂直业务用例随插即用各种世界模型。 - **数百万自主经济智能体的诞生**：作为一名坚定的“智能体最大主义者（Agent Maximalist）”，讲者预测未来两年内现实世界中将涌现出数百万个能够独立发起真实交易的智能体。这些智能体将拥有独立身份，具备在银行开户、持有资产乃至在大宗商品市场撮合小额经纪交易的能力。然而，当前的商业社会与普通人类甚至尚未做好应对这一冲击的准备——“无论是商业基础设施，还是普通人的日常电子邮件收件箱，都还没有准备好面对智能体洪流”。 --- ### [38:16](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2296s) 闭环评测工程、自我改进与传统实体产业的数字化蓝海 在金融机构内部，类似摩根大通（JP Morgan Chase）等机构正在探索“个人数字助理（Personal Assistant）”架构，允许智能体代理用户阅读并回复邮件。然而，如何防止智能体在特定场景下“失控走向极端（go rogue）”并输出违规内容，仍高度依赖严格的治理控制。除了自主助理，研发重心正在全面转向**测试评估闭环工程（Harness Loop Engineering）**：探索智能体如何在完成每个任务后自我评估成败、构建反思与自我改进闭环（Self-improvement Loop），并由智能体框架本身原生提供透明的思考与推理链路，而无需额外拼凑复杂的外部审计组件。 与此同时，智能体最具潜力的变革蓝海恰恰存在于那些“连普通互联网都尚未充分渗透”的传统实体产业中： - **传统产业的巨大真空**：在供应链、重工业制造以及基层医疗诊所中，大量业务流程依然停留在传统的“纸和笔（Paper and Pen）”及初级人力审批阶段。 - **全球技术认知断层**：当前世界呈现出严重的技术割裂——大约仅有 10% 的人群深谙前沿云端 AI 与智能体的演进，其余 90% 的实体世界仍对这场变革毫不知情。 - **实体世界隐性低效的消除**：在物流、建筑和供应链管理中，普遍存在的延误往往并非单纯的人力成本或工时瓶颈，而是因为“某个审批单在某人的案头卡了数天甚至数周”或是缺乏全局跨单合并路线的算法协同。智能体的深度介入并非取代人类价值，而是消除这些被遗忘在角落的隐性低效，让人类得以专注于更高维度的智能决策。 ### [42:11](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2531s) 医疗场景中的 Agent 落地与人机协同价值 探讨 AI Agent 在现实垂直行业中的落地时，讲者分享了此前在 Oracle Health 医疗部门的实际交付经验。当时团队为医生和护士上线了临床辅助 Agent，这些智能体不仅能用于与患者沟通交流，还能处理护士端极其繁琐的医嘱处理（Order Processing）流程。在传统流程中，护士必须手动输入每一条医嘱，并逐一核对药物是否与患者的历史病史、药物过敏或现有处方产生冲突；而引入 Agent 后，系统会在后台自动完成全面的病史交叉核验，并直接呈现结构化的医嘱单，护士只需要进行最终的确认与批准。 这种将 Agent 作为“预处理和交叉核验助手”的人机协同模式带来了立竿见影的效率提升。据介绍，这一应用为每位患者节省了多达 2 到 3 个小时的时间，在高度繁重的临床环境中释放了巨大的临床价值。尽管台上有嘉宾打趣笑道“我可不敢去看完全依赖 Agent 的医生”，但现场讨论明确指出，Agent 在专业场景中的真正定位并不是替代专业人士，而是通过消除重复低效的人工比对环节，大幅压缩流程耗时。 - **Oracle Health 临床应用案例**：部署用于患者沟通和医嘱处理的临床智能体。 - **自动化安全校验**：Agent 自动将处方药物与患者既往病史、禁忌症进行交叉核对，将护士的角色从“繁琐手工排查”转变为“一键审核确认”。 - **量化效益**：在特定工作流中平均为单名患者的处理流程节省 2 至 3 小时。 --- ### [43:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2584s) 金融交易 Agent 的分歧：严监管合规 vs 开源与初创生态的野蛮生长 ```mermaid flowchart LR accTitle: 金融与高风险领域 Agent 发展的两条路径 subgraph Regulatory["传统高合规路径 (Enterprise)"] A["高风险 / 强监管领域<br/>(金融资产、医疗健康)"] --> B["严苛合规与确定性要求<br/>(需 100% 可验证)"] B --> C["大机构推进极其审慎缓慢"] end subgraph OpenSource["开源与初创路径 (Grassroots)"] D["开源 Agent 框架普及"] --> E["Agent 专属银行与独立数字身份"] E --> F["个人/初创团队自主接入金融交易"] end ``` 针对此前关于“Agent 自主执行金融交易”的讨论，Dhivya Nagasubramanian（U.S. Bank）提出了审慎的质疑。她直言，除非在技术上能够提供 100% 确定且可验证的正确性保障（"unless we have a verifiable way that like it is 100% right"），否则在未来几年内她对智能体真正自主进行交易持高度怀疑态度。与会者进一步达成共识：无论金融资产交易还是医疗健康，两者发展节奏相对迟缓的共同根源都在于强监管约束（Regulatory Controls）以及极高的错误代价——“涉及的风险利益实在太重大了（there's a lot at stake）”。 然而，现场也出现了激烈的观点交锋。有嘉宾反驳指出，虽然大型传统金融机构可能会因风控与合规而止步不前，但如今 Agent 生态的核心驱动力来自开源社区。事实上，市场上已经涌现出一批为 AI Agent 提供专属银行服务（Banking for AI Agents）的初创公司，并为智能体分发数字身份，使用户能够通过开源框架授权 Agent 以自己的身份在现实世界中行动。初创团队和个人开发者不会受制于传统监管惯性，他们会直接利用开源工具向前推进。讨论最后在关于“是否会因此催生一套全新的全球金融系统或比特币式颠覆”的欢笑与调侃中，为主持人宣布圆满结语画上了句号。 - **金融与医疗的双重阻力**：强监管、严格审计要求与零容错的风险代价，导致大机构在交易级自主 Agent 上推进缓慢。 - **开源与初创破局点**：开发者正在绕过传统流程，利用开源框架、Agent 代理身份以及新型“AI Agent 专属银行”基础设施快速尝试交易自治。 ## Notable quotes > [29:12](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=1752s) In my personal life I definitely don't put any gates on my agents. I'm like here's my debit card. Enjoy. Please get the groceries to the front door... very much so a different story when I'm talking to the military. — Jason Bonvie > [43:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2584s) unless we have a verifiable way that like it is 100% right — Dhivya Nagasubramanian > [43:04](https://www.youtube.com/watch?v=DvU6JVlQF0o&t=2584s) there's a lot at stake ## Mentioned Ai4, Ashok Prakash, Dhivya Nagasubramanian, Seetaram Rao Rayarao, João Moura, Darrell Etherington, Jason Bonvie, Apple, U.S. Bank, JP Morgan Chase, CrewAI, The Angle, Siri, Apple Music, Apple AI Platform, Claude, Claude Sonnet, UiPath, AWS, CloudWatch, IAM, OpenAI, Anthropic, NemoClaw, Google, Open Knowledge Document Format (OKDF), Oracle Health ## Video description Ashok Prakash - Staff ML Engineer, Apple Dhivya Nagasubramanian - VP, AI Transformation and Innovation, U.S. Bank Seetaram Rao Rayarao - VP, Senior Lead Software Engineer, JP Morgan Chase João Moura - Founder & CEO, CrewAI Darrell Etherington - Editor, The Angle Jason Bonvie - Staff Forward Deployed Engineer, CrewAI Beneath every capable AI agent lies a stack of model architectures and training strategies that determine how it reasons, plans, and adapts. This panel will dive into the core “thinking layers” of agents—from large language models and world models to planning modules, memory systems, and tool-use capabilities. Panelists will discuss the latest approaches to pre-training and fine-tuning, the role of reinforcement learning and simulation environments, and emerging architectures for multi-step reasoning and long-horizon planning. The conversation will highlight both the technical trade-offs and the open research questions: how do we balance generality vs. specialization, how do we evaluate reasoning quality, and what’s needed to make agents robust outside of demos? Attendees will gain a deeper understanding of the model and training foundations that underpin effective agentic systems—and where research is headed next. [Source](https://www.youtube.com/watch?v=DvU6JVlQF0o)

[29:12] Jason Bonvie: In my personal life I definitely don't put any gates on my agents. I'm like here's my debit card. Enjoy. Please get the groceries to the front door... very much so a different story when I'm talking to the military. [43:04] Dhivya Nagasubramanian: unless we have a verifiable way that like it is 100% right [43:04] there's a lot at stake

[0:04] Darrell Etherington: didn't do any preparation [1:42] Ashok Prakash: press one, press two [4:16] Jason Bonvie: running rampantly [11:04] Jason Bonvie: token maxing [11:04] Darrell Etherington: vibe coded [29:12] Darrell Etherington: caging the animal [29:12] Jason Bonvie: In my personal life I definitely don't put any gates on my agents. I'm like, here's my debit card. Enjoy. Please get the groceries to the front door. Very much so a different story when, you know, I'm talking to, you know, the military, right? It's like, you know, how do I make sure this thing doesn't see the document I sent it? [31:37] chaos theory [31:37] three-body problem [36:07] thousand flowers bloom [43:04] Dhivya Nagasubramanian: really surprised if that happens in the next couple of years unless we have a verifiable way that like it is 100% right [43:04] there's a lot at stake [43:04] build an entire new global financial system

[0:04] Uh we got a great panel here today and we're going  to talk about something that is very interesting [0:09] to me and I'm sure very interesting to all of you  which is how you build an agentic brain and what [0:15] you should consider when doing that. I think uh  I'm super interested to find this out because I [0:19] didn't do any preparation and I'm going to find  out live with all of you. Uh but I'm going to have [0:23] everybody introduce themselves one by one and give  a just a quick background on what it is they do. [0:29] Um let's start with you Diva. Hey everyone. Uh I'm  Diva Naga Subdudman. I'm VP of AI transformation [0:38] and innovation uh working at US Bank. Hey,  hello everyone. Thank you for joining this panel [0:45] discussion. My name is Sitharam Rayaro. I work  in JP Morgan Chase as a senior lead engineer and [0:51] uh my primary work area is a engineering. So we  enable uh the uh engineering agentic framework [0:58] for like 200k employees. So that's a background.  Hello everyone. My name is uh Jason Bonvi. I'm [1:06] a forward deployed engineer at Crew AI. Uh so  really living on the uh deployment side of the [1:12] house. Uh I've probably been responsible  for deploying over a thousand agents into [1:16] production at large enterprises at this point.  Hey everyone, this is Ashok. I'm uh from Apple. [1:23] I work in Apple AI platform. Um so one of the  Apple services um kind of you know operates [1:29] like Siri and Apple Music. So all the AI services  which runs on top. Um so my team actually powers [1:35] the platform behind that. So I work as a staff  engineer there. Great. Uh I think maybe it's [1:42] useful to start off this discussion with just kind  of some level set on on definitions. So what is [1:48] an agentic brain? How is it distinct distinct and  different from kind of like what we would think of [1:53] as non-aggentic AI? Uh what goes into that? What  are the architecture considerations? Can I guess? [2:02] Hey, sure. Um so um you know so agents versus  non-agents as um right now people are moving into [2:11] agentic systems. How I actually would distinguish  is any intelligent decisions which you need to [2:19] make which is basically not static you know that's  where you actually won't have a agent system right [2:26] so wherever you today actually have a very static  workflow as simple as for example let's say you [2:33] have voice um calling pre previously people used  to have IVR systems in the IVR systems there is [2:40] something called a DAC flow which people used to  have where if this happens then you know press [2:44] one two everyone probably would have gone through  that. So today you don't have to do that right. So [2:49] you basically speak to an agent it understands  your intent it then classifies and redirects [2:56] to the right person or it talks by itself. So  that decision is an intelligent decision and [3:03] essentially wherever you think you need to have  intelligent decisions involved that's basically [3:08] where you want agents. Great. Anyone else have  anything to add? Yeah, I can add. So to me, [3:16] um, an agent versus non-aggent, um, a non-agent  is like anything you can traditionally automate [3:22] using like a regular rule-based engine is like  doesn't belong to agentech. But anywhere there's [3:28] a dynamic workflow with a bounded policies and  bounded guard rails that need to be incorporated [3:34] where there's a little bit of dynamism, that's  where we start thinking about incorporating agents [3:38] with the workflow. Yeah, maybe I can add more. So  as like panelist fellow panelist like now before [3:47] uh agents come into the picture right there.  So what we see is like uh the workflows are [3:52] predefined. You don't have a way to change it.  But with the agentic workflows now you have the [3:58] react loop like where it will reason first. it  will identify what task it needs to be completed [4:04] then it will do the observe and it will uh respond  act on it basically it's a kind of a cycle so this [4:10] kind of feature was not available before agent  thing so that I think that is a power so then [4:16] how do you look at uh standing one up like if  you're setting out of the task of I'm going to [4:21] build an agent what are your considerations like  what are your uh requirements and then what do [4:25] you do to evaluate what ingredients go into that  sure so um So I guess coming from the framework [4:32] side of the house, right? Why have an agent rather  than just you know submitting maybe an LLM call [4:38] um and the idea really is around agent harnesses  kind of gives you the ability to control how the [4:44] agents operate at a task level and then you can be  granular with the expertise and the context that [4:49] gets fed into those agents. Uh and then ultimately  uh this kind of lends itself to to just being [4:55] better at more specialized tasks and having a  lot more controls and you can do things like [4:59] uh state management or context management uh to  make sure that these LLM calls just aren't running [5:04] rampidly. Um and and I think this kind of lends  itself to uh how do we actually create effective [5:12] longunning tasks with these agents? And I and  I think you need harnesses of of any sort um to [5:18] kind of control how these LLMs are are giving you  outputs and like so is the harness then the key [5:25] ingredient and kind of like when you're setting  up but you also need to consider what models to [5:29] use right at the underlying for the intelligence  component. So how do you make those evaluations? [5:33] Yeah, I mean personally, you know, as I'm I'm as  I kind of go out and evaluate people's use cases, [5:39] most of the time at this point, the frontier model  question like which to pick really comes down to [5:44] cost at this point. Um, so and so there's  kind of like the cost versus the cognitive [5:49] uh output that is required for any given task. Um,  with that being said, it's it's usually for things [5:55] like summarization or just data retrieval. You  know, we can do very cheap things with that. uh [6:01] the more autonomy that you need to give your agent  or the more responsibility that it ultimately has [6:05] for decision, that's kind of when we want to  probably ramp up some of that cognitive ability [6:09] and and then you know you're picking more and  more expensive frontier models in that respect. [6:15] Yeah, I can I can add from my experience is  that in a given workflow, let's say there are [6:20] like five or six steps and that let's say it's  a multi- aent orchestration of the workflow. [6:26] Um it is pretty important that um we we understand  what the task is and the goal is for the given [6:33] step and understand if that cognitive ability  is required. Um to use a large models um and use [6:43] the right model at the right step or the right for  the right agent for the right purpose is something [6:47] where you can optimize on the cost and how do you  how does that decision-m process go? Do you work [6:52] with stakeholders like how do you kind of optimize  in in a specific case right with a specific agent? [6:58] Do you do like requirements gathering and  then evaluate the intelligence against those [7:04] requirements or right? Uh so if as as he as um Joe  pointed out if it's um a simple summarization task [7:12] you don't need a very large model let's say if  it's a coding task you know there are specialized [7:16] agent a specialized large language models  specialized on coding for so based on the task [7:21] that you're trying to do uh if the summarization  can be done using a much cheaper models we would [7:27] stick on to that and that decision not necessarily  stays with the stakeholder but it would be on the [7:33] architects and the technologist ologist who  builds those solution. Yeah, I could maybe [7:40] think of this as multiple layers. So, um just to  simplify, I I kind of think this has like three [7:47] layers of complexity to deal with. One is the  actual agents isolation itself. So for example, [7:53] you can work on like frontier models but again  if it's uh something like third party something [7:59] like open AAI or in your company are using cloud  models then you actually have to take your data [8:04] outside. So then it really depends on like that  data isolation. So that's your first layer like [8:09] for example do you want to isolate your data to  get like kind of a sovereign model or it has to go [8:15] out it's is it fine to go out. So that's basically  data isolation layer. Second is basically your [8:20] model decision and complexity layer where which I  think is uh you you kind of need a bigger model or [8:26] a smaller model or you are thinking about latency  frequency of your inputs and outputs. So you [8:33] think about that latency aspect. Sometimes bigger  models again can take a long time, right? Smaller [8:39] smaller models can actually answer very quickly.  But again that you have to [clears throat] do [8:43] the cost trade-off as well as as the intelligence  trade-off because smaller models can be sometimes [8:48] dumber compared to your frontier models. So that  balance is basically your model complexity. So [8:53] there's data isolation complexity, there's model  complexity, then your business complexity, right? [8:59] That's this is my three layer if I understand this  correctly. Right? So the the business complexity [9:04] is where you basically want to do workflows, you  want to do like IVA calls like you mentioned or or [9:10] you want to have as simple as you know a chatbot  like whatever that business you're trying to [9:15] implement. You kind of want to have these three  things kind of understood really well to be able [9:20] to you know incorporate them at the right layer  and those cascade like this defines the set of [9:25] the the layer above right each one. So as as let's  say you take a simple business use case then you [9:31] try to understand the use case really well then  then once you again within business use case there [9:36] is more grandless right for example background  agents um you know there is functional agents so [9:41] you understand the grand granularity of like where  it sits. So you once you understand the business [9:46] complexity that defines your what's your model  complexity looks like then from models complexity [9:51] that defines your data isolation your network  isolation and so on like your state isolation [9:56] and so on. So every every single layer defines  each other. So it kind of opens up that nice [10:01] framework for you once you you know deal with each  of that. But again there's so many options as you [10:07] get right. So that that's where you really really  need an architect to decide like what should I do [10:13] what path should I go right that's where I really  like a crewa this is drag and drop I love that UI [10:19] and I play around I also SDK right so I love that  and yeah as you know Ashuk mentioned right we have [10:26] we need to have the different kinds of models  that we need some are good at reasoning some [10:30] are good at the coding and all this stuff so what  we did is like uh when user we give the option to [10:35] users to select the workflow and they can choose  which model they want to use it and what would be [10:41] the system prompts and everything. So that's the  most customized experience that we can give and [10:46] they can define whole workflow. Again it's based  on the business use case also. is I I'm interested [10:52] in the I think you've alluded to it when you talk  about the architect ingredient in all of this [10:56] like as the speed and um sophistication of these  things evolves so quickly like a lot of the stuff [11:04] the decisions you've made in the architecture are  out of date almost instantly right like it's like [11:09] oh this is no longer the right mix of models  and techniques and um specialties so how do [11:15] you deal with that with the pace of the technology  coming down the the pike yeah that's That's a good [11:20] question and and I don't think there's a really  great answer for that um you know at scale at the [11:25] moment. I think you kind of handle it very much  so on a case- by case basis. Um and I guess this [11:30] is also why leveraging a an an agentic harness is  useful. It makes it very easy to kind of replace [11:36] what models are are kind of running there and then  doing tests against uh one model or another. Um, [11:43] transparently, I don't see too many people kind  of winding back and changing models unless there's [11:49] essentially degradation of the support of  those models for the most instance. Um, [11:53] once something hits production, it's usually  tested enough to to to feel confident in it or [11:57] you can do things like cost optimization uh later  on down the line. But, um, I don't hear too much [12:03] of it from customers at the moment. I think where  people are spending a lot more time is is not kind [12:09] of on performance or trying to improve it, but  rather token optimization. Um, and that's that's [12:13] really kind of the heart of everything right now.  The conversation is turned to ROI. Yeah. Uh, it's [12:19] big topic conversation here at this conference  specifically, but I think yeah, just broadly. Um, [12:24] it was funny to see how fast that move how fast we  moved from a token maxing, let the tokens max to [12:29] a wait a minute. No, don't do that actually. Um,  but yeah, I hear you on that because I'm thinking [12:36] about even like personal experiences with uh  my very fragile, brittle, vibe coded things. [12:41] And you don't go back and adjust like I've made  some podcast transcription apps for personal use, [12:46] right? Like I don't go back and adjust it unless  I get the email from assembly AI that's like, hey, [12:51] we're deprecating this model or like you know this  is going away or we're introducing something new [12:55] and maybe it has these features, but otherwise I'm  content to leave it alone and kind of just use it [13:00] in production, right? So that's something  you see generally across. Yeah. I mean I I [13:03] would liken it to just about any automation,  right? Like if you have a UiPath, you know, [13:07] RPA bot that's been working for five years,  you probably don't go back and visit it too [13:11] often unless uh unless it suddenly stops working  and then it's maybe worth another look. Yeah, [13:15] that's that's pretty much like uh you know that's  a model for example this clot 4.6 then you got [13:22] 4.7 then you got 4.8 like now fable and mythos  right. So the intelligence of these actually [13:29] these models did go up in benchmarks right you  it has really substantial improvements in all of [13:35] these different benchmarks but for certain tasks  um I really didn't see certain differences when I [13:41] apply either fable or 4.6 certain tasks it it did  really good job in that specific task so the com [13:47] so depending on the complexity of that uh specific  task the models really did well do I want to pay [13:53] $50 for fable for that task I can just use 4.6 and  six for that which is actually much cheaper or or [13:59] sonnet 5 which is f further cheaper than that.  So basically if your if your task is really well [14:05] structured as an architect as an engineer or  as anyone who is essentially understanding a [14:11] problem and breaking them into smaller you know  subsolutions that you can solve it. Um you want [14:16] to structurize the problem such that any models  in the future should be able to just quickly [14:20] understand it. So even the probably the least uh  intelligent model today which exists might still [14:25] be able to solve it because you have structured it  really well. I think that matters more. You also I [14:30] think maybe there's a part of that where you talk  about the success of model increases. This is like [14:35] maybe a unique time in technological development  where it's not a linear gain across each right [14:40] like sometimes in specific function areas it's  regression maybe model to model. So there's also [14:46] that risk. uh I I specifically think some of this  regressions in my personal opinion don't hold me [14:53] but um is due to guard rails um there are systems  that actually they keep training um I think maybe [15:02] somewhere around five to six trillion parameter  models are right now out there um so you know you [15:07] had like a trillion dollar parameter model before  which are really which really did an amazing job [15:12] in some of the tasks now we have five or  six trillion dollar parameter models they [15:16] are they are much better but again what happens  is they are also creating some loopholes cyber [15:21] cyber attacks possibility so if you just take  them out and give it to like everyone even if [15:27] uh user doesn't know that they are actually you  know uh doing it the models have the capability to [15:33] go do something which user might not be aware of  sometimes they might just simply go run let's say [15:39] rm- rf command on like let's remove a particular  folder I'll create a temp temp folder you will [15:44] just say oh yeah temp folder go ahead but then it  might just create create some you know some sort [15:49] of shell script which probably is doing something  malicious which you don't understand. So there are [15:54] things like this possible. So then what happens  is you're trying to do guardrails on top of these [15:59] models and that also you know prohibits them from  doing some tasks really well which the previous [16:05] model really did well. So that's kind of why I  feel this is kind of going down in some tasks but [16:10] in some tasks they're really amazing and I I don't  want to specifically tell like what tasks they are [16:16] given some some some of some some of them I've  worked in environment but but I can really say [16:20] that with with guardrails some of them are going  down but some some tasks are really amazing. Yeah, [16:26] I think the uh another thing you bring up there is  kind of like the impact of cascading complexities [16:33] when you're applying harnesses and things to  things that are already themselves harnessed [16:38] their guard rails by their original creators,  right? And like how does that create interactions [16:43] that are unpredictable and unusual and possibly  degrade service over time? And like I guess you [16:48] just look for them in production and then kind of  address them as they come out or how do you kind [16:52] of think about the harness's impact on the models  especially as the models change capability? Yeah. [17:00] As I'm developing uh agentic workflows, I think um  I really look at the guardrails. I want to build [17:06] a system that can connect to everything that I  would like it to connect to initially. Um and then [17:12] usually organizations kind of have policies that  live around how they want these agentic systems [17:17] to work, right? So um tiers of authentication.  How is this agent uh authenticated? You know, [17:23] does the user who's interfacing with the agent  change uh the accessibility of the tools that that [17:28] agent might leverage? Um how do I control token  limits or you know the validity of a task output? [17:33] Um, and so these are all kind of different  areas in terms of like how how they're actually [17:38] implemented. Um, but but guardrails can work  really quite simply on a task level that that are [17:43] operationally very sound where an agent performs a  task first, right? We don't know whether that task [17:48] was performed safely or whether the output  will be valid. Um, but then you can have a [17:52] a simple semantic or a Python guardrail that  lives at the end of that which basically asks, [17:56] you know, a boolean expression, you know, did are  these criteria true? Is this safe? you know, does [18:01] this look like an exploit of some sort as somebody  abusing this? And then, you know, if true, great, [18:06] let's proceed. If false, let's recontextualize  this. Let's reprocess this task or exit the loop [18:11] or do something along those lines. Um, I can add  a little bit of flavor there from the financial [18:17] background. So, in finance industry, it's really  really critical that when you hand over something [18:22] to the agent in a workflow, it is important that  it is doing the intended job towards its goal. [18:28] And one thing that I'm really chasing is the  verifiability of those task as he pointed out [18:34] like okay this agent is being given this goal  and this list of task is it reasoning right [18:41] who's validating that today we validate the input  we validate the output um two agents uh having the [18:49] same goal same task can have two completely  different reasoning and end up with the same [18:53] results luckily first time second time but not  the third time or the 40th time and it runs in [18:59] production. So who verifies the reasonability? So  that's something that we are trying to consciously [19:06] embed into the system as he said like there are  multiple ways you can validate that. One is like a [19:12] deterministic process of like okay were the right  tools called in the right as right way and was the [19:20] right input went into that tool. Did the output  came out true? was the reasoning trace adheres to [19:26] what it was intended to. That's one way. Another  way is like having a simple deterministic Python [19:30] function to validate uh the overall um traces and  then letting the action stop right there instead [19:39] of it cascading it into multiple agents quietly uh  without us noticing. So that's something that we [19:46] consciously built into our system today. Although  it's agents, although it is trying to do things [19:51] dynamically, still there are controls in place  that like it does not have much wiggle room but [19:56] narrowly follow that path to the intended goal  final goal. So the rules based is still a very [20:01] important piece possibly the most important.  [laughter] Absolutely. As much as we leverage [20:06] these capabilities of these large language models,  which is great, but uh having that verifiable [20:12] layer embedded into that agent, not just on  the orchestrator agent, but to the individual [20:17] agents that carry out those tasks is really  critical. And then this also comes back to cost, [20:22] right? Because in theory with a system like that,  you could do a volume approach and just say, [20:27] well, we'll discard like almost as if you're  manufacturing semiconductors or something, [20:31] right? you're like, "Oh, there's going to be a  certain yield and that's fine cuz the good ones [20:35] will come through and that's worth the margin,  right?" But on this as we looked at token spend [20:40] and like costs, we're like, "Oh, we want more  success rate out and less dead end." It it comes [20:46] with a cost. Um it's nothing is free, right? It  comes with a cost, but um there are ways you can [20:53] optimize it. As I said, like deterministic  is not going to take a lot of cost per year. [20:58] you're going to have like multiple LLM judges.  Of course, that's going to have have a cost. And [21:03] even if you have an LLM judges of from different  families of models, uh can I rely on that? How [21:10] do I know? So, okay. Now, go ahead and build a  rubric. Okay. And if this I follow this rubric [21:16] and now if it's above this cutoff, I let it  go. If it is between this uh unknown zone, [21:23] I let a human. If it is below that, no, no, no,  no, no. Stop. let it go through the human the [21:28] regular route. So, so, so there are many ways um  many creative ways on how you can do it. Again, [21:35] if you're going to use LLM judge, multiple LM  judge, it's going to increase your token cost [21:39] for sure. I think I would like to add something.  [clears throat] So, here like as Ashoke mentioned [21:45] and Joe mentioned like uh like in the banking it's  a very strictly regulated environment, right? So, [21:52] we'll have the tenant based kind of uh setup. So  each tenant will have access to some certain kind [21:57] of a data sources, data sets available and uh  what models they can access and everything. So [22:02] they can they cannot basically cross a boundary.  So each one will have their own set of rules and [22:07] regulations and policies in in place. So coming  to this uh Diva's point like if you have the [22:13] deterministic workflows then you are losing the  autonomy right you don't you now you are telling [22:18] agent to do XYZ steps in in sequence whereas the  whole agentic harness and the loop concept is [22:27] like agent can reason and it will plan and it will  okay execute what tasks it has to execute and the [22:34] variable said that uh like there is no control  like Okay. Did it reason it very well? If two [22:41] models are given the same prompt, it will give  two different uh reasoning plan for you. How do [22:47] you test that? How do you validate that? There is  there is gap I think in the industry as a whole, [22:52] right? There is no particular like we are only  validating the final outcome but we never validate [22:57] what like how many tasks are executed and what is  the you know response uh all these things. Yeah, [23:04] that's yeah that's that's a funny one too, right?  I I think um you know when we're looking at [23:08] protecting agents and how they're how they're  performing um we always look at the outputs [23:14] um to see you know was was this right or wrong um  what agents are really good at is creating a plan [23:20] whether that plan is logical or not is kind of the  question and and there's not particularly a good [23:24] way of validating whether the plan that it came up  with was logical uh at all and I think that's kind [23:32] of an obstacle that we encounter you know when  when creating agentic workflows is validating that [23:38] um there's some sort of structure that the agent  follows um to come up with a plan that is within [23:43] reason uh of what the expected outputs are.  And this is why I think kind of creating very [23:48] specialized agents um kind of lends itself to  to to accomplishing that because then you can [23:54] be very fine-tuned with the context that you feed  it, what sort of capabilities it has, and you can [23:59] scope down the tools that it's leveraging to to  kind of limit the the the surface areas that that [24:05] agent could kind of work against. So you're saying  like it's still not decipherable in terms of how [24:10] it's reasoning, but it's it's reasoning from such  a limited set that you're like it can only be one [24:14] of however many combinations of decision. Yeah, I  would say yeah, the the smaller you scope it down, [24:19] I think the the uh more logical things become,  right? Like if we look at how um uh how how the [24:26] task degradation happens, it's really on the  longer horizon tasks and and why is that? Well, [24:32] the window the context windows have gotten  larger. Uh the reasoning has gotten better. Um [24:36] but its ability to perform tasks that take let's  say more than an hour has kind of remained about [24:41] the same for like the past you know 6 months. Um  and and the reason for that is is it's not really [24:46] you know a frontier model issue but rather I think  it's a a context management and state management [24:51] issue where it kind of loses sight of the original  objective and it goes through all these branches [24:55] of reasoning. Uh and so the more you can kind of  control that narrative for longer uh running tasks [25:01] the the better the outcomes will be. the um so  for controlling the reasoning about for an agent [25:08] um usually what the methods currently I'm at least  using or my team's using is knowledge bases so you [25:15] um so what what we can do is you can take a sing  an example business use case right so imagine [25:21] um you're an order inventory system so you  basically want to uh create an order agent [25:27] which can process the order and then you know to  say that the order is fulfilled or not then you [25:32] know send it over next to the next step once the  order seems to be fulfilled. If not, do the do the [25:37] necessary things in the specific things. If it's  an order agent, you basically just upload this [25:42] entire knowledge base about your company which is  specific to that order in this case. So you just [25:48] say that these are my entire uh you know systems  knowledge base on how order should be processed. [25:54] This is all human coded right because humans have  followed that process through and through. So [25:59] uh this is my you know fulfillment centers. This  is my places where my orders actually are picked [26:04] and packed and this is the you know this is the  you know customer ccentric this is how much how [26:08] how the customers are going to respond you know  so on right so there are different variables you [26:12] basically create this knowledge bases and then  feed this into the agent now when the agent is [26:17] trying to reason it'll basically use these  knowledge bases as hooks so that it doesn't [26:22] reason beyond this so you can actually then make  it more deterministic this way right so now you [26:28] then the outcome supposed to be just deterministic  in this case which I whether processing the order [26:33] or not. So there's only handful of outcomes which  can actually come out of that. So this makes the [26:38] the agent well bounded such that it want it's  only going to reason within that criteria. So [26:42] it it actually doesn't you know go outside this  bound. So that that's kind of what whatever model [26:47] you use you can use the front model it will still  work within that boundary which you just set which [26:52] is basically the knowledge base. So there are  there are evolving techniques and methods again [26:56] here. So knowledge base is currently being used.  I know there's a open knowledge document format [27:00] which is currently Google published which is  also getting more you know it's starting to get [27:05] famous now. Um there is a agent skills which also  starting to uh you know uh get you know get into [27:11] more um of wider usage. So there are things like  that again it's an evolving field as as as we all [27:17] know but the current state-of-the-art seems to be  as open knowledge document format or a knowledge [27:21] basis seems to be working well and hardness which  is basically how you would confirm that my agent [27:27] is giving me this uh you know for x plus y z is  my output and you have n numbers of combinations [27:33] of that you verify with all the combinations and  you have a a kind of a percentage that okay 99% [27:39] of my outputs are expected to be in this way is  my 99% achieved if if not then do something else [27:46] because something is wrong once 99% is achieved  send the agent into production you know that's [27:51] basically how it's happening right I think uh I  want to add on both of them so as Jan mentioned [27:57] that you know longunning agents like this harness  right always have this context like they they will [28:03] have like huge number of context and it will like  it is like uh searching a needle in the hack kind [28:09] of thing like you may the details right So how  we address that? We like as Ashuk mentioned we [28:16] have these rack pipelines that we have and in JP  Morgan like we have this interesting thing like we [28:20] have a centralized data mesh and uh customers  can come and plug in their uh data set into [28:27] the centralized platform that we built and uh it  will be ingest all these documents into the rack [28:33] pipeline and this is tenant based imagine like you  you are ingesting your own data sets and no one [28:38] can see that and you can query them you can using  the API interface you can call that API and find [28:43] out the details So with this it helps context uh  context engineering also right like you only get [28:50] the relevant data from this rack pipeline and  you don't you ignore the rest of the stuff and [28:54] also the skills as uh Ashuk mentioned like it's  a progressive disclosure and when you define a [29:01] skill it will only go with the description and it  will load as it needed. So this both both of these [29:06] techniques will really help in you know context  engineering. Yeah agree with him. So you talked [29:12] a lot about bounding basically, right? Like  this that's most of the conversation. So what [29:16] about kind of the opposite approach? Do you have  thoughts on that because that's like you know the [29:21] agi of it all. But like is there a world in which  that's the preferred option or path or is that [29:26] like still a science fiction kind of No, I mean  if you look at some of the most popular projects [29:32] that have come out for, you know, individuals like  uh Nemo Claw, right? Uh its popularity was that it [29:37] was an unbounded agent that could do whatever  you wanted. Um I think it's it's really about [29:44] you know from an enterprise perspective the issue  is always you know how do I stop agents from doing [29:48] what I don't want them to do. Um but the opposite  is very much so true right like it is very useful [29:53] to have a generalist who has everything that I  need who can reach out into the world. Um but [29:57] even in that case I I think you know the time is  spent in controlling like where where should we [30:01] put the walls uh even even when we want to give it  more and more autonomy. But as you kept talking, [30:07] I just I like my brain went to and maybe this  is the temptation and why it's such a a thing [30:12] that people still pursue is like every time is  it's just like oh and then if I just add this [30:17] other gate like it'll do that and you just want  to you're infinitely pursuing bounding bounding [30:22] bounding trying to like cage the animal and you're  like I just want to go the other direction. I just [30:26] want a super animal that can do all of this right?  I mean in my personal life I definitely don't put [30:31] any gates on my agents. I'm like here's my debit  card. Enjoy. please get the groceries to the front [30:36] door. Um, very much so a different story when you  know I'm talking to uh, you know, the military, [30:42] right? It's like, you know, how do I how do I  make sure this thing doesn't see the document [30:46] I sent it? Um, so yeah, I I think those are kind  of the the conversations there. And then I think [30:51] it it naturally depends on the use case, right?  You know, if you're if you're an HR uh chatbot, [30:56] it's like, yeah, I want to give it the degrees of  freedom to get any information on an individual, [31:00] but I want to make sure there's restrictions so  that it can't cross-pollinate. um versus something [31:04] like document processing where it's relatively  safe, you know, we we don't have to worry about [31:08] too much uh happening there. Yeah. So um on like  unbounded agents, right? So if you don't want to [31:14] bound the agent, um one simple things where I  I I kind of had this I'm I'm asking the agent [31:20] to do a certain task and it's it's doing that.  I'm asking it to create a particular folder and [31:26] the file in within that folder. Um after a 1  million context has actually you know kind of [31:32] exploded it's trying to compact the context  it kind of lost that information and started [31:37] to create the f you know any any folder it wants  any files it wants because it forgot that I told [31:42] it to specifically do that right now in such an  unbounded environment where it just does anything [31:48] it wants it just becomes chaos obviously it's  hard for me to follow simple human brain right [31:53] so similarly I'm hoping like every human would  probably not be able to follow what the particular [31:58] agent is doing in in in such an environment right  now when the agents uh you know keep doing that [32:04] there is a degradation which happens even for  the agent after the 1 millionth context the agent [32:10] uh does the third iteration it just lost where the  folder is it actually put because it doesn't even [32:16] know that anymore so it starts to redo the same  process again and then it starts to you know just [32:21] you know it becomes a chaotic environment if you  guys read chaos theory you would probably would [32:25] know or a threebody problem right so in in any  any in any way what happens is the agents is not [32:31] just in an unbounded envir environment it's not  trustworthy it's just that it doesn't even know [32:36] what it is doing because of the context memory  it doesn't have right now and this comes into [32:40] a certain limitations uh when openai or or or  anthropic says that okay we are actually doing [32:47] a sandbox environment and we are trying to create  an an agent and say do anything you want within [32:52] the sandbox what they're really doing is actually  still preserving the state of the agent they're [32:58] containing the state of the agent and telling what  the agent is doing what it did before. There's [33:02] still humongous amount of memory allocation being  provided that 1 million context window doesn't [33:07] exist anymore in those scenarios. So they have  a bigger budget of money to play with which as [33:12] simple uses we don't have. So when you try to go  do an unbounded experiment, you actually need to [33:18] kind of figure out a same way to you know u mimic  the same environment which these big companies [33:23] are doing which is typically hard for us to do  unless you have your own a lab setup you know [33:29] which we don't but [laughter] I think identity  plays a crucial role right who is executing that [33:34] agent for example recently like I hear like lot  of uh news came out like you know cloud went and [33:40] deleted all the production data right so like that  scares me right so I was playing around in like [33:46] uh I recently had a issue in AWS I want to go  and check why I'm lazy enough because agents [33:51] right so I asked agent why don't you go and check  what happened so it went to the cloud cloud watch [33:56] logs and it verified everything and it give me  this this happened but I was scared like that [34:01] has a privilege to go and delete any data also in  the production in in environment it came with my [34:07] identity now everyone will tag to me because  it goes with my identity so yeah identity plays [34:12] crucial role in Yeah, that's I mean I'm curious  too about especially in regulated environment, [34:17] right? Like how do you think about um doing  the governance like kind of is it it's upfront, [34:23] right? Like the government's happens before it  takes action. It's not like a retroactive thing [34:27] or how do you manage that? So the governance come  into is it's embedded in our design process. Um so [34:34] when we think about building a solution right at  the design phase we start thinking about how are [34:39] we going to govern this um agents if it's a multi-  aent or a single agent doesn't matter but we start [34:45] thinking about that right away and the level of  autonomy that we give for certain decisions even [34:50] within one workflow might differ um and usually  it is we are pretty sensitive and like pretty [34:58] conservative in a way that like Okay, even if we  know that for sure 99.999% it is doing its job [35:09] right still we take it through different phases  in the pilot phase we still have that human in [35:16] the loop for 100% monitor it and once the business  gets that confidence then we slowly start I mean [35:25] we we put this plan right in the design like okay  how are we going to hand off certain things to the [35:30] agent let it go versus where are things where  human should step in and like where should that [35:36] human AI collaboration should it still exist.  So we start with 100% human and AI together and [35:42] then like we slowly hand off certain activities  to complete AI based on certain metrics that we [35:48] continuously monitor and then uh if anything that  falls below that rubric that I mentioned it still [35:56] goes through the human route for validation  and all the evidences all the reasoning the [36:00] traces is still visible to the human. It still  saves time for a human but now they start with [36:07] something instead of starting from scratch. Uh  I think just um one topic that I would love to [36:13] hear all of your opinion on is kind of what do  you think is an exciting future area for agentic [36:18] development? What are you paying close attention  to? Uh different architectures, different whatever [36:24] anything I can take it. Um I'm really excited to  see about these different world models getting [36:32] commoditized just like how we have these frontier  models being built and being commoditized in the [36:38] last couple of years. Um so I'm really looking  forward to see how that space looks like where [36:44] we have different world models for different  uh use cases that we can plug and play just [36:51] like how we have LLMs today. I think it's super  exciting. It's interesting because the category [36:56] doesn't seem at all settled like like world model  doesn't mean the same as like transformer model [37:00] right because it's like oh so many different  architectures so many different approaches so [37:04] that part I wonder if that settles first and then  there's commodification or if it's like a thousand [37:09] flowers bloom and there's different a bunch of  different architecture approaches then those [37:13] each individually risomatically become their own  commodified component. Um I think uh a space that [37:21] really excites me right now is probably the idea  that um there will be a large I'm I'm a little bit [37:26] of an agent maximalist but I I think there will  be probably within the next two years um millions [37:32] of agents doing transactions in the real world.  Uh and the implications of that I don't think [37:36] are clear uh to to humans and and I think that's  like I don't think businesses are ready for that. [37:41] I don't think humans are ready for what that means  to their email inbox. Um but uh I'm really excited [37:48] for kind of like agents you know having their own  identity having human functions uh you know in [37:53] terms of autonomy of of banking and and uh owning  assets and and kind of you know what does that do [37:58] to a commodities market when every agent can uh  you know broker a small deal um and I think these [38:04] these will I've seen a lot of you know really  interesting substack uh kind of posts around these [38:10] um and I think that's a really exciting place  to to begin giving some attention Y I want to [38:16] give the chance for the banking crew to respond.  [laughter] Yeah, sorry. [snorts] So actually like [38:22] uh this is some innovation is already happening  in uh chase like you know personal assistance [38:28] right. So we are building kind of something  called a personal assistance where you can [38:32] uh agent will be answering behalf of you. It can  have access to your emails and it can reply by [38:38] on behalf of you and it's like mostly going to  work but is it's like a you know giving whole [38:43] access but what will happen if something agent is  go rogue and it gives wild response back to right [38:49] it's a so there should be a controls governance  controls and everything it's a banking is very [38:53] regulatory environment so it's not easy something  to do but uh another area that I'm looking forward [38:59] is the harness loop engineering and where uh how  the reasoning will be going to evolve like how it [39:06] is going to identify what exactly tasks are going  to be run and how how it is going to evaluate [39:12] after the each task whether it is done right or  not right these kind of things I think it will [39:17] be going to evaluate more I'm excited to see you  mean like um the self-improvement type stuff like [39:23] examining the results and just being like good  or bad and how can I improve in a loop improve [39:28] and how do we know okay two agents if we give the  same task how do you know what is the process it [39:34] it is thinking and maybe agentic framework will  give that in internally like we don't have to [39:41] implement anything external like if it can able to  give that capability then it'll be actually a good [39:46] thing yeah few places I'm very excited about  is u places where um even internet that just [39:55] just a normal internet was not able to penetrate  like supply chain manufacturing you know there [40:00] are places which actually um people still use  paper paper and pen to do things. Um I I think [40:07] there's a huge potential in where where you can  actually have autonomous agents there and a lot [40:12] of people are starting to see value in that  right now and basically starting to you know [40:16] have a lot of startups there coming up and then  helping out uh go into that even healthcare for [40:21] example there's still a lot of [snorts] um clinics  hospitals still use paper and pen and probably HR [40:27] is something they use but they're not really up  to the advanced capabilities what agents can do [40:31] right now. So it feels like there's this there's  a section of people who actually is like all cloud [40:37] AI and there's like you know probably very you  know you know very excited about the AI future [40:42] but there's section of the world who actually  doesn't know anything what's happening on the [40:45] side of the world and I I'm more excited about  you know that section basically being closing [40:51] the gap or that's a huge gap right now I feel  maybe just 10% of the people in the world really [40:57] knows about what's happening maybe the rest of  the 90% you know stay as they don't understand [41:02] and what's have going on. I really hope that  that you know that gaps kind of closed and [41:07] um you know a lot of things um you know kind of  become more productive as agents are involved [41:12] doesn't mean again uh there's a less human value  there's actually more human value as you have more [41:18] intelligent tasks to be made and done by the human  itself. So yeah let's see how the future looks [41:23] like in that case I guess but I'm very excited  for that. Yeah, that reminds me of like the AI [41:28] and construction and AI and like the supply chain  is another great one. Logistics, right? Like those [41:33] places are where you have real tangible benefit in  the in the physical world. Largely I wouldn't say [41:39] unexplored because people are doing things there,  but largely like uncapitalized in production, [41:43] right? let's say. So I think that could be super  interesting cuz there's so much inefficiency [41:48] there and it's not inefficiency related to labor  costs or like people hours like it's inefficiency [41:53] just like stuff left on the table because we  don't know that we could route these shipments [41:58] together or whatever it is right so that is  a very exciting area. Yeah, like there are a [42:02] lot of things which just get stuck because the  someone has not approved that particular order, [42:06] you know. So that just takes weeks and days right  now at this point. So maybe with agents you can [42:11] solve those problems. As as simple as I know that  I used to work for Oracle health before and was [42:17] part of the healthcare group where uh we shipped  um uh clinical agents for doctors and nurses where [42:24] they can use agents to converse with patients or  even as simple as order processing for the nurses. [42:30] So they can you know see the order right away. So  they don't have to enter and and check against if [42:35] this particular medication has issues with that  patient's history and all that. They don't have [42:39] to individually manually check that. The agent  would have done that for them and they just show [42:42] the order right away and the nurse just need to  approve it. So there are there are things which [42:47] saves like 2 hours 3 hours of each patient time in  that. So feels like there's a lot of value there [42:52] and probably a lot of values in other places  as well. I don't go to the doctor who relays [42:57] on agent. U Divia, did you have something else  to add? I I wanted to maybe it was earlier. No, [43:04] I just wanted to comment quickly on Jason's point  on the transactions. Agents making transaction. I [43:09] would be really surprised if that happens in the  next couple of years unless we have a verifiable [43:14] way that like it is 100% right. Well, I think two  of those like the the the transactions and like [43:21] asset ownership and that and then also healthcare  like they're both there's reasons why things have [43:26] moved much slower there, right? And it is a  lot of the the regulation controls. There's a [43:30] lot at stake. Yeah. And there's a lot at stake.  Yeah. My my push back on that comment would be [43:36] uh probably yeah maybe there's not going to be  big companies who are enabling it, but most of [43:41] what agents are doing today are coming from open  source libraries and and people can just do that. [43:46] Um, so it's like, yeah, maybe it's kind of frowned  upon, but there's already startups that do banking [43:50] for AI agents and can provide identities that,  you know, can adopt my own identity and I can [43:55] release that into the world using some open source  framework. And I think people will just do it and [43:59] startups will do it and people will do it. You can  just build an entire new global financial system, [44:05] [laughter] Bitcoin. Uh, all right. All right.  Well, I think that's a good place to end it.