Vertex Slashes Trial Database Setup to Five Days in AI Push

Vertex发力AI：将试验数据库搭建时间压缩至五天 企业AI的ROI取决于压缩周期时间和扩展组织能力，而非将节省的员工工时视作虚荣指标。 企业领袖阐述了部署AI智能体的具体战略路线图，详细介绍了将周期时间从数周压缩至数天的实际落地案例。专家小组探讨了如何计算真正的ROI，并得出结论：单纯的“节省时间”若不与经过验证的成果、扩展的能力或降低风险挂钩，就是一个具有误导性的指标。他们还指出了软件瓶颈向上游业务需求转移的运营转变，并概述了健全的企业行动指南，涵盖安全的LLM架构、跨职能AI治理委员会以及多层技术栈。 企业AI落地需要同时兼顾宏观战略规划与细节执行，才能实现有意义的ROI。 (0:08) Vertex Pharmaceuticals采用以个人智能体框架、针对性业务需求以及规模化提升运营生产力为核心的三管齐下AI战略。 (0:08) 部署引入人在回路（human-in-the-loop）的AI智能体使Vertex的临床试验数据库搭建时间从14周缩短至5天。 (0:08) Artisan Agility利用AI智能体模拟团队动态和快速绩效干预，将极速测试周期比作蜉蝣遗传学研究。 (4:51) Toptal采用基于现有采纳度、基线数据和明确的KPI责任归属的三项试金石测试，来评估专用AI应用。 (4:51)

Deploying enterprise AI agents requires redesigning core workflows around validated outcomes and cycle-time compression rather than isolated individual time savings. According to industry leaders, scalable value depends on end-to-end process changes, cross-functional risk governance, and addressing upstream organizational bottlenecks. Shifting ROI from Time Saved to Validated Outcomes (12:01) Panelists caution that tracking employee 'time saved' acts as a vanity metric unless an organization deliberately converts that reclaimed capacity into measurable business output. Jim Schiel stresses that true ROI originates from enabling previously impossible capabilities and accelerating cycle times. Marcus Spillane notes that internal surveys at Howard Hughes showed off-the-shelf AI tools increased employee capacity by 20%, but realizing financial value required actively restructuring workflows. Jeff Mills adds that time savings translate into EBITDA improvements only when paired with quality, defining ROI around 'validated outcomes' that weigh total execution costs—including inference, tokens, and human evaluation—against actual delivered value. In exploratory contexts, Mills notes Toptal sees roughly a 1-in-10 success rate, whereas automating core, baseline-backed workflows with measurable KPIs yields significantly higher certainty. Compressing Multi-Fold Cycle Times and Work Volumes Rather than seeking incremental optimizations, enterprise leaders prioritize use cases that deliver orders-of-magnitude reductions in project timelines and work volumes. Agam Upadhyay explains that Vertex Pharmaceuticals evaluates AI agents by their capacity to sharply compress cycle times, citing a reduction in clinical trial database setup from 14 weeks down to 5 days using human-in-the-loop agents. Additionally, Vertex used AI authoring to generate 60% of the initial draft for a regulatory filing that historically takes 6 to 9 months and hundreds of contributors. Operational responsiveness drives commercial impact across other sectors as well. Toptal compressed request-for-proposal response times from 5 days to 5 hours, and inbound lead response latency dropped from 1–2 hours to 1–2 minutes, which directly bolsters top-line revenue. At Howard Hughes, an agentic-first workflow automates real estate leasing from the initial letter of intent through contract drafting, signing, and enterprise resource planning updates. Shifting Bottlenecks Upstream in the Value Chain Accelerating task-level execution inevitably exposes systemic friction elsewhere in the organization. Schiel notes that while AI coding tools generate software 15 to 20 times faster, writing code is no longer the primary constraint. Instead, the constraint moves upstream into business requirement elicitation—a phase AI cannot automatically resolve—introducing the risk of rapidly engineering the wrong solutions. Downstream constraints also surface around user acceptance testing and the absorption rate of end users and databases handling frequent releases. Because hyper-fast task execution can outpace an organization's capacity to absorb changes, panelists emphasize that AI implementation must be measured across the entire value chain rather than judged through siloed speed gains. Enterprise Governance and Architectural Stacks Deploying agentic systems in heavily regulated settings demands robust containment and strict oversight to manage risk. Upadhyay outlines Vertex's framework, which centers on curated cross-functional data, mandatory employee AI training, and an AI General Council composed of legal, privacy, HR, and cybersecurity architects that must approve every model. To eliminate breach hazards, Vertex isolates non-production environments and maintains vendor independence across models, telemetry, and orchestration so corporate data never leaves its boundary. Howard Hughes mirrors this defensive design by utilizing an isolated sandbox for testing before solutions go before an IT steering committee, alongside mandatory quarterly employee AI re-certification. Mills conceptualizes the operational tech stack into five connected layers: capture from enterprise platforms, knowledge mapping, ontologies and artifacts, multi-step agent orchestration with human review checkpoints, and continuous post-deployment evaluation systems. Realizing the strategic potential of agentic AI depends on moving past disconnected productivity experiments into gated, programmatic execution. While automated execution mitigates risk and shortens lead times, enterprises still face unresolved organizational challenges in requirement elicitation and downstream absorption.

在 Ai4 主办的研讨会上，企业高管们指出，企业推进智能体（AI Agent）落地的核心不在于单纯节省个人工时，而在于重塑业务工作流、大幅压缩端到端周期并建立跨职能治理体系。 重构ROI标准：从虚荣指标到验证结果 (12:01) 多位嘉宾对“节省时间”这一传统衡量指标提出质疑。Jim Schiel 认为时间是廉价资产，真正的投资回报率（ROI）源于拓展组织原先不具备的能力；Marcus Spillane 进一步将孤立的工时节约称为“虚荣指标”，并透露 Howard Hughes 虽通过现成工具为员工增加了 20% 的产能，但若没有明确的机制将多余时间转化为实质产出，这种节省毫无价值。Jeff Mills 则提出应以“经过验证的产出”来评估 ROI，即在考量推理、Token 消耗以及人机协同（human-in-the-loop）审核等全部成本后，计算单位完成操作的总成本与业务价值。Agam Upadhyay 同样强调，零散的个人效率提升必须与明确的价值变现挂钩才有意义。 全流程重构与周期大幅压缩实践 (4:51) 企业推进智能体部署时，普遍倾向于从核心业务瓶颈入手，实现工作流数量级的提速。Vertex Pharmaceuticals 的 Agam Upadhyay 介绍，公司通过人机协同智能体将临床试验数据库的搭建周期从 14 周、数百人参与骤降至 5 天，并在需要 6 至 9 个月的新药监管申报中让 AI 独立完成 60% 的初稿撰写；在研发与生产中，智能体还承担了分子设计筛选和“异常质量监控”。Toptal 的 Jeff Mills 则指出，优化已有基准与负责人的核心流程胜率更高，其征求建议书（RFP）回复时间从 5 天压缩到 5 小时，销售线索响应延迟由 1 至 2 小时缩短至 1 至 2 分钟。Howard Hughes 亦构建了从意向书到合同起草、签署及 ERP 系统自动更新的端到端房地产租赁智能体流程。 技术加速暴露上下游系统瓶颈 代码和任务生成速度的大幅提高反向放大了组织前后的结构性制约。Jim Schiel 拥有 40 年软件工程经验，他指出虽然 AI 编程工具能够实现 15 至 20 倍的代码生成提速，但软件工程的核心瓶颈已从编写代码转移到了上游的业务需求获取环节，如果业务逻辑不清，AI 只会“以极快的速度制造错误的产物”。与此同时，组织在下游还面临最终用户和数据库系统对持续发布功能的吸收承载能力（UAT）限制。研讨嘉宾一致认为，智能体的效能评估必须覆盖从需求分析到用户交付的整条价值链，而不是单一环节的速度指标。 受监管行业的安全隔离与技术栈架构 在医药、地产等重合规领域，与会者拒绝“快速行动并打破陈规”的激进做法，转而采用人机协同降低操作风险。Vertex 建立了跨法律、隐私、HR、网络安全与技术架构的 AI 总理事会（AIGC）进行模型准入审查，实行全员合规培训，并严格隔离非生产环境以防智能体影响生产系统，同时避免与特定厂商绑定。Howard Hughes 则利用完全物理隔离的沙箱环境试跑方案，再由 IT 指导委员会逐级审批并推行季度政策重训。Jeff Mills 总结了企业理想的技术分层：包括从核心应用抓取信息的捕获层、关联知识库的映射层、定义逻辑的本体与工件层、嵌入人工判断的智能体编排层，以及投产后持续监测实际产出的评估系统。 与会者展示了智能体在大幅压缩核心周期上的确定性成效，但企业仍需面对如何把释放出的富余产能转化为实际业务收入，以及如何克服业务需求不清与下游吸收承载限制等系统性挑战。

- [00:08](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=8s) 企业落地 AI Agent 的核心目标在于深入探讨 Enterprise 级别的战略部署，并切实交付业务价值与衡量投资回报率（ROI）。 ## Key takeaways - [01:44](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=104s) Vertex Pharmaceuticals 通过个人 Agent 框架、解决未满足的业务需求以及大规模生产力（Technable Workforce）三大支柱推进 AI 战略。 - [01:44](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=104s) Toptal 提出了智能体落地的三个关键要素：明确的工作流、数据基线以及具体的 KPI，缺乏这三要素的尝试成功率仅约 1/10。 - [01:44](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=104s) Howard Hughes 采用 Agent-First 模式重构流程，在商业地产租赁中实现了从意向书（LOI）到 ERP 入库的全自动智能体化运作。 - [12:01](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=721s) 关于 ROI 的评估存在深刻辩论：“节省时间”可能只是虚荣指标，企业更应关注 AI 带来的全新能力（Capabilities）以及整体业务响应速度。 - [12:01](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=721s) Jeff Mills 提出评估 ROI 应关注“验证结果成本（Cost of Validated Outcomes）”，即综合计算 Token 成本、推理成本与人工审核评估系统的总投入。 - [01:17:59](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=4679s) AI 智能体的部署能够通过在标准化重复性工作流中结合“人机协同”（Human-in-the-loop）机制，大幅降低组织的操作与人为风险。 - [01:19:12](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=4752s) Vertex 严格筛选业务优先级，要求 AI 落地必须提升决策质量、实现倍数级周期缩短，并彻底剥离重复性工作量（如由 AI 完成 60% 的新药申报文档撰写）。 - [01:22:01](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=4921s) Jim Schiel 警告单纯提升开发或中间环节速度可能引发上游（需求提取）与下游（用户吸收与测试）的瓶颈，导致高效构建出错误的产品。 - [01:27:21](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=5241s) 在重度监管行业中部署 AI 需要坚实的数据平台、负责任 AI 政策、无供应商锁定的安全隔离区以及全路遥测监控（Telemetry）。 - [01:33:21](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=5601s) 企业级智能体操作系统架构包含数据捕获层、映射与知识层（本体）、编排与智能体层，以及持续监控的评估系统。 - [34:43](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2083s) 应将 AI Agent 视为动态演进的“数字员工”（Digital FTEs），企业必须为其建立类似于 HR 管理的治理体系与评估护栏。 - [37:23](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2243s) 高风险领域（如医疗和 R&D）落实 AI 护栏需搭建集中控制的企业级 AI 平台（如 VIDA），防止开发者绕过安全规范机制。 - [39:09](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2349s) 评估 AI 带来新能力的内部价值时，不应局限于财务金钱指标，而应看其是否推动企业实现核心战略目标，并防范无使用率的风险。 ## Detailed walkthrough ### [00:08](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=8s) 研讨会开场与嘉宾背景介绍 小组讨论由 Promenade Advisory 创始人兼 CEO **Aaron Stanley** 主持。Promenade Advisory 是一家专注于前沿科技、数字资产与人工智能（AI）领域的战略咨询与传播公司。Aaron 表达了对参会的热情，并提到这是他参加的第一个专门聚焦于 AI 的行业峰会。他强调，本次研讨会的核心任务在于深入探讨 Enterprise（企业级）Agent（智能体）的落地策略，不仅涵盖宏观的技术愿景，还将着重剖析如何真实交付业务价值与衡量投资回报率（ROI）。 随后，Aaron 依次介绍了同台的四位行业专家： * **Agam Upadhyay**：Vertex Pharmaceuticals（福泰制药）的资深副总裁（SVP）兼首席数字与信息官（CDIO）。 * **Jim Schiel**：敏捷转型咨询机构 Artisan Agility 的创始人兼 CEO。 * **Jeff Mills**：全球顶尖人才平台 Toptal 的 AI 首席客户官（Chief Customer Officer）。 * **Marcus Spillane**：大型房地产开发与社区建设企业 Howard Hughes 的创新业务资深副总裁。 --- ### [00:08](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=8s) 演讲嘉宾名录与背景摘要 - **Aaron Stanley**：Promenade Advisory 创始人，主持人 - **Agam Upadhyay**：Vertex Pharmaceuticals SVP 兼 CDIO，制药与生物科技领域 AI 实践者 - **Jim Schiel**：Artisan Agility CEO，敏捷开发与团队效能专家 - **Jeff Mills**：Toptal 首席客户官，企业级 AI 应用部署专家 - **Marcus Spillane**：Howard Hughes 创新业务 SVP，房地产技术与流程重塑专家 --- ### [01:44](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=104s) 各企业 AI Agent 的战略蓝图与实践探索 在自我介绍环节，四位嘉宾分享了各自企业在 AI Agent 部署上的核心路径与落地成果： **Vertex Pharmaceuticals 的三大战略支柱** Agam Upadhyay 指出，作为一家专注于重疾研发的领头生物制药公司，Vertex 早在 AI 成为流行概念前就建立了数据与算法根基。因此，公司目前将 AI 的发展视为“自然的演进而非剧烈的转型”。公司围绕三个维度构建智能体战略： 1. **个人 Agent 框架（Personal Agentic Framework）**：为每位员工提供安全的 LLM 环境，允许员工构建个人助手以提升日常办公效率，将其视为全员普及的基础工具。 2. **未被满足的业务需求（Unmet Business Needs）**：重点破解关键业务瓶颈。以临床试验中的“临床数据库”建库为例，过去该流程需要上百人协作、耗时约 14 周。Vertex 团队与业务部门协同重构流程，引入人机协作（Humans + Agents）机制后，成功将建库周期缩短至 **5 天**。 3. **大规模规模化生产力（Productivity at Scale）**：推出“Technable Workforce”项目，在全公司范围内梳理出 **22 个核心业务场景**，旨在随着公司业务呈双位数增长时，保持运营复杂度与人力成本曲线的平稳。 **Artisan Agility 的团队模拟试验** Jim Schiel 介绍，Artisan Agility 自 2009 年成立以来一直致力于敏捷开发的落地。如今公司利用 AI Agent 模拟具有严重功能障碍（dysfunctional）的真实团队，以此评估 AI 敏捷教练工具的介入成效。他形象地比喻道：这就像遗传学研究中的“蜉蝣”（Mayflies），人类团队需要几代人才能完成的迭代观察，AI Agent 可以在数分钟或数小时内快速模拟生成成百上千代的团队演进过程。 **Toptal 的“工作流-基线-KPI”落地三要素** Jeff Mills 强调，Toptal 拥有超 35,000 家客户与 30,000 名顶尖人才，其 AI 部署标准非常直接：**是否存在明确的工作流？是否存在数据基线？是否存在具体的 KPI？** 只要满足这三点，就能打造出高采纳率且具备商业价值的智能体应用。Toptal 内部已落地的智能体场景包括： * **RFP（招标应答）引擎**：将响应时间从 **5 天缩短至 5 小时**。 * **Inbound Lead（线索响应）引擎**：将线索处理时间从 **1-2 小时压缩至 1 分钟以内**。 * **SDR / Recruiter 智能套盒**（SDR in a box / Recruiter in a box）。 Jeff 同时提醒，缺乏上述三要素的尝试本质上是概率极低的“实验”（成功率仅约 1/10）。虽然 Toptal 最近成功的“品牌生成引擎”（Brand Engine）项目源于实验，但对初学者而言，Playbook 应优先从成熟的既有工作流切入。 **Howard Hughes 的 Agent-First 流程重构** Marcus Spillane 介绍，其创新团队拥有独立的沙箱环境，但将项目推向生产环境（Production）时仍需经过严密的治理、风险与合规（GRC）审查。Howard Hughes 为全公司 550 名员工搭建了安全统一的数据基层与权限控制平台，使员工能在确保上下文安全的前提下构建自用 Agent。在外部业务合作中，他们针对商业地产租赁流程进行了**端到端重构**：从意向书（LOI）、合同草案生成到最终签署并自动同步至 ERP 与租户门户，全程以智能体为主导，事后才将“Human-in-the-loop”（人机环路）按需安插回关键节点。 --- ### [01:44](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=104s) 流程重塑与商业落地对比 ``` [传统流程] ───► 现有工作流 ───► 直接套用 AI ───► 效率微升 (局部优化) [Agent-First] ─► 终点目标 ───► 智能体全程重构 ─► 按需插入人类介入 (颠覆式降本) ``` - **Vertex 临床数据库搭建**：周期从 14 周缩短至 5 天 - **Toptal RFP 响应流程**：耗时从 5 天降至 5 小时 - **Toptal 线索响应流程**：耗时从 1-2 小时降至 1 分钟以内 - **Howard Hughes 租赁流程**：实现 LOI 签约、合同生成至 ERP 入库全链路智能体化 --- ### [12:01](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=721s) 投资回报率（ROI）辩论：节省时间 vs. 业务价值 讨论进入核心议题——企业如何评估 AI Agent 的投资回报率（ROI）。四位嘉宾对“节省时间”（Time Saved）这一指标展开了精彩的辩论： **Jim Schiel 的观点：“节省时间”是错觉** Jim 明确指出，在评估 ROI 时，最不应该向客户承诺的就是“节省时间”。他认为时间往往是企业中最廉价的资源，周期缩短属于战略规划问题，而非单纯依靠 AI 解决的问题。企业应该关注的是 AI 带来了哪些**此前不具备的全新能力**（Capabilities）以及是否带来了更高质量的业务响应。 **Jeff Mills 的反驳：时间即生产力，需精准核算“验证结果成本”** Jeff 提出了不同看法。他认为对于大型企业而言，上千人的时间节省会直接转化为显著的 EBITDA（税息折旧及摊销前利润）提升。但他同意“仅省时间而牺牲质量”毫无意义。Jeff 提出了评估 ROI 的核心公式——**验证结果成本（Cost of Validated Outcomes）**： * **不能单看 Token 成本或推理成本（Inference Costs）**：这些只是变量的一部分。 * 需综合计算 **Token 成本 + 推理成本 + 人工审核/评估成本（Eval System）**。 * 如果一个原本需要 8 个人工作 5 天的 RFP 流程，最终耗费 200 美元的 Token 及推理算力即完成了高质量输出，那么其 ROI 依然是极高的。 **Marcus Spillane 的补充：“节省时间”属于虚荣指标** Marcus 赞同 Jim 的部分观点，并直言“节省的时间等于生产力”是个误区，甚至是一个“虚荣指标”（Vanity Metric）。Howard Hughes 的内部调查显示，尽管 AI 工具为员工平均释放了 20% 的产能空间，但如何将这 20% 释放出来的空余时间真正转化为额外的业务产出（Additional Productivity），依然是管理层面临的巨大挑战。 --- <FollowUp label="想要了解专家们对“Human-in-the-loop（人机环路）”的具体设计建议吗？" query="请详细梳理研讨会中关于 Human-in-the-loop（人机协作与风险控制）的核心观点与落地实践。"/> ### [01:17:59](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=4679s) 重新定义工作模式与风险控制 在探讨 AI 智能体（AI Agents）落地时，与会专家指出，如果仅仅将生产力提升理解为“让员工今天过得轻松一点”，这并不是真正的生产力。企业必须花大量时间重新设计工作流程，明确由谁、以何种方式去完成哪些具体任务，才能真正将效率转化为商业价值。 除了效率提升，智能体为企业带来的另一个核心指标是**显著降低组织风险**。通过在被充分验证的标准化重复性工作流中部署智能体，并辅以“人机协同”（Human-in-the-loop）的把控机制，企业可以极大地减少人为差错。虽然外界普遍担忧 AI 会增加合规与安全风险，但只要管理得当，这种基于规则与自动化校验的“程序化运行”反而能够成为企业降低风险的强有力屏障。 * **工作流重构**：重新分工并明确权责，确保时间节省能转化为可衡量的业务价值。 * **风险控制**：在标准化流程中引入智能体与人工审核闸门（gating），实现风险可控与流程合规。 --- ### [01:19:12](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=4752s) Vertex Pharmaceuticals 的三大优先战略与价值落地 针对“节省下来的时间究竟有什么用”这一难题，来自 Vertex Pharmaceuticals 的 Agam Upadhyay 提出了极其严苛的业务优先级排序标准。他强调，仅仅做到“省时”是不够的，必须将智能体深耕于整个业务价值链（Value Chain）中。 ``` +-------------------------------------------------------------------+ | Vertex Pharmaceuticals AI 落地三大核心 | +-------------------------------------------------------------------+ | 1. 提升决策质量 (Decision Quality) | | • 化学家在进入湿实验之前，利用 AI 框架设计分子 | | • 销售团队基于与 HCP 的互动，获取“下一步最佳行动计划” | | • 制造环节采用“例外质量管理”(Quality by Exception) 快速排查缺陷 | +-------------------------------------------------------------------+ | 2. 极速缩短周期 (Cycle Time Multiple Percentage Drop) | | • 拒绝小幅改善：周期必须呈“倍数级”缩短（如从 14 周降至 5 天） | +-------------------------------------------------------------------+ | 3. 彻底移除重复性工作量 (Removing Body of Work) | | • 自动化生成复杂的 regulatory 申报文档（如 NDA 及临床研究报告） | | • 60% 的新申报初稿由 AI 完成，大幅缩短药品上市时间 | +-------------------------------------------------------------------+ ``` Agam 特别分享了他们利用 AI 进行自动化文档撰写（AI Authoring）的成功经验。对于制药公司而言，提交新药申请（NDA）和临床研究报告（Clinical Study Reports）通常涉及数百页极其繁复的文件，往往需要数百名专家耗时 6 到 9 个月。而在 Vertex 最新提交的一份监管申报中，**高达 60% 的初始撰写工作直接由 AI 智能体完成**，这不仅减少了人力投入，更显著加快了新药推向市场的时间。 - **决策质量**：利用 AI 辅助化学家进行分子设计；销售人员获得与医疗专业人士（HCP）互动的最佳策略；生产线实现“例外质量管理”（Quality by Exception）。 - **周期缩短**：拒绝 10% 或 20% 的微小边际改善，追求如“14 周降至 5 天”这种量级突破。 - **工作量剥离**：将撰写复杂监管文档等繁重、重复的工作整体从人力负担中剥离。 --- ### [01:22:01](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=4921s) 重新看待“时间”：流程响应速度与上下游瓶颈 对于“时间”这一维度，Jeff Mills 提出了不同层面的解构。他指出，个体员工节省几个小时固然是好事，但企业更应聚焦于**整体业务流程的时间优化**。例如，如果将线索响应时间（Lead Response Time）从 2 天缩短至 2 小时，甚至从 2 小时缩短至 2 分钟，能够直接反映在企业的营收增长上；将产品发布周期从 2 周压缩到 1 周，也能带来巨大的市场竞争优势。 然而，拥有 40 年软件开发经验的 Jim Schiel 提出了极其深刻的警告：**仅仅提升流程中间环节的速度，可能会引发上游和下游的瓶颈阻塞**。 > "AI is really good at building the wrong thing. Really, really fast." — Jim Schiel 在软件开发领域，如果借助 AI 使得编写代码的速度提高了 15 到 20 倍，代码便从最昂贵的资产变成了最廉价的资源。此时，约束条件（Constraint）就会迅速向上游转移——即“业务需求从何而来”。如果上游业务方无法快速提供详尽、准确的需求定义，开发人员就会陷入无米之炊的困境，甚至极其高效地构建出“错误的产品”。 ``` +-------------------------------------------------------------------+ | 软件开发中的 AI 约束转移链 | +-------------------------------------------------------------------+ | 【上游：需求获取】 【中游：开发编码】 | | 需求难以快速/精准提取 ---> AI 编码速度提升 15-20 倍 | | (Elicitation Bottleneck) (不再是业务瓶颈) | +-------------------------------------------------------------------+ | v 【下游：客户吸收/测试】 用户测试与接受度跟不上 (Absorption Rate Bottleneck) +-------------------------------------------------------------------+ ``` - **上游瓶颈（Upstream）**：业务需求提取（Requirements Elicitation）跟不上代码生成速度，导致 AI 高效构建错误需求。 - **下游瓶颈（Downstream）**：新功能发布过快，超出用户的吸收与测试能力（Absorption Rate），容易造成验收卡顿。 - **ROI 核心**：AI 落地必须针对整条价值链（Value Chain）进行端到端优化，而非单一口径的任务（Task）级加速。 --- ### [01:27:21](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=5241s) 重度监管行业下的 AI Playbook 与合规架构 讨论转向实际落地与技术栈选择时，主持人 Aaron Stanley 提到，医药等行业属于重度监管领域，绝非硅谷“快速行动，打破常规”（Move fast and break things）的套路。Agam Upadhyay 详细拆解了 Vertex 如何在严苛的合规红线内构建智能体架构： 1. **数据平台沉淀**：花大量时间清洗与整理数据，构建坚实的数据栈（Data Stack），确保 R&D、商业、供应链、制造及财务数据高质可用，避免“劣质数据导出劣质结果”。 2. **负责任 AI 政策与通用委员会（AIGC）**：全员强制接受“负责任 AI 政策”培训。组建 AI General Council（由隐私、HR、法务、网络安全与技术架构师共同组成），每一个 AI 能力与模型均需通过审批备案。 3. **安全闭环与零锁定（No Lock-ins）**：搭建预训练模型与 Agentic 框架的安全隔离区，确保敏感数据绝不出界。在技术选型上坚决保持“无供应商锁定”（No Lock-ins）。 4. **全路遥测与安全监控（Telemetry）**：对所有调用进行全面审计与遥测。针对类似 Hugging Face 出现的安全漏洞或智能体越权隐患，Vertex 严格锁定非生产环境（Non-production），防止测试智能体“逃逸”并影响生产环境。 与此同时，Howard Hughes 的 Marcus Spillane 补充道，作为上市公司，他们设立了由法务、网络安全、高管代表组成的 IT 指导委员会，并通过“季度复训”确保政策落地。为了既保持创新敏捷度又兼顾安全，他们建立了**独立的创新沙箱（Sandbox）**，与企业生产环境完全隔离。创新团队可以在沙箱内自由尝试市面上的工具或自研原型，只有当项目展现出明确价值时，才会启动严格的法务与安全审查，正式引入生产网。 --- ### [01:33:21](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=5601s) 企业智能体系统的技术栈拆解 在谈及智能体工作流的架构设计时，Jeff Mills 为现场观众拆解了一套清晰的企业级智能体操作系统（Operating System）构建路径： ``` +-------------------------------------------------------------------+ | 企业级智能体操作系统技术栈 | +-------------------------------------------------------------------+ | 1. 数据捕获层 (Capture Layer) | | 捕获员工在 Salesforce, ServiceNow 等系统中的业务情报与操作行为 | +-------------------------------------------------------------------+ | v | 2. 映射与知识层 (Mapping & Knowledge Layers) | | 将业务情报映射至知识层，构建领域本体 (Ontologies) 与构件 | +-------------------------------------------------------------------+ | v | 3. 编排与智能体层 (Orchestration & Agentic Layer) | | 智能体编排工具调度业务逻辑，融合人类判断 (Judgments) 生成响应 | +-------------------------------------------------------------------+ | v | 4. 评估系统 (Evaluation System) | | 智能体进入真实业务环境后的效果评估与持续监控 | +-------------------------------------------------------------------+ ``` - **数据捕获（Capture）**：捕获员工日常使用 Salesforce、ServiceNow 等 SaaS 应用时的业务习惯与关键情报。 - **映射与本体（Mapping & Ontologies）**：将业务逻辑转化为知识层、领域本体与结构化构件。 - **编排与判断（Orchestration & Judgments）**：调度智能体构建对工作流的自动化响应，并在关键节点融入人类判断。 - **评估反馈（Evaluation）**：对发布至真实环境中的智能体进行持续的评估与闭环优化。 ### [34:43](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2083s) 将 AI Agent 比作青少年与数字员工的管理之道 讨论首先引入了一个生动的比喻：将 AI Agent 的管理比作管教家中的青少年。讲者分享道，就像为十几岁的孩子提供安全且富有营养的成长环境一样，构建 AI 平台时也会为 Agent 提供最优质的数据（正如 Andrew Ng 所言“数据是 AI 的食粮”）、明确的 KPI 以及标准作业程序（SOP）。然而，Agent 在进入实际应用环境后，往往会产生意料之外的行为——就像孩子早晨出门穿着一套衣服，晚上回家却换了一身完全没见过的衣服，让人大吃一惊。这表明 AI 绝不是“设置后即可一劳永逸”的传统软件，而是一个处于动态变化中的生机勃勃的生态系统。因此，企业的技术栈不仅要能捕获工作流，更必须建立完善的评估与监控体系，设置严密的护栏（Guardrails）。 随后，与会嘉宾进一步拓展了这一观点，提出应将 Agent 视为企业的“数字员工”（Digital FTEs）。对待数字员工不能仅看初始设计，更要建立长期的技能评估、结果验证与持续监控机制。正如企业为人类员工建立人力资源（HR）管理流程一样，企业也必须为数字 Agent 建立相应的治理（Governance）体系。 ``` +-----------------------------------------------------------------+ | AI Agent 治理与管理模型 | +-----------------------------------------------------------------+ | [ 输入层 ] 优质数据 (Data) + KPI + 标准作业程序 (SOP) | | | | [ 运行层 ] 自主运行与学习 (与实际环境交互，产生未知行为) | | | | [ 控制层 ] 护栏机制 (Guardrails) + 评估系统 (Evaluation System) | | | | [ 治理层 ] 数字员工 HR 治理体系 (对齐业务目标与风险控制) | +-----------------------------------------------------------------+ ``` - **核心风险观**：不管是软件流程、业务工作流还是 AI Agent，其治理核心都在于风险控制；业务风险越高、潜在影响越恶劣，就越需要施加严格的管控和边界。 - **行业经验借鉴**：讲者结合自身在医疗器械行业 23 年的经验指出，高风险领域必须对 Agent 的行为权限进行严密限制。 --- ### [37:23](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2243s) 提问：如何在医疗与 R&D 等高风险领域落实 AI 护栏？ 现场观众（来自医疗器械研发领域）提问：单纯依靠政策和规章制度（Policies）属于过程导向，往往无法有效约束开发人员的实际操作，那么在实际框架中，该如何为开发者搭建一个既安全又能自由探索 AI 的“沙盒/游乐场”（Playground）？ Vertex Pharmaceuticals 的 Agam Upadhyay 对此进行了详细解答。他强调，仅靠宣贯政策和培训绝对无法杜绝人为错误，因此 Vertex 打造了一个高度集中的企业级 AI 平台——Vortex AI 数字助手平台（简称 VIDA）。该平台为全公司员工提供统一入口，严禁员工私自散乱地构建底层平台能力。 ``` +-----------------------------------------------------------------+ | Vertex Pharmaceuticals (VIDA 平台架构) | +-----------------------------------------------------------------+ | [ 全员统一入口 ] Vortex AI Digital Assistant (VIDA) Platform | | | | [ 通用层 ] 企业级标准 Agent (科研/HR/采购等，全员共享) | | | | [ 沙盒层 ] 个人/团队定制 Agent (受监控的受控开发空间) | | | | [ 安全层 ] 集中式监控与护栏 (严防越界与合规风险) | +-----------------------------------------------------------------+ ``` - **统一治理策略**：VIDA 平台预置了企业级的公共 Agent 供全员使用，同时也为员工开发个人 Agent 或特定领域 Agent（如科研、HR、采购等）提供了受控的个人空间。 - **集中式与强监管**：由于医疗行业容错率极低，目前所有 Agent 的运行与开发均建立在集中监控的平台之上，确保无人能绕过安全护栏。未来随着成熟度的提升，可能会为了提升速度而逐步去中心化，但现阶段仍坚持严格的集中式治理模式。 --- ### [39:09](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2349s) 提问：如何为内部 AI 创新与新能力评估真实价值？ 观众提出第二个问题：AI 实现的“省时降本”容易量化，但 AI 带来的“全新能力”（即以前无法实现的事物）在缺乏市场定价的情况下，企业该如何评估并赋予其合理的内部价值？ 专家 panel 针对这一难题从工作流基线、业务目标映射以及用户采用率三个维度给出了具体建议： 1. **建立工作流基线与避免“凭感觉”（Vibes）**：要评估新能力，首先必须梳理明确的工作流和质量基线。如果缺少可对比的基线，AI 探索就会沦为凭感觉的“摇水晶球”式猜想。项目不能仅凭单个利益相关者的头脑风暴，而必须像真正的内部项目一样进行工作流映射，明确定义每个环节的成功标准。 2. **选择高频痛点工作流**：建议优先选择那些重复性高且让现有员工感到极其繁琐、头疼的工作流。直接听取一线员工的抱怨，针对痛点进行改进，不仅能显著提升质量和加快生命周期，还能有效降低风险。 3. **重新定义“价值”：远离纯财务指标，对齐业务目标**：很多企业最大的误区在于试图把所有价值都硬性折算成金钱。**价值的本质是推动企业实现战略目标的程度**。如果一项新能力能够大幅推动企业迈向核心目标，它就是高价值的；如果推动微乎其微，那就是低价值；如果毫无帮助，哪怕看起来再炫酷也不应该做。业务目标才是真正带来经济回报的源头。 4. **关注采用率（Adoption Rate）风险**：嘉宾特别补充提醒，对于原本不存在的新工作流，最大的隐患在于“无人使用”。如果企业内部缺乏现成的承接土壤，这项新能力该如何推动落地和推广，是评估其最终价值时最不可忽视的一环。 --- ### [42:00](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=2520s) 研讨会总结与致谢 主持人对现场观众的热情参与和高质量提问表示感谢，并宣布上午的研讨环节顺利结束，同时预告了下午即将进行的第二部分 panel 讨论，欢迎感兴趣的参会者在台下与嘉宾继续交流。 --- ### 视频信息与参考链接 - **视频标题**：The AI Agent Playbook: A Strategic Roadmap for Implementation - **发布频道**：[Ai4](https://www.youtube.com/watch?v=vhOiDiyXFUs) - **观看链接**：[https://www.youtube.com/watch?v=vhOiDiyXFUs](https://www.youtube.com/watch?v=vhOiDiyXFUs) ## Notable quotes > [01:23:21](https://www.youtube.com/watch?v=vhOiDiyXFUs&t=5001s) AI is really good at building the wrong thing. Really, really fast. — Jim Schiel ## Mentioned Aaron Stanley, Agam Upadhyay, Ai4, Artisan Agility, EBITDA, Howard Hughes, Hugging Face, Jeff Mills, Jim Schiel, Marcus Spillane, Promenade Advisory, RFP, Salesforce, ServiceNow, SDR, Toptal, Vertex Pharmaceuticals, VIDA ## Video description Agam Upadhyay - SVP, Chief Digital and Information Officer, Vertex Pharmaceuticals Jeff Mills - Chief Customer Officer, Toptal Jim Schiel - CEO/Founder, Artisan Agility Marcus Spillane - SVP Innovation, Howard Hughes Aaron Stanley - Managing Director, Promenade Advisory Successfully integrating AI agents requires a clear strategic roadmap. This panel provides a comprehensive guide to creating your organization's AI agent playbook. Experts will discuss essential elements of strategy formulation, from initial vision and goal-setting to practical execution steps. Learn how to anticipate and address challenges proactively, measure effectiveness continuously, and iteratively refine your approach to ensure sustained value and strategic alignment. [Source](https://www.youtube.com/watch?v=vhOiDiyXFUs)

[83:21] Jim Schiel: AI is really good at building the wrong thing. Really, really fast.

[12:01] Jim Schiel: When you're looking at AI in terms of ROI, you really have to be thinking about: what capabilities does this give us that we don't have right now? [22:01] Jim Schiel: If all of a sudden code, which used to be the most expensive asset we had in IT, has now become the least expensive asset we have... guess where it moves? It moves upstream to the business.

[0:08] All right, guys. Uh, so my name is Aaron Stanley.  Uh, I'm the founder of Prominade Advisory. It's [0:12] a strategic advisory and communications firm  focused on emerging tech, digital assets, AI, the [0:18] whole gamut. Um, really excited to be here today.  This is uh really been an incredible conference. [0:22] Actually, my first AI conference, first purely  focused AI conference. So, it's just been like a [0:26] really incredible experience just learning from  all of you and and just being a part of these [0:30] conversations and just really diving into kind  of the nitty-gritty of how this technology is is [0:35] um really kind of fulfilling its transformative  potential right now. So, um and on that theme, [0:40] we've got a really interesting panel here  with some some great minds um discussing [0:44] uh agentic implementation and strategies uh in the  enterprise level. So, we've got a number of really [0:49] great perspectives here that we're going to be  bringing. Um going to be diving into um you know, [0:53] some um you know, just some of the the um the  big picture stuff, but also the nitty-gritty [0:58] details of you know, where are we at with actually  implementing this stuff and delivering value for [1:02] our customers and and finding ROI on these types  of investments and all that kind of stuff. So, [1:06] appreciate you all being here today for this  and hope you enjoy the uh the conversation. But, [1:12] um going to introduce everybody here. So, we've  got um Agum who's the SDP and CDIO at Vert.Ex. [1:17] text pharmaceuticals. Um, we've also got to my  uh my left here is Jim Sheiel of Artisan Agility [1:25] and Jim's been in the the digital transformation  business for a long time. So, he's got some great [1:28] insights for us. We've got Jeff Mills of uh chief  customer officer for AI at TopT. And we've got uh [1:34] Marcus Spain who's uh from Howard Hughes. Uh  nice to have an Irishman on the panel. Has a [1:40] little extra, you know, a little flavor here. So,  um so to kick off, we're going to do just kind of [1:44] a quick um intro round here. We'll go down the  line and um Augum we'll kick off with you but [1:49] just maybe tell us a bit more about yourself and  what you're focusing on and um you know how are [1:52] you guys thinking through this you know AI agent  playbook uh in at at Vertex Pharmaceuticals. Yeah, [1:58] thank you Erin. Great to be here. Um, so little  bit about Vortex Pharmaceutical. If you don't [2:03] know, uh, we are one of the leading biotech  biopharma companies. Um, we do, uh, medicine for [2:11] serious diseases for our patient. We work in dis  four plus disease areas. Um, the company formed [2:18] uh, on the AI foundation uh, when AI was not that  popular um, as has been there. So for us is like [2:25] evolution rather than pivot point. um and the way  we are approaching um the AI with with the company [2:31] is more strategic and intentful. Um so the way  in the organization and the way we have planned [2:38] it and the decided there the three areas one area  is personal um agentic framework. So means every [2:47] employee has access to the secure LLM and they  can create their own personal agent for their [2:55] own efficiency and productivity and we generally  almost count it as pretty much having everybody [3:01] tool in the organization. There is nothing  else more than that. So higher engagement, [3:05] higher adoption and then people create stuff for  them so they can be more efficient on their work. [3:10] The second area is unmet business needs. So for  example, if you if you know the pharmaceutical [3:15] organizations um finding the right molecule um  taking the molecules faster from clinical trials [3:23] and then commercialization is a key part. So we're  finding the area intentful where we can actually [3:28] drive it. So creating agentic framework just  giving a quick example in one of the development [3:32] cycle of molecule something called uh clinical  database. So you create a clinical database that [3:37] can take about 14 weeks in general like you and  and hundreds of people work on it. We sat down [3:44] with our business team uh and my organization  and we reimagined the whole process because [3:49] that's a key part. We imagine the whole process  and we implemented agents and humans together [3:56] um and we brought down that 14 weeks into 5  days. I mean it was incredible outcome and [4:01] you think of it how much better it is for our  patients. So we driving that way and the third [4:06] area is basically for us which is very important  is productivity in a manner where you can realize [4:11] the productivity it's not personal but taking  at the work at the at the grand level. So we [4:16] initiated something called technable workforce  in the organization which basically is looking [4:21] across the board and we found 22 such key areas  that we're working on right now to drive that [4:28] value for the organization which will provide you  know huge amount of value in terms of resourcing. [4:33] We are double- digit growth company and we want to  make sure that our operational complexity and our [4:39] cost curve including our headcount stays flatter  because of this. So three areas personal unmet [4:45] business need and the productivity at scale.  Cool Jim. So good morning everybody and thank [4:51] you for coming. Um as Aaron said my name is Jim  Sheiel. Uh Artisan Agility is is a pretty small [4:58] organization. It's me and a few other people  and and I founded it back in 2009 to take uh [5:04] agile development which I'm sure a lot of you have  heard of uh and really get it out into the teams [5:10] and into organizations and make a difference. Uh  AI has been something that has come along uh as [5:17] of late um the last couple years and we're using  it I have to say for just about everything uh from [5:24] internal needs uh building our company website and  putting all the cool tools in it that that I've [5:29] always wanted in it um to using it with teams uh  to a lot of my focus these days is enhancing team [5:37] productivity whether it's with agile development  or it's with AI or usually both it's how do we [5:43] get the most out of the people that are in our  organizations today. One of the I think one of [5:48] the most exciting things that we're that we're uh  I'd say playing with right now that kind of makes [5:54] it sound wrong but developing I we're what we're  working on is actually creating using AI agents [6:02] to simulate human teams with dysfunctions uh and  see how quickly our tools can actually provide the [6:10] right coaching information. Um, so it's kind  of an incremental thing. It we we create a [6:15] super dysfunctional team. Think of the worst team  you've ever seen. All right. Uh, and then see what [6:20] kind of advice uh is is rendered up to that team.  What happens if they follow? What happens if they [6:26] don't? And you can imagine it's uh it's something  that's in progress. So, it's a very iterative [6:32] thing. Uh, it the the advice isn't always the  best, but you try again and you improve. And [6:38] it's kind of like uh mayflies in genetics, right?  You can't wait for humans to go through an entire [6:44] cycle, an entire generation, but may flies do it  in a couple weeks. We're doing the same thing with [6:50] teams. Just using AI as agents, using AI agents,  excuse me, to create generations and generations [6:58] and generations of teams in minutes, hours.  Uh so it's pretty cool stuff. There's a lot of [7:05] nice stuff going on. Uh but that's kind of what  artisan agility is all about is improving team [7:10] productivity through whatever tools are available  now including AI. Love it. Yeah. So Topt towel [7:18] has 35,000 customers and over 30,000 best-in-class  talent that's supporting them. So we're deploying [7:25] uh purpose-built AI applications across every  industry and across every function. Right now [7:31] my litmus test is extremely simple. Do you have a  workflow? Do you have a baseline and do you have a [7:38] KPI? If you have these three things, you can build  amazing purpose-built applications within the AI [7:45] space. Basically, what happens with the workflow  is you already know you have adoption. With the [7:51] baseline, you have data that you can leverage to  build real agentic workflows. And with the KPIs, [7:56] you have a business owner that you know  wants to see things move forward. So, [8:00] as we talk about agent playbooks, those are the  types of things that I'm going to be focusing on [8:04] during during this uh during this panel. Uh we've  launched a whole bunch of them. A couple internal [8:09] recently are like our RFP response engine taking  RFP response from what was about 5 days down to [8:15] about 5 hours. Inbound lead response engine  taking inbound leads what were about an hour [8:20] or two down to about a minute or two. Uh SDR in  a box, recruiter in a box. Anytime you can grab [8:25] a workflow that you know your team is working  on that is part of your day-to-day operations [8:31] and business, this is where you should start to  deploy AI. When it's not that, what you're doing [8:38] is an experiment. And experiments are awesome.  And sometimes experiments can be the best AI [8:44] application that you ever build, but you're going  to have about a one in 10 uh result for that. So, [8:50] you're basically taking the aball, you're  shaking it, you're coming out with something, [8:54] and then it's like optimistic looks and seeing  what happens. Uh, one of our experiments we ran [9:00] was a brand engine recently. We are demoing it  at our booth. Uh, there's some people in the room [9:04] here that that demoed it yesterday. Uh, and it it  works, right? And and so that was one of the one [9:10] and 10s that that really did well. And so, I don't  want to say not to do experiments because I think [9:15] experiments are incredibly important. But for  those in here who are at the AI agent playbook, [9:20] uh my guess is you're pretty much pretty early  in the in the process. So start with workflows [9:26] that are try and true to to your operation. Uh  thanks Aaron. Um so at Howard Hughes, my team [9:34] kind of sits off to the side. I run innovation  and we have you know our own sandbox to play [9:40] with. So it's it's really interesting because we  start in that sandbox experimenting with a lot of [9:45] things. Obviously kicking it across the wall into  production in Howard Hughes itself requires us to [9:51] go through governance risk compliance a whole  bunch of things that by its nature tend to slow [9:56] things down. And so we've been very deliberate  about what do we need as a baseline to be able [10:02] to deploy AI efficiently within the company.  And and the reality is we have 550 people. Um, [10:11] everyone has access to an offtheshelf platform  that keeps our data secure, provides context, [10:18] uh, connects all of our data into this layer.  Uh, all of our people have access to that with [10:23] their user base, uh, access control coming from  the subsidiary system. And so the great thing [10:29] is it gives them contextual search. It provides  a platform in a secure environment for them to [10:34] build agents for themselves. And you're exactly  right where uh it is about workflows that are [10:42] where you you know what the baseline is, you  know what it's supposed to do, but that's just [10:46] you know pretty repetitive and that allows people  serve themselves. Separate to that, we have uh a [10:53] number of external partnerships where it's not by  the the product. We've actually built it ourselves [10:59] and we have some partners with that expertise uh  that have gone. And so if you take leasing in the [11:04] real estate world, we've gone end to end from a a  letter of intent to a lease draft to a lease being [11:11] signed pushed to all the requisite systems, our  ERP, our tenant portals, etc., all agentically. [11:18] And we have a bunch of sanity checks in there.  But we designed that process first with the agents [11:25] in mind and then retrospectively figured out how  where the humans in the loop needed to be rather [11:31] than taking our existing uh workflows and just  plugging AI into into that existing one. And I [11:37] think that's the other kind of bit of guidance  I would say is that take a a difficult workflow [11:42] but that's pretty repetitive but spits out the  same thing every time and then reimagine it end [11:48] to end agentic first and then put people back into  the loop. That was great. Thank you everybody for [11:54] for those introductions. It's great hearing  just the different angles you guys different [11:57] perspectives you guys bring to this conversation.  And I I think that to maybe kick kickstart the [12:01] uh our our kind of more uh formal discussion here.  I really want to talk about just this question of [12:06] ROI because I think that's always the main thing  that you know folks are looking at when they're [12:10] um you know on one hand it's okay these are these  are cool new toys and we can experiment we can do [12:14] cool stuff and like and there is a lot to be said  for just kind of getting in and getting your hands [12:17] dirty and playing around with things and and you  know you solve one problem then it kind of the the [12:22] light bulb goes off in your head like oh I could  redeploy this for this and this other thing but at [12:25] the same time there's also uh cost to these things  there's the cost of your time that you spend it [12:29] takes time to like learn these tools takes time  to develop these tools um you know there's there's [12:33] cost to the tokens that you're burning while  you're while you're deploying these things and [12:37] using these things. So, um maybe um Jim, I'm going  to kick this over to you first. Uh but I would [12:43] love you know when you're looking at this question  of um just factoring in the ROI into these or when [12:49] you're when you're approaching a new project and  you're looking at the ROI of it like how are you [12:52] kind of working through this um this this this um  dichotomy with with your customers and with your [12:57] your clients? Good question. So ROI, the the the  one thing that I've learned over the last couple [13:05] years with ROI is the last thing you want to hear  or even give to your customer is it's going to [13:11] save you time. Uh number one, time is probably  the least expensive asset that you have to play [13:19] with. Yes, I know. Well, we got got to get this  done by a certain date or this has to be ready by [13:24] I I get that. I understand that. But that's not  an AI solvable issue. That's got to do more with [13:32] how we're planning our strategic approach and so  on. When you're looking at AI in terms of ROI, [13:37] you really have to be thinking about things like  what capabilities does this give us that we don't [13:44] have right now or that we don't have enough of  right now where we can really speed up. And I [13:51] don't mean speed in terms of time. I mean speed in  terms of trim the cycle time and be able to move [13:57] more quickly. That that's you don't do that at the  cost of expertise and you don't do that to save [14:02] time. You you implement AI where it can really  make a valuable response to your organization. Is [14:09] it going to give us something better than we had  before? Is it going to give us something that like [14:14] I said we couldn't do before? uh is it going to  expand our capabilities uh in a way that we can't [14:21] do just by hiring really smart people? Uh so that  that would be my just a very short approach to ROI [14:28] is please please please take anything away from  this time saved is not a factor. Don't don't look [14:36] at that. Look at the value it brings. look at the  the skills and capabilities it provides before you [14:42] try to implement AI in something and say, "Well,  that's that's what we need to do." Well, that's [14:47] an interesting perspective because I feel like in  a lot of conversations I've had with people this [14:51] week about like, "Okay, what are you deploying  agents for?" And, you know, people will be like, [14:54] "Oh, I did it, you know, I I figured out a way to  take this, you know, X workflow that used to take [14:58] me, you know, four hours to do and now the agent  does it. Now, it only takes me like one hours. [15:01] I saved three." Like, that's always kind of the  first immediate metric you're thinking of. It's [15:05] like, is this thing effective or not? It's like,  oh, it saved me some time, right? Right. So your [15:08] perspective is like okay that shouldn't be  necessarily like the the you know the north [15:12] star as whether this is a good idea or not but  um maybe um you know do you want to try? Yeah [15:17] I I agree 100% that uh time saved is not the only  metric. Uh but productivity is an important thing [15:24] and I think of time saved as productivity. So  Jim and I will will tussle a little bit here. [15:28] Uh you know my organization is a lot bigger  than your organization. So when you're talking [15:33] about thousands of people working, uh time saved  actually adds up a lot of value to our Ebida and [15:39] things like that over time. So um time saved is  one metric. The next metric is the quality. If you [15:44] save a bunch of time, but the quality is not very  good, well that's not that's not going to help [15:48] us. So the first baseline for me is productivity,  then it's going to go to quality because if I save [15:53] time, but the output wasn't good, uh that's not  very helpful for me. And so then I'm going to get [15:57] to ROI. And ROI for me is validated outcomes.  So at the end of the day, how do you actually [16:03] get all the way down to what is the cost of I  mentioned RFP and a response uh engine earlier, [16:09] saving time from 5 days to 5 hours, but ultimately  how much does it cost me to get a new RFP out? [16:17] And if that all of a sudden ends up being call it  $200 in token costs and in inference and in all of [16:24] these p you know places but I saved eight people  who had worked on that for 5 days before I'll tell [16:31] you my ROI is going to be really really high but  you want to get it down to validated outcomes. And [16:37] so it's not token cost. I could care less about  token costs. Token costs are one variable in your [16:44] validated outcome cost. Inference is one variable  in your validated outcome cost. Human in the loop, [16:50] which we're going to get into at some point,  right? You have your operating system and then you [16:53] have your eval system. As those eval are coming  through, how much are those costing you for that [16:58] validated outcome? All that needs to be in there  so you know that the cost of whatever it is was [17:04] X. Um, and so to me, the ROI is actually getting  down to that validated outcome price. I think the [17:10] bit that I'd add and maybe slightly delving into  it is that I think time saved is a vanity metric. [17:16] Um I don't agree for a second and I'll explain  this that time saved equals productivity. We when [17:24] we survey our all of our employees in in Howard  Hughes we we see that yeah we've increased the [17:30] capacity by 20%. Each person has 20% more time  because of the AI tools we've provided them. But [17:38] we're in right now we're working on a a process  to figure out how do we convert 20% more time [17:45] into additional productivity. Sure certain people  will drive that forward and they will do 20% 50% [17:52] 80% more work with that time but if somebody  is sitting there saying you know what I'm just [17:59] going to have an easier day today that isn't  productivity. And so we're spending a lot of [18:03] time now thinking about how we redesign work  and who does what work to actually get that [18:10] uh productivity out. And the other metric that  we would certainly um uh kind of a kind of look [18:17] at and measure is what's the reduction in risk  for our organization by having these agents run [18:24] repeatable workflows that have been validated  that have humans in the loop. But I think that [18:30] risk mitigation is a is a really big big thing for  us and I you know that's quite something that you [18:35] don't often hear as it relates to AI because  everybody is saying oh AI adds risk to the [18:40] organization yes if you don't manage it properly  but from our perspective having repeatable [18:45] workflows that are done very programmatically  with the right kind of a val and right gating [18:51] uh reduces risk for us and I I'm wondering um  yeah so the way we so I actually agree with you [18:58] the time saving and if you can't realize the time  then I just see a save for what for what like how [19:03] where's value so what a value realization and very  hard to get when you save time and individual's [19:09] productivity so the way we actually prioritizing  it number one what is the the business need that [19:15] we are solving so one of one of them for us is  decision- making so can I have a better decision [19:22] making quality now what that means in terms of for  us is we enable all our chemists with AI molecule [19:27] design So before they can bring any molecule  in the bet lab they actually designing using AI [19:33] and agentic framework to get a better results. We  are basically almost all our workforce salesforce [19:40] people who are out there working with SCPs have  next best action plan which means they know that [19:45] how are they going to take action based on the  interaction with the SCP and how they can actually [19:49] drive further. We are doing for quality quality  by exception because remember there's a large [19:54] manufacturing and so forth. How can I get a better  decision through agitics framework where this lot [19:59] of data that actually human look at it today and  they are basically figuring out where the quality [20:04] defects can be or where the deviation but now  those agents are can look at this huge amount [20:09] of data much faster with better quality within  human loop going there and validating it. So your [20:14] quality is improving you have better prediction  on drift on quality and so forth. So that's a [20:19] number one which means decision- making quality  and and quality of decision making and faster is [20:23] a key part. The second thing that we believe in  a cycle time and I think you you touched on it [20:28] cycle time part which basically is is something  that takes today x amount of time am I making it [20:34] enough so it can't be certain percentage has to  be multiple percentage of it in order to make a [20:39] dance example I gave earlier from 14 weeks to  5 days 14 weeks to 12 weeks will not cut it it [20:44] just you know it really is not that valuable to  spend all this time it has to be considerable [20:48] so we are brutally brutally prioritizing it to  make sure that the good idea actually a good idea [20:54] Third productivity part so not time saving you  need to take the work out in the sense the body [21:00] of work has to go out example for us is the AI  authoring now as a pharmaceutical company all [21:05] of them including our it's a lot of authoring lot  of document writing that goes on everything from [21:10] submission everything from submitting submitting  NDAs and and then clinical study reports and so [21:16] forth these are very complex hundreds and pages of  document that's created and it's done by hundreds [21:21] of people it takes six 8 n months to do that  for every drug release. Now we just actually [21:26] implemented where our first thing 60% of the first  immediate the filing that we just did last month [21:31] was done by AI. It reduced the amount of uh  people that they're working on in terms of um [21:38] war but also increased the time to take the drug  to the market. So that is a combination of these [21:43] things where the time saving is not enough. It  has to be realized value and then more importantly [21:49] we are prioritizing the more business where our  scientists our manufacturing even our knowledge [21:54] worker getting the thing that they didn't have  before. I just I just want to add like uh one just [22:01] um we're talking about time in multiple different  ways here and so I think that's something that I [22:04] think people have to to realize. If we're talking  about time savings of an individual employee and [22:09] stuff like that, now I agree with you. But I can  guarantee you right now that if your lead response [22:15] is two days and we cut it to two hours, you're  going to increase your revenue. Absolutely. If you [22:22] drop it from 2 hours to 2 minutes, I guarantee  you we're going to increase your revenue. So, [22:27] we have to think about time in a couple different  pieces. I think uh in the in the kind of bigger AI [22:32] world right now, people are thinking about time on  an individual use case on how they leverage cloud [22:37] or whatever it is to code or something and that's  fine. I'm talking about time in how does your [22:42] um your you know release cycle has dropped from  taking two weeks to one week. That's the time I'm [22:48] talking about. And so I just wanted to make that  uh clear that I'm talking about time in actual [22:53] workflows, not just time about an individual  itself. May I one more you make me think. I [23:01] completely agree. He's absolutely right and  I I key at this from time. My 40 years has [23:06] been in software development, software product  development. U and and when I think about time, [23:11] it's something that Agam said uh even before  the panel this morning. You save time, [23:15] great. What are you going to do with it? If  you take four hours and make it one hour, [23:18] what are you gonna do with the three? Could do  you have an effective use of it? Right? And what [23:22] that made me think of when both of these guys  were talking was that in software development, [23:27] let's say you start using AI and you can write  code and I'm not making up numbers here, 15 to [23:32] 20 times faster than you could write it before.  That's fantastic. But where are the requirements [23:39] coming from? Right. Right. If all of a sudden  code, which used to be the most expensive asset [23:44] we had in it, has now become the least expensive  asset we have, it's still valuable, but it's easy [23:50] to create. When when that constraint is removed,  guess where it moves? It moves upstream to the [23:58] business. and they provide you. If anybody that's  been involved in software development knows how [24:04] hard it is to get business requirements, detailed  elicited requirements out of the business and then [24:11] when you can get somebody, they're not even  sure how to answer your questions. So, okay, [24:16] you've got a bunch of developers that are now  writing code like that. Where are the requirements [24:22] coming from? And you know what AI is really  good at doing? building the wrong thing. Really, [24:28] really fast. So, if you think about a a playbook,  you want to implement AI in software development, [24:35] by all means, do it. Okay? There are going to be  some people that going to freak out about it. I [24:40] can tell you from experience, there going to  be coders going to go, I can't trust AI. Guys, [24:45] if you go back far enough, we used to not trust  the compilers. Okay? So, we'll get there. We'll [24:52] trust AI. It'll build the code that we need. But  if you don't look upstream, you're going to be [24:58] where Agam was talking a little while ago,  what are you going to do with those hours, [25:03] right? Because if the requirements aren't there,  you're stuck. So when I said earlier, time is [25:08] probably not your biggest constraint. I I agree  with Jeff. If you look at it from the the lead [25:14] time, the cycle time, if you're really shortening  your business process, absolutely. But if you're [25:19] thinking software development, you're probably  not going to shorten that business process. If [25:24] you just stop at your developers, at your software  developers, you're going to have to look not only [25:29] upstream, but downstream, because you all know how  hard it is to roll out new features, right? Oh, [25:36] we have to get user acceptance testing and we  have to get the hands on. How many people do you [25:41] think are out there ready to try out your next  feature? Right. Well, I'm ready. There's going [25:49] to be there's going to be a lot of that because  of AI. There's going to be a whole bunch of we're [25:53] going to roll this out and as long as it doesn't  like mess up the database, try it out and if you [25:57] don't like it, we'll either fix it or we'll get  rid of it because hey, it's it's easy. But there's [26:02] your absorption rate downstream and then there's  elicitation rate upstream that just because you [26:09] shorten it in the middle, you're going to be stuck  asking the same question that AAB was asking. What [26:14] are you going to do with those hours? I mean  the bottom line is you have to have a tangible [26:18] ROI defined whether it is cycle time to get RFP  out quickly or medical device. Yeah. But has to [26:25] be has to be a value chain. Yeah. Can't be a  task because then it just it goes nowhere. So [26:30] this is this is really fascinating actually uh  because I I mean on one hand make more like kind [26:34] of arguing about like you know pedic pedentic  definitions of what is time and whatnot but [26:38] like I think it highlights a really important  dilemma that people within the enterprise have [26:41] to face is like okay like what are we actually  optimizing here for like how are we even going [26:45] to define you know what like what type of time are  we talking about how are we trying to uh what is [26:49] the end goal of building out these systems right  it's like we're not just trying to like virtue [26:53] signal and be like hey we're using AI it's like no  there has to be some sort of AI I AI we're all AI [27:01] So like so I I think you know this conversation  is really highlighting an important dilemma which [27:05] is like really trying to understand like okay you  know for our particular organization for our needs [27:08] like what are we actually trying to to optimize  here for and and and like the the the answer to [27:13] that question is not always as easy as it might  be it's like oh we want to be more productive [27:16] like what does that actually look like right um  I want to pivot here uh a bit uh a bit more to [27:22] talking about some of the um some of the actual  implementation and the stack and the tools and [27:26] things of that nature um that you guys are using  that you guys have been experimenting with and [27:31] uh maybe a we'll start with you because  pharmaceuticals is obviously a a very heavily [27:35] regulated industry uh this is not you know Silicon  Valley move fast break things type of uh business [27:40] that you're in um thankfully that's probably you  know a good thing um but maybe talk through you [27:46] know how you when you're looking at as far as you  know the nuts and bolts of actually implementation [27:51] like model selection like how we're going to you  know what types of harnesses we're going to be [27:54] using what types of what this playbook is actually  going to look like how does um that look like [28:00] inside of you know an organization that has very  kind of very firm guard rails on the compliance [28:05] side. So maybe I'll have you take that question.  Would love uh you guys to weigh in as well. No, [28:09] I'll quickly run that. It's a very very important  question for us for us means for Vortex but also [28:13] for industry. So yeah, we are a very highly  regulated company because I mean our industry [28:17] because we save lives. I mean we are this is not  just losing money or having a black quarter in [28:23] the business. It's actually you know you you are  you're impacting patients patient safety. So it's [28:28] very important. So here's here's a key things  that goes in playbook. Number one is we spend [28:34] enough time to make sure that our data platform  and our data is actually clean and because if [28:40] the bad data then bad outcome and it just creates  down downward impact big time. So we spend enough [28:46] enough time preparing our data investing in data  stack and data platform to make sure that we have [28:51] all our R&D commercial supply chain manufacturing  and our financial data all properly curated and [28:57] available to us. Second thing we did was build  a responsible AI policy and we trained every [29:03] vertexian. Every employee actually went through  a training understanding what the policy is. It's [29:07] a living document and it's basically making sure  that all the legals laws compliance and everything [29:12] is followed through. So every model that we  build, every AI capability we're building is [29:18] falling through building the AIGC, AI general  council, this is comprises of privacy people, [29:23] HR, legal, cyber security, technology architects,  all these people together and they are taking care [29:30] of every single registry. So every model built in  um every AI capability that's built in in Vertex [29:36] goes through this approval process to make sure  there are no fallouts. There is no something that [29:40] we're going to end up paying later or creating a  business problem. Third thing is that we created [29:47] a secure LLM agentic framework structure with  pre-trained model within vortex. So that means [29:54] every information that that's there never leaves.  less that's other thing for this just governing [30:00] like today I'm having this discussion and talk  everything I'm saying to some extent obviously I'm [30:04] not reading a script is being is being actually  reviewed by policy people and and corporate [30:11] compassion because we just cannot go ahead and say  things outside which is not allowed by by laws and [30:16] government governance so forth so series of pieces  then comes the technology stack look technology [30:21] stack is still growing so one thing I have to say  if you are in the in the process of implementing [30:26] no lockins. I think this is the time when you do  want to make sure that whether whether your LLM [30:31] models that you are whether it is the kind of any  foundry or agentic framework that you're building [30:36] your agents on it or whether you're putting  any telemetry and security aspect of it. Make [30:41] sure you no lockins. Finally, we have created  immense amount of telemetry in our system which [30:47] means making sure that we know exactly where  who is using it for what purposes. We can audit [30:52] everything and actually have an understanding  what's what's going on there. so that there are no [30:57] security or compliance issues in the organization.  So it it took a lot of work but we and it took [31:03] some time but we just went through this whole  thing to building it's still a learning process. [31:06] We're still kind of figuring out are areas like we  just saw what hugging face what just happened like [31:11] your own just it just escaped. So that created  like a like almost nightmare situation for me. [31:16] So I told my CEO I said look we need to find out  way that all the non-production environment has [31:21] to be perfectly locked down. these agents cannot  escape and just start impacting the production [31:25] environment because they are capable of it. So  every day is a learning and then everything you go [31:29] back and you update your your policy, you update  your structure, you up your technology stack and [31:34] so forth. So still it's a lot of work in progress.  Uh and maybe I'll take it next. We're a we're a [31:39] public company. Uh and as a result of that, we  have, you know, certain regulations that we need [31:44] to to abide by. Not quite as stringent as AOM, but  it a lot of what he said is true for us as well, [31:50] but with a slightly less uh kind of serious uh  albeit that regulation is serious, but without [31:56] that quite that level of intensity, if I could  describe it like that. And we approach it in in [32:01] somewhat similar aims. We have an IT steering  committee that in effect is made up of legal [32:06] uh governance risk and compliance or cyber guys.  My team which is the innovation team uh our seuite [32:13] uh a representative from our seuite and so same  situation all of our people have responsible AI [32:19] policies. It's a living document. Similarly  we actually force them into training every [32:24] quarter so that they acknowledge they've reread  and understand it. It's updated. Uh we have my [32:31] unit sits off slightly to the side. We have our  own sandbox. We're completely separate from our [32:36] corporate uh environment and that's where we get  to play. Uh and genuinely that's a lot of the fun [32:41] uh things we get to do. We get to experiment.  We get to try tools from you know off the shelf. [32:46] We develop stuff ourselves. And it's really only  when something begins to show promise that we then [32:52] start going through the process of what's required  from a legal perspective from our cyber team, [32:58] governance, risk and compliance. And it's only  then that we and we have a a pretty regimented [33:03] process to bring something behind the walls  into production. Uh but at least that gets us [33:07] to move quickly and trial a lot of things  without it having a an impact on our on [33:13] our business. And so that's really, you know,  all the other things you've said spot on. Um, [33:17] but just a few little twists on it for for our  particular company. I'm going to spin it back to [33:22] you guys real quick. Uh, so I'm going to talk to  you a little bit about the stack I think you guys [33:26] should be building. Uh, so when we talk about  these workflows that we've been talking about, [33:31] uh, the first part of these workflows you really  need to understand is your capture. Uh, so how [33:35] are you going through your business? How are you  finding that business intelligence and capturing [33:39] what kind of business intelligence you're building  within your organization? And that's sitting with [33:44] how your employees are using things. So how they  use Salesforce, how they use uh Service Now, [33:50] how they use whatever these different uh  applications are. Capturing is capturing that [33:55] business intelligence. From there, you have to  map it. How do I take that business intelligence [34:00] and then map that back to what we're actually  building within our knowledge layers themselves, [34:05] okay? Within the workflows themselves. From there,  you start creating ontologies. You start creating [34:12] uh artifacts. those feed into your agentic  process, your orchestration tools that agent [34:19] orchestration will then take that right and  be able to start building what we would call [34:24] responses to these workflows. So to me, that's the  operating system that you're running. And we talk [34:28] about judgment a lot, judgments and every one of  these pieces going through. Now on the other side [34:34] of it, that's just the workflow part. Then this  thing goes out into the real world. So now you [34:39] have to have the evaluation part. So now you're  talking about the evaluation system. I think of [34:43] agents a lot like I think of my kids. Okay, I've  got teenagers. I don't know if anyone else in here [34:47] has dealt with this before. Uh but I give my  teenagers a really safe environment to learn. [34:52] At least I think it is. Uh Andrew Wing talks  about, you know, the data is the food for AI, [34:58] right? I feed my kids the most nutritious data.  Uh and so they have the the greatest data sets and [35:03] ready to go. I give them the KPIs. I give them the  uh the core, you know, operating procedures. Then [35:08] they go out into the world and then they come back  and they're like 67 and then they're like I'm like [35:15] what's going on right now? And then they're like,  you know, eating food I've never seen before. I've [35:20] had my kids literally wearing an outfit leave in  the morning, come home in a completely different [35:24] outfit. And I'm like, how'd that happen? Where  did that come from? And I don't think I approve [35:28] of said outfit, right? And so understand this is  not software that you're setting and forgetting. [35:33] This is a living, breathing environment. And  so your stack needs to capture the workflows, [35:40] but then make sure you're building the evaluation  system to keep them within these guard rails that [35:45] these very smart people are talking about. Uh, and  so everyone will build slightly different stacks [35:50] based on your businesses. But that's kind of the  playbook that I want to make sure you guys got [35:55] today from maybe your from a maybe a different  lens thinking about it from your guys' might be [35:59] the most memorable line of the whole conference.  Yeah, just it's like almost like treating agents [36:03] like FTES, digital employees actually, right?  I mean, and and so not just so making sure that [36:09] you are evaluating the skills, the validation  and also the outcome because these things are [36:14] definitely improving and doing stuff that may not  design for it. So I think having an eye on it. So [36:19] you are almost like treating them your digital FDA  pretty much. So the same process that we created, [36:25] HR process that you created, you have to create  for these that's what we call governance, [36:29] you know. like like you know ultimately so I've  gone through four of those which is why this hair [36:35] is gray at this point um and and what these guys  are saying it it boils down to what how we how you [36:42] would raise a a kid honestly um not all workflows  are created the same not all kids are created the [36:48] same and it's all about risk okay whether you're  talking about uh software has software processes [36:54] or business workflows or AI agents it's the  same thing that the riskier it is, the more [37:00] you want the governance on it. Okay? I I I grew  up in the medical device field for 23 years. So, [37:07] and so we're kind of related there. Uh it's  all about risk. The the the worse the impact, [37:12] the more you got to control it, the more you want  to put boundaries around what your agents can do. [37:18] Awesome. Um we might have time for a couple  of questions if there are audience time. Oh, [37:23] no audience question. Yeah. Be right here  in the in the front here. Got time for one [37:27] question. All right. talk fast. Thank you. Hi. Uh  thanks again and great conversation. I'm coming [37:35] from the R&D of a medical device field. So I  understand the pain point and literally went [37:40] through this in past few weeks. Question to you  is that is there something specific in terms [37:45] of the guard rails. You mentioned policies,  you mentioned things being placed in place, [37:49] but that's usually a more processoriented. You  can't control developers by doing that. Yeah. [37:55] So I just want to understand a little bit more  if there is something in a framework that you [37:59] put together for them that safe to play use it  use AI as a playground there. That's right. So [38:04] the look policies you're actually right just  just giving policy to people and training them [38:07] is not going to actually prevent people to  make mistakes. So we build this centralized [38:12] AI platform. So what we call Vortex AI digital Va  Vida which is Vortex AI digital assistant uh pro [38:19] uh platform. Every employee has access to this.  So we are not letting people just build their [38:24] own platform capabilities. We are building this  actually a space where there are enterprise agents [38:29] that everybody can leverage and there is space  for them to create their personal agents again [38:33] monitored but then every other agent whether  it's a scientific research agent or whether [38:37] it is agent for HR or agent for procurement or  anything everything else is built on that platform [38:42] which is we monitor it and we making sure that  actually nobody is leaving those guards outside. [38:48] So it is a very tightly run because we we do  believe that we just don't have a time or we [38:53] just can't afford people to make mistake in our  in our field. So it is a highly governed highly [38:58] managed platform. Um and and as we getting better  at it we might start decentralizing it just for [39:04] the speed of it but right now it's a very highly  governed. All right so I couldn't agree more. Um [39:10] our AI implementations yes they save a lot of  time. Yes, that's great. But the core value [39:14] is new capabilities, things that we couldn't do  previously. It comes with a problem. Out in the [39:20] market, we can evaluate a new capability by what  people are willing to pay for it. When we have [39:24] an internal solution, how would you recommend  we attach a value to that new capability? So I [39:32] I I can start with I mean you still have to go  through the whole process again. So my my issue [39:36] in the very beginning around workflows and why  you want to have a baseline, right, is because [39:40] you want to have quality you can compare it  to. If you can't compare it to something, [39:44] it's very difficult when you do experiments,  which is what you're talking about ultimately [39:48] is an experiment. It's not something you know  how to do yet. And so what you're really doing [39:52] there is vibes, you know, like like you're working  with a stakeholder on the other side who's like, [39:56] I want to build an intelligent system that  creates news and it reads the news reports and [40:01] then is able to build all of these other things.  And so from there, then you have to actually go [40:06] do the work and build the workflow and really talk  about what would success look like in all of these [40:11] things and spend the time that you would if it  was a real process internally. Again, if not, then [40:17] you're just shaking the magic eight-ball and and  and hoping for the best. The reality is you have [40:21] to do the mapping. You have to really build out  what it is that you're doing. And it can't just be [40:26] one stakeholder with like one idea thinking about  it. It's got to be like a real project that's got, [40:32] you know, a lot of thought into it or else again  it turns into the vibes pretty quick. And I just [40:36] add pick a repeatable workflow that is frustrating  to current people that work for you. And if you [40:42] go and and talk to them, they'll tell you very  quickly why they're frustrated. And as soon as [40:46] you work out what why, and it's repeatable, you  can go with the uh the the quality that you know, [40:53] you reduce risk and you speed the life cycle up.  And I'll I'll throw 30 seconds on top of it. The [40:59] biggest problem with value that I see with a lot  of companies is we try to turn value into money. [41:06] Okay, value. How many dollars does this make for  us? Guys, no. Value is about business goals. How [41:12] much will knowing this thing or having this  capability or being able to do this project [41:17] move us to our business goals? If it moves us  a long way, it's big value. It's it's large, [41:22] high value. If it doesn't move us much, it's low.  And if it doesn't move us at all, it's no value. [41:27] Don't do it. Right? The business goals are the  things that bring the money in. As soon as you [41:34] try to translate that down into the projects, it  rips itself apart. The only part that I would I [41:39] would push back or add to what you're saying,  it's not push back, but like is the adoption. [41:44] So like where's adoption going to be in this thing  that doesn't exist today in your company? So it's [41:48] it's like I don't if if that workflow doesn't  exist, like who's going to take it and actually [41:52] do something with it, right? And so that adoption  rate is probably what I'm most scared of, right? [41:56] Is is how are you building adoption if it's not  already in there? That's absolutely yes. Totally [42:00] agreed. Great. You guys have been a fabulous  group. Thank you. Warmed up here, but we have [42:06] part two this afternoon. But thanks everybody for  for the opportunity. Thanks for hanging out with [42:11] us this morning and we'll be around up here after  if you want to come chat with any of our panels.