Lyft, Pinterest Build In-House Platforms as 90% of GenAI Fails

90%生成式AI项目折戟，Lyft与Pinterest自建内部平台破局 90%的企业级生成式AI部署以失败告终，原因是机构缺乏严格的评估框架和标准化数据血缘来支持生产规模的工作负载。 本次讨论探讨了企业基础设施策略如何在复用杠杆与自主控制之间取得平衡，对比了Lyft和Pinterest等高吞吐量平台依赖单云架构实现低延迟服务，与金融服务业应对混合数据集成的不同方式。专家们阐述了各自的架构分层，即将用于训练工作负载的Amazon SageMaker等托管服务，与基于Kubernetes的定制引擎相结合，以实现精细化实时推理和长尾延迟控制。对话强调，由于严重的评估瓶颈、验证数据集缺失及治理脱节，约90%的企业生成式AI项目在投产前陷入停滞。为解决这些痛点，发言嘉宾强调应构建隐形基础设施、自动化合规流水线以及标准化的“铺就道路”（paved roads），在赢得开发者青睐的同时保留端到端的数据与模型血缘。 为避免碎片化的可观测性和复杂的网络调优，单云架构仍是Lyft和Pinterest等大型消费级交易平台的标准选择；而受严格监管的企业与金融机构则往往需要混合连接来集成外部合作伙伴的数据。 (2:26) 混合工具策略通过先采用开箱即用的托管方案以掌握运维模式，再随平台成熟逐步向模块化开源标准迁移，从而加速实现业务价值。 (9:55) Pinterest运行着一套包含计算、数据、模型和可观测性的四层机器学习架构，该架构利用带有自定义补丁的KubeRay算子的Kubernetes、内存级Ray与Apache Iceberg数据处理，以及经过强化的Prometheus/Grafana流水线。 (10:34) MLflow等标准追踪框架存在端到端血缘断层，需要通过自研内部平台扩展来将具体的训练数据版本直接映射到生产模型输出。 (12:42) 机器学习平台架构需要在复用杠杆与自主控制之间取得平衡：托管平台（如Amazon SageMaker）为分布式训练提供杠杆效益，而基于Kubernetes的自研运行时则为低延迟推理路由、金丝雀发布和版本回滚提供细粒度控制。 (15:52)

Scaling enterprise machine learning and generative AI requires balancing managed services for rapid leverage against internal platforms for granular control, with operational bottlenecks centering heavily on evaluation, lineage, and developer ergonomics. Cloud Topology and the Control–Leverage Trade-Off (2:26) Infrastructure architecture reflects distinct operating environments. Consumer marketplaces operating at scale, such as Lyft and Pinterest, favor single-cloud deployments on AWS to eliminate bare-metal overhead, avoid duplicated networking stacks, and support spiky, low-latency workloads. For Lyft, dynamic dispatch and pricing rely on tight integration with native AWS services like Amazon Kinesis and DynamoDB, while Pinterest uses a single-cloud approach to avoid duplicating low-level networking optimizations. In contrast, financial services firms like American Century Investments require hybrid cloud connectivity to integrate with custodial networks and third-party data providers. Prove AI notes that running early generative AI experiments on local or on-premise hardware can prevent premature cloud GPU spend. To balance leverage and control, organizations often combine managed services for compute-heavy tasks with proprietary systems for critical paths. Lyft relies on Amazon SageMaker for distributed model training across deep learning and gradient boosted decision trees, but maintains a custom Kubernetes platform for real-time inference, canary deployments, and automated rollbacks. Pinterest similarly extends Kubernetes with modified KubeRay operators to surpass standard 6,000-node limits, and combines Ray with Apache Iceberg for in-memory feature processing. The Generative AI Production and Evaluation Bottleneck (21:12) Deploying generative AI into production introduces severe validation challenges. Citing findings from Prove AI and a March 2025 IDC report, roughly 90% of generative AI initiatives fail to reach production despite organizations investing an average of $5 million to $10 million. Panelists attribute this attrition not to an absence of software tools, but to the operational difficulty of building evaluation datasets, integrating fragmented toolchains, and finding specialized evaluation talent. Off-the-shelf observability and 'LLM-as-a-judge' methods introduce black-box opacity that fails to meet regulatory and explainability requirements. Prove AI's Greg Whalen emphasizes that teams often believe they are near deployment before discovering significant testing and data gaps, expanding launch schedules from one month to six. Because application logic in generative models is dictated by underlying data quality, panelists emphasize shifting mindsets to treat data pipelines with the same engineering rigor, versioning, and test suites as production software. Lineage Tracking and Feature Serving Parity (12:42) Standard open-source tools often expose operational gaps during live production incidents. While MLflow is widely adopted for experiment tracking, it lacks end-to-end lineage connecting training data versions, pipeline steps, and model outputs. Without internal extensions, diagnosing production incidents forces engineers to manually reconcile metadata across disparate systems, driving up triage times. Production systems also face training-serving skew, where online inputs drift from training baselines. To eliminate this at Lyft, the platform team deployed a centralized, self-service feature serving architecture. Data scientists define feature configurations once, and the platform automatically deploys them to both offline batch pipelines and low-latency streaming endpoints. This unified ingestion enforces consistency and protects tail latency (such as p99 spikes), where millisecond-level delays in ride dispatch could lead to passenger cancellations and churn. Developer Ergonomics and Platform Adoption Strategies (29:45) Platform success depends on developer adoption rather than top-down mandates. Infrastructure teams establish 'paved roads'—curated starter templates, integrated Jupyter notebooks linked to GitHub, and managed feature stores—that eliminate manual configuration tasks like writing boilerplate deployment files. By embedding governance, audit tracking, and telemetry directly into these tools, compliance becomes less burdensome for developers than bypassing official workflows. When executing migrations, such as onboarding teams to Amazon SageMaker, platform teams mitigate risk through targeted pilot programs and side-by-side benchmarking. Concurrently evaluating legacy and new execution paths against existing service-level agreements provides measurable validation. Shared documentation, office hours, and joint on-call rotations between machine learning engineers and core software engineers further resolve early operational friction, such as restrictive permissions or inconsistent logging schemas. The panel emphasizes that MLOps maturity is ultimately measured by predictable deployment cadences and rapid weekly iteration rather than the breadth of tooling catalogs. However, reconciling proprietary governance requirements with rapidly evolving generative AI stacks remains an ongoing engineering challenge.

在规模化机器学习与大语言模型运维（MLOps/LLMOps）的实践中，技术团队正从盲目拼接工具转向平衡工程杠杆与底层控制权的务实架构。Lyft、Pinterest、美世纪投资（American Century Investments）与 Prove AI 的专家一致认为，企业必须依据业务场景硬性约束确立基础设施形态，通过托管通用计算、自研关键服务与沉淀科学评估体系，消除阻碍模型转化为实际商业价值的生产瓶颈。 基础设施抉择：单云扩展与混合集成权衡 (2:26) 面对单云、多云及混合云的技术路线选择，高并发实时平台与传统受规管机构呈现出截然不同的工程取向。Lyft 的 Rachita Naik 与 Pinterest 的 Balaji Muthazhagan 均指出，面对数以百万计的高峰期实时请求与微秒级服务响应，单云模式（如全面依托 AWS）能够消除自建裸机与跨云打通底层网络的繁重运维负担；多云架构往往会迫使工程团队在不同云厂商之间重复构建网络栈、监控观测与性能优化逻辑，大幅拖慢系统效率。Pinterest 引用 OpenAI 将计算集群从 2,500 节点扩展至 7,500 节点的实践说明，超大规模集群极度依赖底层网络流量精细调优，单云是支撑突发性实时流量的最优解。 然而，来自资产管理领域的 Kevin Kelley 提出，金融行业因必须对接外部托管机构、第三方数据供应商及各类 SaaS 平台，混合云交互在数据共享场景下不可避免。Prove AI 首席技术官 Greg Whalen 则提醒处于早期探索阶段的团队，盲目绑定单一云厂商的专有工具容易陷入技术锁定风险；在尚未明确产品与市场契合点（PMF）之前，直接在云端不受限制地调用昂贵 GPU 极易迅速耗尽研发预算，利用本地或私有算力进行小规模原型验证往往更具经济效益。 架构控制与杠杆：托管训练与自研服务分工 (10:34) 成熟的机器学习平台普遍采取“分层解耦”架构，在通用算力上利用托管方案换取杠杆，在关键业务链路上保留自研工具以确保精细控制。Pinterest 将平台划分为四层：在计算层基于 Kubernetes 与 Ray（辅以定制的 KubeRay 算子）突破标准 6,000 节点的集群上限；在数据层利用 Ray 与 Apache Iceberg 开展内存即时特征处理，淘汰耗时的离线 Spark 批处理；在模型与监控层，基于 MLflow 扩展端到端的数据与模型血缘追踪，并对高基数监控指标下的 Prometheus 进行定制修补以防止崩溃。 Lyft 同样在离线训练与在线推理之间设立了清晰边界。Naik 介绍，Lyft 采用 Amazon SageMaker 统一编排分布式深度学习与梯度提升决策树（GBDT）的离线训练，但在实时推理侧选择自研基于 Kubernetes 的部署系统，以实现严格的灰度发布、影子流量测试与快速回滚。由于网约车平台的派单与定价若出现毫秒级延迟毛刺（尾部延迟 p99），便会导致用户取消订单流向竞争对手，通用托管平台的平均延迟指标无法满足严苛的 SLA 要求，企业必须自建低延迟特征服务与服务网格以掌控核心生命周期。 生成式 AI 落地瓶颈：科学评估与数据范式 (21:12) 在生成式 AI 与大模型落地过程中，系统评估体系的缺失已成为阻碍项目投产的首要结构性瓶颈。Whalen 援引 Prove AI 与 IDC 于 2025 年 3 月发布的调研数据指出，尽管企业在生成式 AI 上的平均投入高达 500 万至 1000 万美元，但由于缺乏完善的评估机制、测试数据集以及专业评估人才，约 90% 的生成式 AI 项目最终未能成功推向生产环境。许多团队在原型构建完成后，由于缺乏衡量输出安全与生成质量的方法，导致原定一个月的上线计划拖延半年以上。 专家强调，单纯依赖“LLM 作为裁判”（LLM-as-a-judge）的黑盒评估方案缺乏可解释性，无法满足严格合规领域的监管审查要求。工程组织必须经历认知转变，将离散的经验性测试升级为科学化的基准测试套件。同时，企业必须确立“数据即应用”的核心范式：在生成式模型中，数据的质量、清洗与编排直接决定了下游应用的最终逻辑，团队必须将管理生产级代码的严谨版本控制与审查流程，同等应用到数据管道与黄金测试集的构建中。 平台采纳与治理：铺平道路降低开发者摩擦 (29:45) 高效的 MLOps 平台应当将底层基础设施对于数据科学家完全透明化，避免开发者陷入编写部署 YAML 文件等非核心业务消耗。Lyft 通过自服务集中特征服务，支持数据科学人员“一次定义、两处部署”，配置自动同步至离线批处理与在线低延迟流式管道，消除了因代码两套实现引发的训练与线上服务特征漂移问题。此外，将 Jupyter Notebook 直接集成至 GitHub 自动化工作流与集中特征库，有效防止了离线私有数据变换带来的合规漏洞。 在推动全公司范围的技术栈迁移与标准化规范时，单纯依靠强制命令往往导致执行阻力。最佳推广策略是“铺设标准化道路”（Paved Roads），提供易用的统一 CLI 与图形界面，并通过平台内置自动化处理合规元数据和审计追踪，使遵循标准流程的工程阻力显著小于绕过规则的自建脚本。在迁移至新服务（如 SageMaker）过程中，平台团队应先选取内部种子团队开展小规模并行验证，以旧链路与新链路对照基准指标评估 SLA，并通过跨部门共享知识库与轮值机制构建协同信任。 衡量 MLOps 平台成熟度的关键标准，在于能否建立可持续、低摩擦且支持按天或按周高频更新的生产交付闭环，而非工具链清单的复杂程度。然而，如何有效降低多机构数据协作中的跨云壁垒，并在大模型生命周期中建立兼具高透明度与学术严谨性的标准化评估基础设施，仍是业界亟待攻克的共同难题。

## Background 该视频由人工智能行业平台 Ai4 于 2025 年 8 月 24 日发布在 YouTube 上。内容为一场汇集了来自 Lyft、Pinterest、美世纪投资（American Century Investments）及 Prove AI 等企业技术领袖的专题圆桌论坛，重点探讨规模化机器学习运维（MLOps）的最佳实践、端到端数据与模型流水线管理、云基础设施架构权衡以及生成式 AI 生产落地的核心挑战。 ## TL;DR 圆桌深入剖析了工业级 MLOps 架构的选型权衡：Lyft 与 Pinterest 坚定采用单云架构以换取极限网络调优与低延迟，而金融机构和工具厂商则因合规与多源协同需要适配多云及混合云模式。嘉宾们指出当前 90% 的生成式 AI 项目因评估（Eval）困难而卡在上线前夕，并直言开源工具在数据血缘与业务治理模型上存在断层。为此，构建顶级平台必须打造以特征一致性引擎、弹性异构算力调度和“隐形基础设施”为核心的铺平道路（Paved Roads），将模型迭代速率作为终极业务指标。最后，讨论提出了“数据即应用”的全新组织认知，并通过速赢验证与渐进式试点实现无摩擦的平台迁移与合规落地。 ## Atmosphere & style 整场研讨会专业硬核且极具实战气息，嘉宾们毫不避讳地指出流行工具与主流云厂商方案在应对高并发、毫秒级尾延迟和复杂治理时的局限性，激发出强烈的行业共鸣与默契笑声。主持人的控场张弛有度，在宏观架构策略与微观工程踩坑之间自如穿梭。观众能够清晰感受到来自不同行业领军团队在海量流量冲击与严苛监管约束下的理性思考，整场对话既充满高密度的技术干货，又饱含对开发者体验与组织变革阻力的深刻洞察。 ## Key takeaways - [02:28](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=148s) 高并发在线预测系统倾向于选择单云架构以避免跨云网络延迟损耗与碎片化可观测性，而受监管行业由于外部合规数据对接常被迫采用多云或混合架构。 - [07:47](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=467s) 在架构选型中切忌“先选云后选工具”，初期探索阶段通过本地或单机环境低成本试错，待业务成熟后再收敛到公有云平台以获取规模效益。 - [09:55](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=595s) 机器学习平台采用“开源底座 + 现成商业方案”的混合选型策略，能够缩短早期价值实现时间（Time to Value），并在成熟后平滑迁移至自建开源方案。 - [10:34](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=634s) Pinterest 搭建了计算（K8s/Ray）、数据（Iceberg/Ray）、模型（定制 MLflow）与观测（定制 Prometheus/Grafana）四层高吞吐 ML 基础设施。 - [12:42](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=762s) 原生 MLflow 难以直接提供端到端的数据与模型血缘关系（Lineage），无法直接解答“特定输出由哪一份具体数据训练得出”的核心业务问题。 - [13:53](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=833s) Lyft 采取了分治策略：在模型训练环节利用托管的 AWS SageMaker 降低作业编排负担，但在推理部署环节自研平台以满足极致低延迟与精细发布控制。 - [21:12](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1272s) 生成式 AI（GenAI）项目投产面临高达 90% 的失败率，评估（Eval）体系构建困难、缺乏高质量评测集及黑盒机制不可解释是核心瓶颈。 - [24:36](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1476s) 理想的 MLOps 平台应实现“隐形基础设施”，让数据科学家免于调试复杂的 YAML 文件，专注于模型算法与实验本身。 - [26:39](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1599s) Lyft 通过自研中心化特征服务基础设施，统一定义并双向分发特征至离线批处理与在线推理，彻底消除训练-推理特征偏差（Training-Serving Parity）。 - [26:39](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1599s) 出行双边市场对 P99/P99.9 尾延迟（Tail Latency）极度敏感，毫秒级毛刺会直接导致订单取消和用户流失，系统必须针对尾延迟进行深度优化。 - [29:45](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1785s) 顶级 MLOps 平台的三大支柱是弹性算力（异构资源与碎片化 GPU 调度）、开发者人体工程学（铺平的道路）以及严格的可观测性与治理。 - [35:14](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2114s) 生成式 AI 时代要求组织彻底转变思维，认识到“数据本身就是应用程序”（Data IS the application），像维护核心代码一样维护数据质量与资产。 - [39:05](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2345s) 在强监管行业中推行规范需反转摩擦力，通过自动化接管合规与审计流程，让遵循平台规范比绕过规范更加省时省力。 - [39:54](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2394s) 推动内部平台采纳应采取“速赢”（Quick Wins）打法，通过新旧链路并行对比和一页纸 SLA 确认，以低风险、高收益说服业务团队平滑迁移。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=4s) 圆桌嘉宾介绍与开场 圆桌主持人 Chris Brousseau 首先登台致辞，介绍自己是《LLMs in Production》一书的作者，目前在 Veox 担任 AI 副总裁。他带着一丝神秘感透露，团队正在研发一种新型的人工智能体系，虽然目前仍处于隐蔽研发阶段，但次日就会正式对外公开发布。随后，他邀请台上的四位嘉宾依次做简要的自我介绍。 ```mermaid flowchart LR accTitle: Panelist roles and core focus areas subgraph S1["平台与技术供应商"] G["Greg Whalen<br/>Prove AI CTO<br/>(GenAI/LLMOps 评测与治理)"] C["Chris Brousseau<br/>Veox AI 副总裁<br/>(圆桌主持 / LLM 落地)"] end subgraph S2["大规模互联网企业"] R["Rachita Naik<br/>Lyft MLE<br/>(实时调度/ETA/司乘匹配)"] B["Balaji<br/>Pinterest 工程师<br/>(广告召回/检索/打分工程)"] end subgraph S3["传统金融投资"] K["Kevin Kelley<br/>American Century Investments<br/>(AI/ML 标准化与中台赋能)"] end ``` Prove AI 的首席技术官 Greg Whalen 首先发言，他介绍 Prove AI 是一家专注于生成式人工智能（Generative AI）的平台型工具厂商，核心使命是帮助企业跨越落地障碍、更快地将 GenAI 应用推进至生产环境。平台重点聚焦在模型评测（evals）与数据管理两项关键能力。接着，来自美国中西部金融服务机构美世纪投资（American Century Investments）的 AI/ML 赋能与创新主管 Kevin Kelley 介绍了团队的定位：他们致力于为整个企业标准化 AI 开发与部署的工具链及底层架构，目标是让内部的数据科学家和业务开发者从繁琐的基础架构运维中解脱出来，把精力聚焦在业务成果本身。 随后发言的是 Pinterest 广告投放平台（Ad Serving Platform）的软件工程师 Balaji Muthazhagan。他形象地指出，用户在任何客户端设备上刷到的 Pinterest 广告，背后都由其团队维护的底层系统所驱动；从海量广告库的检索、索引构建到机器学习模型在线打分，整套高吞吐链路的构建、部署与全天候维护都由他们团队承担。最后，Lyft 调度团队（Dispatch Team）的机器学习工程师 Rachita Naik 介绍了网约车场景下的严苛挑战：团队主要研发用于实时行程预估（ETA Estimation）、乘客与司机高效撮合匹配的核心预测系统，以维持整个出行双边市场的高效动态平衡。在正式切入正题前，主持人特别呼吁全场为本次活动的赞助商 Prove AI 送上掌声。 --- ### [02:28](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=148s) 云架构选型之争：单云、多云与混合云 主持人 Chris 抛出了机器学习工程领域最核心的架构决策难题：在单云（Single Cloud）、多云（Multi-cloud）、本地部署（On-prem）与混合云（Hybrid Cloud）之间，各位专家所在的组织究竟是如何取舍的？ Lyft 的 Rachita Naik 率先分享了网约车业务的解法——Lyft 坚定不移地选择了由 AWS 深度支持的“全单云战略”（All-in on a single cloud）。她解释道，Lyft 本质上是一个超大规模的实时动态市场，每周必须承接数以百万计的乘客与司机请求，全天各时段的每秒查询率（RPS）呈现出极度剧烈的峰值波动，同时系统对延迟有着极其严苛的要求。在早期评估架构时，团队认定单一云厂商能提供最直接的弹性扩缩容能力，免去了团队自行打理底层硬件与运维基础设施的巨大包袱。目前 Lyft 的 ML 基础设施深度绑定 AWS 生态，从基于 Amazon Kinesis 的流式数据管道到用于存储的 DynamoDB 均实现了紧密集成。尽管他们曾审慎评估过多云和混合云方案，但最终发现跨云带来的“组件重复度过高、系统复杂度激增以及可观测性断层（fragmented observability）”，其弊端远大于潜在的冗余收益。 ```mermaid flowchart TD accTitle: Cloud Strategy Trade-offs subgraph Single["单云架构 (AWS / 核心业务)"] S_Pro["优势：深度网络/硬件优化<br/>统一可观测性栈<br/>应对突发高并发 (Spiky RPS)"] S_Con["隐患：厂商锁定风险"] end subgraph Multi["多云 / 混合架构 (金融合作 / 本地实训)"] M_Pro["优势：数据跨机构合规交互<br/>本地离线 GPU 成本可控<br/>灵活适配各类专业工具"] M_Con["代价：网络/监控重复建设<br/>碎片化可观测性<br/>极高集成维护成本"] end Single -->|"Lyft / Pinterest：追求极限吞吐与低延迟"| Output1["极致工程与单一管控"] Multi -->|"金融服务 / 初创探索：追求合规与成本防线"| Output2["多源适配与本地摸索"] ``` Pinterest 的 Balaji 对此深有共鸣。他强调多年的工业界实践证明，单云方案在大规模工业级应用中表现最为卓越，能够支撑极深度的工程优化。为了进一步印证观点，他援引了前一天读到的 OpenAI 技术白皮书：OpenAI 在 2021 年将自身云基础设施从 2,500 个节点扩容到 7,500 个节点的过程中，绝大多数关键技术优化都深入到了网络底层流量调优层面。如果企业选择多云厂商，就必须在每一个云平台上将网络调优、遥测与可观测性链路等“苦活累活”全部重新造一遍轮子，工程代价呈指数级倍增。Balaji 提到，自己早年在加州大学圣地亚哥分校（UCSD）的研究实验室里接触过不少混合云与本地 GPU 集群，那种架构之所以可行，纯粹是因为学术模型的训练负载极为平稳；面对工业界秒级变化、脉冲强烈的实时在线预测，本地或混合架构往往捉襟见肘。 金融行业则展现出了截然不同的现实约束。来自美世纪投资的 Kevin Kelley 表示，虽然他们在自研算法和自主应用架构上也尽量坚守单云原则，但在金融服务领域，企业高度依赖外部行业合作伙伴、数据服务商及特定共享平台。“很多时候我们别无选择，跨云和混合集成是不可避免的现实，”Kevin 坦言。主持人 Chris 随即幽默地插话调侃道：“所以是你们把这令人头大的麻烦（headache）全揽下来了？”Kevin 笑着回应，虽然谁都不想要这种麻烦，但涉及跨机构数据分发与平台对接时，多云集成属于绕不开的硬性刚需。 Prove AI 的 CTO Greg Whalen 则从技术提供商的宏观视角给出了不同维度的警告。由于 Prove AI 服务的客户遍布各大云平台，其产品天然必须支持多云、混合云及私有化部署。针对台前两位大规模成熟企业专家的“单云论”，Greg 提醒那些刚刚踏上 GenAI 探索之旅的中小团队切勿生搬硬套。他指出初学者最常踩的一个致命误区是**“先选云，后选工具”**——很多公司因为此前与某家主流云厂商签订了巨额采购合同，便理所当然地要求 AI 团队直接使用该云厂商自带的一揽子套件，结果发现那些针对特定应用场景真正高效的工具根本无法在该云环境中顺畅运行。此外，当下云端 GPU 算力极其昂贵，除非企业财力雄厚到“不在乎每天往外大把烧钱”，否则对于早期探索与实验阶段，借助本地机器（On-prem）或单机环境低成本试错往往是更为务实的选择，待业务成熟后再收敛到单一云平台以获得规模经济与统一可观测性。 - **单云核心优势**：消除跨云网络延迟损耗，打通端到端监控与排障可观测性，极大简化流量洪峰下的弹性调度。 - **混合/多云典型场景**：金融业外部机构合规数据对接、第三方专有服务集成、算法原型期控制 GPU 账单支出的本地实验阶段。 --- ### [07:47](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=467s) 机器学习与大模型工具栈选型及权衡 讨论随即深入到具体的技术工具栈选型。主持人 Chris 列举了当下的主流阵营：云厂商托管全家桶（AWS、Azure、GCP）、一站式商业平台（Databricks、DataRobot 等）以及各类繁荣的开源生态，并要求各位嘉宾精炼地剖析自家当下的技术栈构成，重点说明选型背后的核心考量与架构权衡。 Prove AI 的 Greg Whalen 率先展开阐述。他指出，由于 Prove AI 本身就是一款 MLOps/LLMOps 工具，团队在底层技术选型上全方位拥抱了开源生态。当下行业里最让技术负责人头痛的难题就是“工具选型与编排（tool curation）太难”。很多行业峰会的演示胶片上动辄塞进上百个微型工具图标，企业若在 GenAI 初期耗费数月时间试图将这上百个互不相通的零散组件生搬硬凑在一起，“那完全是在浪费时间，严重耽误将应用推向生产环境的黄金窗口”。 针对封闭式生态厂商对客户造成的供应商绑定（vendor lock-in），Prove AI 采取了基于成熟开源底座的演进策略： - **模型计算与推理生态**：深度依托 PyTorch 以及 NVIDIA 的开源软件栈； - **安全与防护护栏**：基于开源的 NVIDIA NeMo Guardrails 构建安全合规能力； - **遥测与监控层**：全面对接开放标准 OpenTelemetry，确保日志、指标与调用链的可移植性。 Greg 强调，AI 领域的工具和技术框架迭代速度极快，唯有依托最主流的开源标准进行模块化封装，才能在生态发生剧烈变革时保持敏捷，既不锁死技术路线，又能快速为企业客户消除碎片化集成的痛苦。美世纪投资的 Kevin Kelley 对此深表赞同，并指出这与他们团队的思考高度契合：选用何种工具链，本质上取决于具体工作负载的成熟度、整个行业的标准化进程，以及在“开箱即用（turnkey）的商用供应商产品”与“纯自建开源架构”之间寻找动态的平衡点。 ### [09:55](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=595s) 开源与商业软件的混合选型策略 在构建机器学习平台时，采用“开源 + 现成商业套件（Off-the-shelf）/ 供应商解决方案”的混合模式是一种非常务实的路径。这一决策背后的核心驱动力是“价值实现时间”（Time to Value）。从零开始将多个开源组件拼接整合往往需要耗费大量工程时间，而直接使用商业方案能够帮助团队在特定业务场景中快速起量，并通过实践与踩坑积累经验。随着平台建设的成熟、业务规模的扩张以及可观测性数据的沉淀，团队在掌握了充分的信息后，再做出理性判断，将部分工作负载逐步平滑迁移至自建开源方案上。 ### [10:34](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=634s) Pinterest 的四层机器学习基础设施架构 Balaji Muthazhagan Thirugnana Muthuvelan 详细拆解了 Pinterest 的四层机器学习工具链体系： ```mermaid flowchart TD accTitle: Four-Layer ML Platform Architecture subgraph Compute["算力层 (Compute Layer)"] K8s["Kubernetes"] KubeRay["KubeRay Operator<br/>(突破单集群节点上限)"] end subgraph Data["数据层 (Data Layer)"] MemoryCompute["内存/实时计算 (Ray)"] DataLake["Apache Iceberg<br/>(取代 Spark 离线全量预计算)"] end subgraph Model["模型层 (Model Layer)"] MLflow["MLflow 基础引擎"] CustomLineage["定制血缘追踪与平台扩展"] end subgraph Obs["可观测性层 (Observability Layer)"] PromGraf["Prometheus + Grafana"] CustomPatch["客户端深度打补丁<br/>(解决高基数 Metric 崩溃)"] end Compute --> Data Data --> Model Model --> Obs ``` - **算力层（Compute Layer）**：Kubernetes 是标准底座，但社区近期开始直接改造 Kubernetes Operator 本身。例如基于 Python 的分布式计算平台 Ray 以及对应的 KubeRay Operator，使得集群能突破 Kubernetes 原生约 6,000 个节点的上限（如 OpenAI 曾从 2,500 节点扩展至 7,500 节点），大幅拓宽算力扩展瓶颈。 - **数据层（Data Layer）**：过去主要依赖 Spark 进行离线预计算生成静态数据资产和宽表，现在转向利用 Ray 与 Apache Iceberg 进行内存级（In-memory）的即时动态处理。这极大地缩短了模型重训周期，并在加入新特征时显著压缩了实验成本。 - **模型层（Model Layer）**：以 MLflow 为核心基础，针对内部数据科学家和软件工程师的需求构建了一套定制化统一平台。 - **可观测性层（Observability Layer）**：基于 Prometheus 和 Grafana 搭建监控体系。但在超大规模下，高指标基数（High Metric Cardinality）会导致原生客户端崩溃，因此团队必须针对客户端进行专属的底层打补丁与定制改造。 ### [12:42](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=762s) MLflow 的局限性与端到端数据血缘痛点 在被问及为何要在 MLflow 之上二次开发时，Balaji 直言尽管 MLflow 提供了基础的版本追踪，但它依然无法清晰解答一个最核心的业务问题：“**这一个特定版本的模型，究竟是使用哪一份具体数据训练出来的，从而导致输出了这一项特定的预测结果？**” 为了还原完整的模型与数据血缘（Lineage），工程师们往往需要在 3 到 4 个不同的工具之间来回跳转查找。现场嘉宾也对此产生强烈共鸣，笑称“不知道台下有没有 MLflow 的代码贡献者在听，这简直是所有人的痛点”。Pinterest 围绕业务特性和工程师使用体验打造定制工具，核心宗旨始终是“开发者体验优先”（Keeping developers first）。 ### [13:53](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=833s) Lyft 的迭代历程：托管方案与自研部署底座的平衡 Lyft 的机器学习工程师 Rachita Naik 分享了与 Pinterest 高度相似的架构演进路线。针对其实时出行与调度场景，Lyft 采取了分治策略： - **模型训练**：最初运行在纯 Kubernetes 上，但随着模型数量剧增、用户和服务规模扩大，作业编排（Job Orchestration）的维护负担过重。Lyft 随后引入了 AWS SageMaker 托管服务，既简化了分布式训练，又让工程师无论使用深度学习还是梯度提升树（Gradient Boosted Trees 在 Lyft 内部极为流行）都能保持框架无关的灵活性。 - **模型推理与部署**：由于对极低延迟和高并发有严苛要求，Lyft 并没有采用全托管方案，而是在 Kubernetes 之上自研了一套统一的模型部署平台。该平台向开发者暴露一致的标准接口，同时将自动扩缩容、影子部署（Shadow Deployment）、金丝雀发布（Canary Deployment）以及精细的回滚逻辑完全掌控在自己手中。 - **生命周期追踪**：同样结合 MLflow 监控跨生命周期的各项性能指标。 面对主持人关于“为何各家头部公司技术栈趋于同质化”的提问，Rachita 解释道，Lyft 和 Pinterest 等企业本质上都在处理“实时双边市场”（Real-time Marketplaces）的复杂性。技术选型必须在“杠杆效率”与“掌控力”之间取得平衡：用托管方案换取扩张效率，在核心服务链路自研工具以获得细粒度的控制力与透明度。 ### [17:01](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1021s) 云端与本地（Cloud vs. On-prem）架构选型三要素 针对非单一标准技术栈场景下的云端与本地部署决策，嘉宾们分享了差异化的考量维度： Kevin Kelley 指出，在金融资产管理行业，受制于数据交付和数据延迟限制，许多预测负载依然以批处理（Batch）为主，实时链路仅占少数。数据科学家最关心的是特征工程、EDA 和模型训练，对底层怎么跑并不敏感。因此架构的核心策略是“开发阶段赋予自由度，训练、部署与监控全面标准化”，将工程精力集中在真正能产生业务差异化价值的地方。 Prove AI 的 CTO Greg Whalen 则结合大语言模型与生成式 AI（GenAI）的落地场景，给出了优先级排序框架： 1. **问题与业务约束（Constraints）**：数据来源、敏感数据存储合规位置、数据处理管线中已有的安全校验规则等。避免重复造轮子是首要原则。 2. **团队能力边界（Capabilities）**：团队现有工程人员能够驾驭、维护和运营哪些技术方案。 3. **成本考量（Cost）**：在满足上述硬性约束和工程能力之后再进行综合成本核算。 ### [19:48](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1188s) 基础设施选型决策：价值路径与跨云协作 在讨论模型训练和部署的基础设施时，团队往往面临是在本地私有环境（On-Premise）还是迁移到云端的架构决策。讨论指出，决定留在本地还是上云，首要考虑的核心永远是“我们试图解决什么业务问题？”以及“实现业务价值的最快路径是什么？”。在这些决策背后，真正起主导作用的往往是企业已有的存量流程、已编写好的遗留代码和资产，以及面对全新技术栈时团队人才的学习成本。 ```mermaid flowchart TD accTitle: 基础设施与数据架构决策要素 subgraph Drivers["决策驱动因素"] D1["业务核心问题与价值路径"] D2["存量流程与已有代码资产"] D3["多方外部数据协作需求"] end subgraph Challenges["技术落地痛点"] C1["跨云环境兼容性问题"] C2["训练与推理特征分布偏移"] C3["生成式 AI 评估极其困难"] end Drivers --> Challenges ``` 此外，生成式 AI（GenAI）时代的协作模式带来了传统软件开发中从未见过的棘手挑战。例如，在微调模型时，企业往往需要引入外部合作伙伴的数据集。当合作方使用不同的公有云平台（“你们用那家云，我们用这家云，彼此不互通”）时，数据管道和基础设施就不得不重新选型和迁移。这种外部数据协同与云环境兼容性的摩擦，在实际工程中往往比单纯的技术参数更直接地左右着基础设施决策。 - 决定架构选型的首要原则是寻找实现业务价值的最快路径，其次是兼容现有代码和资产。 - 涉及外部数据提供商的微调协作经常触发跨云平台的架构整合痛点。 --- ### [21:12](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1272s) MLOps 平台的最大盲区：生成式 AI 评估瓶颈与 90% 失败率 主持人向现场观众发起提问：“有多少人的公司本身就在对外销售 MLOps 平台产品？”随后将问题抛给嘉宾：无论是作为 MLOps 平台的构建者（如 Prove AI）还是使用企业，当前平台提供商最显著的短板在哪里？ Prove AI 的 CTO Greg Whalen 直言，当前横亘在所有团队面前的最大障碍正是**性能评估（Evaluation / Eval）**，特别是在生成式 AI 领域。许多团队在开发阶段以为项目已经完成了 90%，但一到准备上线生产环境的关头，才遭遇评估这堵高墙——大家没有预见到 Eval 的复杂度，也没有充足的人才储备，导致原定 1 个月内上线的计划被迫拖延至 6 个月。 ```mermaid flowchart LR accTitle: 生成式 AI 落地困境与评估难题 subgraph Investment["行业现状"] I1["500万 - 1000万美元 投入"] --> I2["90% GenAI 项目未能落地生产"] end subgraph Solution["Prove AI 破局方案"] S1["开源 Eval 工具甄选与管理"] S2["评估测试集自动化构建"] S3["专家服务与白盒化可解释性"] end ``` Whalen 引用了 IDC 2025 年 3 月的报告数据以及 Prove AI 自身的企业调研指出：尽管企业平均在生成式 AI 上的投资高达 500 万至 1000 万美元，但有 **90% 的生成式 AI 项目最终未能成功投入生产**。这背后的核心瓶颈正是评估难： 1. 市面上虽然有评估算法库，但实际配置和调优极其复杂； 2. 缺乏构建高质量评估数据集的方法与工具； 3. 团队内部极度缺乏专业的评测人才。 针对现场关于“难道大家对 LLM as Judge（大模型作为裁判）不满意吗？”的讨论，嘉宾指出这种黑盒机制的可解释性（Explainability）严重不足。企业数据科学家需要根据学术前沿指标自定义精准度与召回率，而市面上大多数黑盒化的可观测性产品往往强行要求客户“削足适履”（Square pegs in square holes），一旦业务需要定制化指标就难以支撑。 - IDC 2025 年 3 月报告显示，90% 的生成式 AI 项目无法成功上线投产。 - 尽管企业单项投资高达 500 万至 1000 万美元，但评估（Eval）能力的缺失成为卡死生产部署的最关键卡点。 --- ### [24:36](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1476s) 平台短板：最后一公里治理、隐形基础设施与全链路血缘 来自金融与社交媒体平台的工程师们进一步剖析了传统 MLOps 工具（如 AWS SageMaker、MLflow 等）在实际工业界落地时的局限性。 #### 1. 治理流程与数据模型的“最后一公里”断层 在资产管理与金融场景下，SageMaker 等托管服务虽然在训练实验阶段表现良好，但在模型服务（Serving）和治理摄入（Registry / Intake）方面存在不足。通用平台的数据模型往往无法贴合企业独特的合规与运营流程，迫使团队必须在平台之外开发大量的“外挂插件”（bolt-on / add-on）来处理审批、元数据登记和治理流程，然后再导回平台中。 #### 2. “隐形基础设施”（Invisible Infra）的缺失 Pinterest 工程师 Balaji Muthazhagan 指出，数据科学家的核心精力应该放在模型算法迭代、生产验证与测试上，而不应该整天陷在编写和调试复杂的 YAML 配置文件中。理想的平台应当让底层基础设施彻底“隐形”，让数据科学家能够自助、安全地完成部署、金丝雀测试与实时监控。 ```mermaid flowchart TD accTitle: 理想 MLOps 平台的三大核心支柱 A["真正优秀的 MLOps 平台"] --> B["隐形基础设施<br/>(让算法人员远离 YAML 文件)"] A --> C["毫秒级全链路血缘追踪<br/>(秒级排查 On-call 线上事故)"] A --> D["训练-推理一致性特征平台<br/>(消除分布偏移，兼顾尾延迟)"] ``` #### 3. 实时全链路血缘（True Lineage）缺失 在线上事故发生时，负责轮值排障（On-call）的工程师往往需要耗费数小时逐层回溯数据源、特征处理逻辑与模型版本。如果 MLOps 平台能够原生提供即时、清晰的端到端数据血缘图谱，将极大缩短故障恢复时间（MTTR）。 - **治理断层**：标准 MLOps 工具的数据模型难以无缝契合企业的合规与治理工作流，导致大量外部拼凑逻辑。 - **研发减负**：平台必须抽象底层运维，停止让数据科学家手写 YAML 配置。 - **血缘追踪**：原生端到端 Lineage 是线上快速定位并解决生产 Incident 的核心能力。 --- ### [26:39](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1599s) Lyft 实战演进：特征一致性引擎与极低尾延迟（Tail Latency）要求 Lyft 的机器学习工程师 Rachita Naik 分享了网约车高并发出行场景下的特殊工程挑战。Lyft 在实践中曾频繁遭遇**训练与推理不一致（Training-Serving Parity）**的问题：模型在离线训练表现优异，部署到线上后却频繁出现输入数据分布偏移（Data Drift），导致预测值与真实值之间的残差（Residuals）持续扩大。 为了从根本上解决这一问题，Lyft 自研了一套**中心化实时特征服务基础设施（Centralized Feature Serving Infrastructure）**。数据科学家和工程师只需定义并配置一次特征规格（Feature Configs & Definitions），该系统就能自动将特征逻辑双向推送到离线 Batch 批处理管道（用于模型训练）和在线 Serving 管道（用于实时在线推理），从架构层面彻底保证了特征计算的一致性。 ```mermaid flowchart LR accTitle: 特征定义一次，双向一致性部署 subgraph Def["统一特征定义与配置"] F["Feature Config & Registry"] end subgraph Pipelines["自动化双向管道"] F -->|离线特征抽取| P1["批处理训练管道 (Batch Pipeline)"] F -->|在线低延迟计算| P2["实时在线推理服务 (Online Inference)"] end ``` 此外，网约车业务对模型在线打分的**尾延迟（Tail Latency，如 P99/P99.9）**有着极度苛刻的要求。在用户叫车或司机派单的核心链路中，即便几毫秒（Millisecond-level）的额外延迟，都可能直接导致用户取消订单（Cancellations）、乘客流失转向竞对平台，造成不可逆的业务损失。由于通用的托管型 MLOps 平台大多只关注平均延迟（Average Latency），无法满足极致的高并发尾延迟控制，Lyft 只能选择自建内部的高性能推理平台。 - **特征双向部署**：通过配置驱动的统一特征引擎，同时服务离线训练与在线推理，彻底消除偏差。 - **尾延迟敏感**：毫秒级延迟毛刺会直接引发叫车取消与用户流失，系统设计必须以 P99 尾延迟而非平均延迟为核心指标。 --- ### [28:23](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1703s) 顶级 MLOps 的标准：可复现工作流与全生命周期抽象 在讨论“究竟什么才算世界级 MLOps 平台”时，嘉宾们提炼出了衡量 MLOps 成熟度的黄金法则：**具备可复现的工作流（Reproducible Workflows）与高可用的自助式基础设施（Self-Service Infrastructure）**。 一个成熟的平台必须将数据接入（Data Ingestion）、模型训练（Training）、打包部署（Deployment）、在线服务（Serving）以及观测监控（Observability & Monitoring）全链路打通并进行高度抽象。讨论特别强调了一个分工原则：“数据科学家除了在最初的原型设计阶段外，其余环节都不应该被迫卷入底层的 MLOps 运维泥潭中。”通过在上层提供自动化的监控、漂移检测与运维兜底，才能让算法团队专注于模型本身的探索与创新。 - **核心指标**：工作流 100% 可复现、底层全生命周期自助化。 - **角色解耦**：将复杂的部署、服务治理与监控层全面下沉至平台底层，使数据科学家真正专注于算法原型。 ### [29:45](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1785s) 打造世界级 MLOps 平台的三大支柱：弹性算力、开发者人体工程学与治理 关于什么是“世界级”的 MLOps 产品与团队，讨论首先从从早期混乱、拼凑的部署流程向统一工作流的演进切入。构建顶级平台的核心要素主要体现在三个技术维度： ```mermaid flowchart TD accTitle: 构建世界级 MLOps 平台的三大支柱 subgraph Core["世界级 MLOps 核心要素"] direction TB E["弹性算力 (Elastic Performance)<br/>• 异构计算资源 (GPU/CPU/TPU)<br/>• 碎片化 GPU 调度<br/>• 拓扑感知调度"] D["开发者人体工程学 (Developer Ergonomics)<br/>• 铺平的道路 (Paved Roads)<br/>• 开箱即用的入门模板<br/>• 降低认知与接入负荷"] G["可观测性与治理 (Governance & Observability)<br/>• 实时指标监控<br/>• 杜绝算力浪费与滥用<br/>• 全流程透明可追溯"] end ``` 首要支柱是“弹性算力”（Elastic Performance）。这里的弹性指的不仅是根据负载动态增减节点的能力，更关键的是在异构算力环境（混合使用 GPU、CPU 和 TPU）下最大化资源利用率。系统必须避免昂贵的 GPU 资源处于空闲状态，需要通过碎片化 GPU 调度（fractional GPU scheduling）和拓扑感知调度（topologically aware scheduling）等技术手段，让系统在有限且异构的算力资源下发挥出极高的效能。 第二个支柱是“开发者人体工程学”（Developer Ergonomics）。让算法工程师和开发者在平台上工作得舒适顺畅，能为整个公司带来巨大的杠杆价值。最有效的落地方案是打造“铺平的道路”（Paved Roads）——平台提供标准化的入门模板和最佳实践文档，各业务团队看到现成的异构解决方案和模板后，会更容易被说服并主动采用统一平台。 第三个支柱是完善的治理（Governance）与可观测性（Observability）。当平台向算法团队开放无限实验的能力时，往往伴随着资源被滥用、GPU 预算迅速耗尽的风险。平台必须具备底层的透明度，提供实时的指标监控和消耗追踪能力，确保每一笔算力开销和实验过程都清晰可见。 --- ### [31:56](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1916s) 开发者体验与无摩擦工程化：以产品化思维赋能数据科学家 在具体落地层面，平台建设需要在灵活性与标准化之间取得平衡，不能追求“无限的自由度”而导致运维流程被彻底稀释。为了给数据科学家创造无摩擦的开发体验，平台应当主动适配他们最熟悉的开发工具链（例如深度集成于 GitHub 的 Jupyter Notebook），将代码扫描、合规检查与自动化测试等繁琐的工程细节隐藏在幕后，以“胡萝卜”（正向激励）的方式引导团队遵循标准规范。 ```mermaid flowchart LR accTitle: 无摩擦 MLOps 开发与数据治理链路 subgraph Dev["开发环境与治理隐形化"] A["Jupyter Notebook"] --> B["GitHub 深度集成"] B --> C["自动化代码扫描 & 测试"] end subgraph Data["数据与运维赋能"] D["Feature Store (特征库)"] --> E["高质量受治理数据产品"] E --> A C --> F["开箱即用的运维指标"] end ``` 同时，MLOps 平台必须与企业级数据基础设施深度打通。数据科学团队常常倾向于将外部数据集直接拉取到本地进行随意转换，这会严重破坏数据治理机制。平台应当通过特征库（Feature Store）等标准数据产品，直接向开发者提供预先治理好、高可用、高质量的数据集；并在模型训练与部署阶段自动输出运维监控指标，让数据科学家无需在业务代码中硬编码监控逻辑，真正把精力集中在数据洞察、模型结构与训练迭代本身。 - **防止运维稀释**：提炼少数可复用的标准工程模式，避免为迎合过度定制而导致 MLOps 架构失控。 - **无摩擦体验（Frictionless）**：在用户熟悉的交互界面背后静默执行 CI/CD 与合规检查。 - **数据平台对齐**：通过特征库提供高质量受控数据，杜绝团队在本地进行未经治理的临时数据清洗。 --- ### [33:28](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2008s) 业务视角的终极标准：以迭代速率衡量 AI 管线成熟度 从非技术的业务视角来看，许多客户在提出“想要一个世界级 MLOps 流水线”时，诉求往往模糊不清。剥离具体工具的技术噪音后，评估一条 AI 管线是否达到“世界级”的核心业务指标只有两条：能否快速且平稳地将模型推向生产环境（没有突发的工作量激增与宕机），以及能否在每次上线后以越来越快的速度持续迭代和优化模型。 与其在一开始纠结挑选何种工具，不如以终为始进行逆向设计（working backwards）。如果系统能够支持团队在没有报错、无需额外紧急调配运维资源的情况下，按天甚至按周的频率持续更新并优化模型，那么该系统就已经达成了顶级 AI 流水线的目标。 --- ### [35:14](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2114s) 生成式 AI 时代的变革管理：评测难题与“数据即应用”的思维重构 在 MLOps 规模化落地的过程中，最大的阻碍往往不是技术本身，而是组织变革管理（Change Management）、流程规范制定以及新工具的推广采纳。随着生成式 AI（Generative AI）与模型微调（Fine-tuning）的大规模应用，组织变革面临着两项尤为严峻的挑战： ```mermaid flowchart TD accTitle: 生成式 AI 组织变革的两大核心挑战 subgraph Eval["挑战一：科学化评测体系构建"] A1["业务定义与成功标准"] --> A2["编写真实测试集与用例"] A2 --> A3["建立严谨的科学评测能力"] end subgraph DataMindset["挑战二：数据认知思维重构"] B1["传统思维：数据与应用程序割裂"] --> B2["现代认知：数据质量决定模型上限"] B2 --> B3["核心转变：数据本身就是应用程序<br/>(Data IS the Application)"] end ``` 第一大挑战是**科学评测体系（Evals）的建立**。在生成式 AI 领域，界定“什么是好结果”、“什么是成功”极其复杂。企业必须培养或指定能够深钻业务逻辑的人员，负责构建评测数据集、设计测试用例，并建立严谨科学的评测流程。作为外部供应商很难完全代劳这项工作，必须依靠企业内部团队建立起这种深度评测能力。 第二大挑战是**对数据管理认知的根本性重构**。传统软件工程往往将“数据”与“应用程序”割裂看待。尽管在传统机器学习阶段企业就已经意识到数据质量直接决定模型效果，但在生成式 AI 时代，这一问题被成倍放大。企业最大的思维障碍在于依旧将数据视为单纯的外部输入，而实际上——**“数据本身就是应用程序”（The data IS the application）**。企业必须像维护核心应用代码一样去对待数据资产、数据清洗和数据流水线。 在变革推进策略上，团队继续依托“人员、流程、技术”（People, Process, Technology）的经典框架，将合规与运维流程直接内嵌进平台底层以降低使用门槛。当工程师抱怨“我的模型跑不通是因为平台不符合我的个人习惯”时，平台团队需要通过机制设计与正向激励，逐步引导算法人员向标准化流程靠拢。 ### [39:05](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2345s) 合规政策与无感自动化的双轨驱动 在受监管行业中推行机器学习最佳实践和治理流程，必须平衡合规刚性与工程体验。Kevin Kelley 解释道，面对需要强监管的业务环境，团队将部分核心流程抽离并确立为企业政策（Policy）。因为在受监管行业中，无论是为了满足现有审计，还是应对未来的监管趋势，团队都必须具备完整的信息感知能力。这就要求在可观测性（Observability）体系中全面采集指标，维护一个保持最新状态、经历持续审查更新以及严格变更管理流程的模型注册表（Model Registry）。面对这类硬性规定，开发者很难反驳政策层面的要求。 然而，单纯依赖合规政策强制推行并不是最理想的方法。平台团队更希望通过激励措施去引导开发者采用正规途径。核心原则在于将流程做到极致的“无摩擦”（Frictionless）——让平台自动化承担大部分繁重的合规与审计任务，以至于对开发者而言，如果试图与这套规范争辩或自行绕开它，所花费的精力反而远大于顺从并使用这套工具。 - **监管硬约束**：利用合规与可观测性要求，使模型版本注册和变更审查成为不可抗拒的强制政策。 - **摩擦力反转**：通过自动化接管繁琐的注册与审计记录，让遵守规范比违规操作更加省时省力。 --- ### [39:54](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2394s) 开发者认同、速赢策略与跨职能共享知识库 Balaji Muthazhagan 深入探讨了推动内部工具落地与采纳的具体路径，强调采纳的第一步永远是“先赢得开发者的心”。如果一个平台工具无法打动最一线的工程师，推广便无从谈起。开发者日常最头疼的痛点之一就是跨工具频繁切换（Tool hopping）带来的摩擦成本。因此，如果能提供一个将 CLI（命令行工具）和 UI 深度整合的统一入口，让开发者在一个地方就能安全地完成测试、部署和监控全流程，工程师的满意度就会大幅提升。 在赢得单个开发者后，下一步是如何说服其他业务团队采用新系统。Balaji 提出的核心打法是“速赢”（Quick Wins）——即向业务团队证明，采用该平台是“低风险、高收益”的。以大规模代码迁移为例，平台团队会并行业务的旧代码路径与新代码路径，拉齐两者的基线指标，尤其是业务团队最为关心的 SLA 承诺。随后输出一份简明的一页纸文档（One-pager），明确列出改动范围与涉及的指标，与业务方充分核对：“你们对这些改动满意吗？”如果业务方有顾虑，说明改动风险偏高，需要调整方案；如果能证明低风险且有显著收益，团队自然乐于采纳。 此外，Balaji 还特别强调了建立“共享文献/知识库”（Shared Literature）的价值。机器学习团队、软件工程团队以及数据科学团队不应各自为战，而应共同建设跨领域的知识体系，甚至可以共享 On-call 值班轮转。这种机制极大地促进了跨职能团队间的理解，让平台开发者切身体会到软件工程师与算法工程师的真实痛点，从而打造出各方都真心热爱、且愿意长年使用的平台产品。 ```mermaid flowchart TD accTitle: 内部工程平台采纳推广路径 subgraph S1["第一步：个体认同"] A["统一 CLI + UI 入口"] --> B["覆盖测试/部署/监控全链路"] B --> C["消除工具切换摩擦 (赢得开发者)"] end subgraph S2["第二步：团队拓展"] D["新旧代码路径并行对比"] --> E["一页纸文档 (SLA/指标验证)"] E --> F["速赢驱动 (低风险高收益采纳)"] end subgraph S3["第三步：组织文化深化"] G["共享知识库 (Shared Literature)"] --> H["共享 On-call 轮转机制"] H --> I["跨角色共情打造长期沉淀平台"] end C --> D F --> G ``` - **统一工作流**：整合 CLI 与 UI，终结跨工具切换碎片化，打造闭环的部署与监控体验。 - **速赢验证机制**：通过新旧代码路径跑分对比与 SLA 一页纸确认，以极低试错风险推动团队落地。 - **跨职能共情**：推动 ML、SWE 与 DS 团队共建知识库与值班体系，在文化层面消除技术壁垒。 --- ### [41:56](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2516s) 渐进式平滑迁移：以 SageMaker 落地实践为例 Rachita Naik 从具体实战角度分享了内部工具及托管服务（Managed Solutions）的采纳经验。她指出，引入任何工具前，必须首先想清楚：这个工具到底解决了什么核心痛点？它如何让机器学习工程师的日常开发变得更加顺畅、迭代更加迅速、规模化更加轻松？ 以在 Lyft 内部推进 Amazon SageMaker 迁移为例，平台团队并没有一开始就强行在全公司铺开，而是先引入几个核心内部团队作为先锋，让他们先行在新平台上训练模型。小规模团队的成功试点不仅能形成标杆示范效应，更能及时暴露跨团队推广时潜在的底层阻碍——例如复杂的 IAM 权限配置漏洞、各团队日志格式（Logging formats）不统一等隐患。只有在先锋团队跑通成功用例后，平台团队才有充分底气向全公司大范围推广。 在迁移策略上，Rachita 坚决反对颠覆式的“连根拔起”（Uproot），即强迫业务团队为了适应新工具而彻底重构原有的日常工作流。相反，平台团队选择将 SageMaker 平滑集成到现有流水线中，把已有的特征版本库（Feature versions）和底层数据源与训练平台无缝对接，由平台层将底层环境搭建的复杂性抽象屏蔽掉。最后，配合详尽的技术文档、一对一实操培训以及定期的答疑交流会（Office Hours），确保业务团队在毫无痛苦的体验下完成现代化架构过渡。 - **试点暴露盲点**：以先锋团队小规模试用为抓手，提前发现 IAM 权限不符与日志格式不一致等阻碍，建立全公司推广信心。 - **抽象底层复杂度**：直接对接现有特征平台与数据源，业务团队无须关注云平台底层繁琐配置。 - **全方位支持交付**：结合详尽文档、1对1 培训及 Office Hours 答疑，确保平滑无痛交接。 --- ### [44:12](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2652s) 嘉宾致谢与结语 在圆桌会议尾声，主持人感谢了各位嘉宾的倾情分享，并邀请台上的四位专家依次重新自我介绍，以便现场观众在会后建立联系与深入交流： - **Rachita Naik**：Lyft 机器学习工程师（Machine Learning Engineer）。 - **Balaji Muthazhagan Thirugnana Muthuvelan**：Pinterest 软件工程师（Software Engineer）。 - **Kevin Kelley**：American Century Investments 人工智能/机器学习赋能与创新主管（Director of AI/ML Enablement and Innovation）。 - **Greg Whalen**：Prove AI 首席技术官（CTO），并热情邀请现场参会者前往 Prove AI 的展台交流互动。 ## Notable quotes > [02:28](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=148s) 先选云，后选工具 — Greg Whalen > [07:47](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=467s) 那完全是在浪费时间，严重耽误将应用推向生产环境的黄金窗口 — Greg Whalen > [12:42](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=762s) 这一个特定版本的模型，究竟是使用哪一份具体数据训练出来的，从而导致输出了这一项特定的预测结果？ — Balaji Muthazhagan > [12:42](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=762s) 开发者体验优先 — Balaji Muthazhagan > [21:12](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1272s) 90% 的生成式 AI 项目最终未能成功投入生产 — Greg Whalen > [28:23](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1703s) 数据科学家除了在最初的原型设计阶段外，其余环节都不应该被迫卷入底层的 MLOps 运维泥潭中。 > [29:45](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=1785s) 铺平的道路 > [35:14](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2114s) 数据本身就是应用程序 > [39:54](https://www.youtube.com/watch?v=CJTeesC2cFQ&t=2394s) 先赢得开发者的心 — Balaji Muthazhagan ## Mentioned Chris Brousseau, Greg Whalen, Kevin Kelley, Balaji Muthazhagan Thirugnana Muthuvelan, Rachita Naik, Veox, Prove AI, American Century Investments, Pinterest, Lyft, AWS, Amazon Kinesis, DynamoDB, OpenAI, Azure, GCP, Databricks, DataRobot, PyTorch, NVIDIA, NVIDIA NeMo Guardrails, OpenTelemetry, Kubernetes, KubeRay, Ray, Apache Iceberg, Apache Spark, MLflow, Prometheus, Grafana, AWS SageMaker, GitHub, Jupyter Notebook, IDC ## Video description Rachita Naik, Machine Learning Engineer, Lyft Chris Brousseau, VP of AI, Veox Greg Whalen, CTO, Prove AI Balaji Muthazhagan Thirugnana Muthuvelan, Software Engineer, Pinterest Kevin Kelley, Head of AI/ML Enablement and Innovation, American Century Investments Abstracts This session will delve into the technical strategies and best practices for effectively scaling machine learning operations (ML Ops) and managing end-to-end ML pipelines. Panelists will discuss critical components for success, including workflow automation, seamless model deployment, continuous monitoring, and version control. Learn how modern platforms are empowering data scientists and engineers to manage data flow, optimize data pipelines, and ensure reliable data management throughout the machine learning lifecycle. Explore the latest end-to-end tools designed to streamline everything from data preparation to model integration and deployment, delivering successful outcomes at scale. The discussion will also cover challenges related to scalability, ensuring model reliability, and optimizing infrastructure for high-performance AI workloads. [Source](https://www.youtube.com/watch?v=CJTeesC2cFQ)

[2:28] Greg Whalen: 先选云，后选工具 [7:47] Greg Whalen: 那完全是在浪费时间，严重耽误将应用推向生产环境的黄金窗口 [12:42] Balaji Muthazhagan: 这一个特定版本的模型，究竟是使用哪一份具体数据训练出来的，从而导致输出了这一项特定的预测结果？ [12:42] Balaji Muthazhagan: 开发者体验优先 [21:12] Greg Whalen: 90% 的生成式 AI 项目最终未能成功投入生产 [28:23] 数据科学家除了在最初的原型设计阶段外，其余环节都不应该被迫卷入底层的 MLOps 运维泥潭中。 [29:45] 铺平的道路 [35:14] 数据本身就是应用程序 [39:54] Balaji Muthazhagan: 先赢得开发者的心

[2:26] Kevin Kelley: it's inevitable with data sharing applications or platforms. [2:26] Greg Whalen: We already have a big contract with cloud provider X. Let's use their tools. [7:45] Greg Whalen: a hundred little boxes of tools. [12:42] Balaji Muthazhagan Thirugnana Muthuvelan: What data was used to train this particular model for this particular version to output this particular result? [15:52] Rachita Naik: You have to have that balance between control and leverage where you're trying to use managed solutions when they help you scale efficiently without giving up on a lot of your flexibility that you need. But then you have to build out internal platforms and tools when you want more control—like finer-grain control and visibility into some parts of the life cycle. [23:49] coding YAML files [26:39] Rachita Naik: With Lyft, I think even like millisecond-level lags can cause ride cancellations and sort of churn our riders and cause them to go to our competitors. And to avoid negative business outcomes, we had to focus on that tail latency aspect and not the average latencies of some of these managed platforms. [29:45] paved roads [35:14] Greg Whalen: the data is the application

[0:04] By way of introduction, I am Chris [0:05] Brusso. I authored LLM's in production [0:08] and I'm the VP of AI at Vox right now [0:11] where we're trying to kind of engineer a [0:14] new type of AI intelligence. It's a [0:17] little stealthy, but we're having a [0:18] public release tomorrow, so look out for [0:19] it. I will be moderating your panel [0:21] today. And let's go down the row and do [0:24] some brief introductions for everybody. [0:26] >> Sure. Uh, hi everybody. Thanks for [0:28] coming. I'm Greg Whan. I'm the CTO at [0:30] Prove AI. Uh so so Prove AI is is an AI [0:34] platform specific to generative uh you [0:36] know generative AI that helps uh helps [0:38] you specifically get to production more [0:41] quickly, right? So if you're having that [0:42] issue, uh that's that's one of the [0:44] things that uh we do. Uh we focus a lot [0:46] on evals. We focus a lot on data [0:48] management and you'll you could hear all [0:50] about it at our booth. So thank you. [0:53] >> Hello, my name is Kevin Kelly. I'm the [0:54] head of a IML enablement and innovation [0:57] at American Century Investments. We're a [0:59] financial services firm in the Midwest. [1:01] Um we uh work towards delivering and [1:04] standardizing our tooling and [1:05] architecture around AI development and [1:07] deployment. Uh so we can have our data [1:09] scientists and developers spend more [1:10] time on the outcomes and less time in [1:12] the architecture. Um and like the rest [1:14] of us we've heard today, you know, [1:16] scaling our geni and AI and ML workloads [1:19] is a important topic. So look forward to [1:21] this. [1:22] >> Good afternoon. Uh I'm Balaji. I'm a [1:25] software engineer at Pinterest. I work [1:27] for a team called as ad serving [1:29] platform. So if you go to Pinterest, you [1:32] see any ad that renders on any client [1:34] device, it's currently being served by [1:36] my team and it gets served after like a [1:38] huge process of retrieval, indexing, ML [1:41] scoring. So I work with the team which [1:43] builds, deploys and maintains the [1:45] serving and excited to be on the panel [1:48] today. [1:50] >> Um my name is Rashita Nike. I'm a [1:52] machine learning engineer at Lyft, the [1:54] right sharing uh application. Um here I [1:58] work on the dispatch team. So we work on [2:00] building prediction systems for [2:02] real-time ETA estimation, um improving [2:05] rider driver matching and just generally [2:09] keeping the marketplace intact and [2:11] efficiently running. [2:13] >> All right, thank you guys so much. And [2:14] before we get into scaling machine [2:17] learning operations, I just want to get [2:18] a round of applause for Prove AI, which [2:20] Greg is representing. They're one of our [2:22] sponsors today. So, [2:25] thank you everybody. All right, let's [2:26] get into our first question then. So, [2:28] one of the biggest architectural [2:30] questions in ML is cloud, multicloud, [2:32] single cloud, hybrid. So, I wanted to [2:35] ask each of you, what's your view? Are [2:37] you using multiple clouds? Are you [2:39] sticking with one or are you doing [2:41] onrem? [2:43] We'll start with you Rashida. [2:44] >> Okay. Um so at Lyft we've chosen to go [2:48] all in on a single cloud strategy uh [2:51] completely backed by AWS today. Um and [2:55] the reason we started out with that was [2:57] Lyft is a real-time marketplace. So we [2:59] have to cater to millions of riders and [3:02] drivers every week. Um and the RPS can [3:06] be very spiky across the day depending [3:08] on peak hours etc etc. and latency [3:11] constraints are really strict. So we [3:13] felt that single relying on a single [3:17] cloud at that point uh proved to be an [3:19] efficient solution to scale our [3:21] infrastructure without having to get [3:24] into the operational complexities of [3:27] managing the hardware etc. And um most [3:31] of our ML infrastructure is now based on [3:33] AWS and we have sort of tightly [3:36] integrated it across our systems right [3:38] from data pipelines, streaming pipelines [3:41] using Kinesis or like Dynamo DB for [3:43] storage. Um we did evaluate hybrid and [3:47] multicloud solutions but at that point [3:49] it seemed like the um duplication [3:53] complexity and fragmented observability [3:55] seem to pose more challenges than [3:58] solutions for us. [4:00] >> Oh absolutely. I think it's incredible [4:02] to try and focus on both scale and also [4:04] uptime. We'll get into that more later. [4:06] What about you Balaji? Yeah, at at [4:08] Pinterest it's uh it's again single uh [4:11] cloud but again uh a single cloud [4:13] provider works best for deep scale and [4:15] it has been proven over the years that [4:18] uh single cloud works well at scale. So [4:22] uh I I don't want to repeat her point so [4:23] I'll just give an uh interesting take on [4:25] this. Um so I was reading an white paper [4:28] from um OpenAI yesterday and uh talked [4:31] about how they scaled their cloud infra [4:33] from 2500 to 7500 nodes in 2021 and the [4:37] kind of optimizations they did were at [4:39] the network traffic level etc. Now uh we [4:43] also spend a lot of time on single cloud [4:45] because these optimizations are fairly [4:48] complex and when you think of multi [4:49] cloud providers you would have to [4:51] duplicate all of your efforts for each [4:52] of those cloud providers. So you have to [4:55] duplicate the networking, the [4:56] observability layers etc. So uh [4:59] Pinterest was always on single cloud is [5:01] still on single cloud. Uh I've had a [5:03] fair bit of experience with hybrid [5:05] clouds in research labs back at UCSD. So [5:08] we have a on-prem GPU which has access [5:10] to the data but it only works because we [5:13] have a very stable training load and [5:15] like she pointed out uh it's not very [5:17] useful in spiky training cases in real [5:20] time predictions etc. [5:24] Uh for us we're a little different [5:26] because even though we do custom [5:27] development as a financial services [5:28] firm, we rely a lot on partners in the [5:31] industry and uh though we probably have [5:33] chosen a single cloud approach for our [5:35] custom build and architecture, we have [5:37] to be open into a hybrid cloud approach [5:39] for integrations with external [5:40] providers. [5:41] >> So you guys take on the headache. [5:43] >> Well, [5:43] >> nobody thinks it's a headache and [5:44] >> it's unavoidable in some cases. Yes. I [5:46] mean I think we we try and stay away [5:48] from multicloud from customuilt [5:50] solutions but it's inevitable with data [5:51] sharing applications or or platforms and [5:54] a financial services provider or data [5:56] provider. [5:58] >> Yeah. So I'll I'll add on so proof AI is [6:01] is of course an LLM ops tool. So we have [6:05] we have clients on all all sorts of [6:07] clouds. So of course our tool is [6:08] multicloud. It supports multi cloud um [6:11] workloads as well as hybrid cloud and [6:13] onrem. Um I will add though so we have [6:15] some very mature um you know individuals [6:18] here with their workloads on the panel. [6:20] I'll I'll warn and caution that a lot of [6:22] the uh a lot of the individuals we work [6:25] with who are early in their Genai [6:27] journey um typically do look at uh [6:31] something more along the lines of [6:32] multicloud andor on prim uh types of [6:35] deployments. So so like I'll caution [6:37] that it's still very early. There's a [6:40] lot of different tooling out there and [6:42] if you you don't want to pick the cloud [6:44] first, right? And we think that that is [6:46] a pitfall where somebody says, "We [6:48] already have a big contract with cloud [6:50] provider X. Let's use their tools." [6:53] That's a pitfall. That's a mistake, [6:54] right? And we've seen people make this [6:56] mistake all the time because um the tool [6:59] that that little specific tool that's [7:01] perfect for your use case may not run on [7:03] that specific cloud, right? So for [7:05] individuals who are early in their um [7:08] journey here right we say that over time [7:10] absolutely pick the cloud that works for [7:12] you you'll get better economies of scale [7:14] you'll get that observability right you [7:16] don't have to do three observability [7:18] stacks but the caution is if you're [7:20] early um you may want to look at [7:22] something else and um especially to [7:24] manage costs we've had a lot of success [7:26] with like uh you know running a bunch of [7:28] workloads on prem right especially [7:30] because GPU compute is so expensive um [7:33] so unless you're a very large enterprise [7:34] and you don't mind blowing immense [7:37] amounts of uh money per day, right? [7:38] You're you're you're probably better off [7:40] running your experiments on a local [7:42] machine. So, all of these factors come [7:44] into it. [7:45] >> Awesome. Okay. Well, actually, let's [7:47] move on to exactly what you mentioned, [7:48] tools, right? We've talked about AWS. Uh [7:51] we there are a bunch of them, data [7:52] bricks, data robot, AWS, Azure, GCP, and [7:55] then there are the open source stacks [7:56] that you're mentioning right now. So, [7:58] what does your current ML stack look [8:00] like? And uh keep that as concise as you [8:03] possibly can and then get into the meat [8:05] of it, which is uh why did you choose [8:07] those tools and what are the trade-offs? [8:10] Go ahead. You first. [8:11] >> Okay. Sure. So, um so again to [8:13] reiterate, uh so Prove AI is an LLM ops [8:16] tool. So, we are an MLOps tool [8:18] ourselves. Um that said, we're we fully [8:20] embrace open-source tool sets um within [8:23] our organization. One of the problems we [8:25] solve is is that simply tool curation is [8:29] hard, right? So, how many slides have [8:31] you seen today with a hundred little [8:34] boxes of tools in them? Um, yeah. So, if [8:37] if you're early in your Genai journey, [8:40] that's a problem. You're going to spend [8:42] months and months trying to stitch [8:43] together a hundred little things and get [8:46] them to cooperate with each other. [8:48] That's a waste of time. That's a waste [8:49] of time if you want to get something [8:51] working in production now. So that is [8:53] one of the things that provei um [8:55] addresses and and our specific approach [8:56] unlike those sort of closed ecosystem [8:58] providers is that we we don't want to [9:01] lock our customers in. We we think that [9:04] the tools tool sets are changing so [9:06] quickly. So why not embrace that and why [9:08] not make that problem easier right for [9:10] our customers right but as a result that [9:12] means that we're built on as many of the [9:13] open source uh the big open source [9:16] packages out there things like pietorch [9:18] things in the Nvidia ecosystem so we're [9:20] built on Nvidia Nemo for guardrail [9:22] support and things like that um and then [9:24] um if you if you get into some of the [9:26] telemetry things right we're built on [9:27] open telemetry um and um and the you [9:31] know and libraries that help you with [9:32] that so um that's our approach right [9:34] it'll you you'll see different opinions [9:36] probably here but that's that's how we [9:38] uh you know that's the vision behind our [9:40] product. [9:42] >> So I think we have a lot of similarities [9:43] there. Um I think it's the maturity of [9:46] the given workload maybe the maturity in [9:48] the industry and how that goes from [9:50] maybe a turnkey solution or vendor [9:53] provided solution versus a pure open [9:55] source. So we do probably have what I [9:56] would say is a hybrid approach with some [9:58] open source as well as some kind of [10:00] offtheshelf or vendor provided [10:02] solutions. Uh some of that is that time [10:04] to value. Um like you uh like the [10:06] previous speaker was talking about you [10:07] can spend a lot of time integrating open [10:08] sources together. Uh and maybe we want [10:11] to get uh quick uh quickly get some [10:13] scale in a given space and then learn [10:15] through doing and learn through [10:16] experience before we maybe uh change [10:18] into a more open source approach. So [10:20] we've seen that with some of our other [10:22] platform builds as things have matured [10:24] as we've gotten scale. we've been able [10:26] to be kind of armed with information on [10:28] observability to make an informed [10:29] decision about maybe transitioning some [10:31] of those workloads to open source. [10:34] >> Yeah. Um to be specific, I would like to [10:37] break up our tooling system into like [10:39] four different layers. So one would be [10:41] at the compute layer. You would already [10:43] know Kubernetes is a very popular choice [10:45] at the compute layer. But u uh recent [10:49] developments have actually uh been um [10:52] attacking the Kubernetes operator [10:53] itself. So you if you read about the [10:56] cube ray uh cube is is ray is an uh [11:00] distributed computing platform uh which [11:02] is built off on top of python and cube [11:05] ray is a is an operator that has been [11:08] modified to make sure that uh you're [11:10] able to scale well like the previous [11:11] example where opena was able to scale [11:14] from 2500 to 7500 nodes they were not [11:17] able to do uh do it due to the [11:19] kubernetes limit of 6,000 nodes but by [11:21] changing the operator itself you can [11:23] actually push the limits of scale there [11:25] and then coming to the actual data [11:27] layer. Uh there are some very [11:29] interesting things that are happening. [11:30] Previously we used to precomputee a lot [11:33] of like spark offline data assets and [11:35] tables but right now it's do it's being [11:38] done in memory. So on the fly we are [11:40] able to do it with ray we are able to do [11:41] it with Apache iceberg and uh that saves [11:44] a lot of time in terms of uh retraining [11:47] and uh saving a lot of times even in ter [11:49] in terms of experimentation cost uh when [11:52] a new feature gets added and then when [11:54] you move to the model layer um uh I I [11:57] think uh a very popular choice is again [11:59] MLflow we we also have a custom tooling [12:02] built on top of MLflow so that our data [12:04] scientists sR software engineers are [12:06] able to use it use the same unified [12:08] platform [12:09] And then coming to observability we have [12:11] Prometheus Graphana. We are built on top [12:13] of Prometheus and Grafana for the [12:15] tooling. But of course you have to solve [12:17] for scale problems there as well because [12:19] the Prometheus and Grafana clients uh [12:21] are not scalable. So you'd have to again [12:24] do custom patching on those uh clients [12:26] individually to make sure your [12:27] observability doesn't crash out. So [12:30] let's say you improve your scale and [12:31] because of a high metric cardality your [12:34] Prometheus client crashes out. That's [12:36] that's again a fault on observability. [12:38] So yeah, four four layers there like all [12:40] of these pieces working together. [12:42] >> Let's actually dig into that for one [12:44] second. You mentioned a platform on top [12:46] of MLflow, right? What are you adding on [12:48] top of there? Like what was not enough [12:50] about MLflow for you guys? I think uh [12:53] although MLflow has good lineage um it [12:57] still doesn't answer the question of [12:59] what data was used to train this [13:02] particular model for this particular [13:03] version to output this particular [13:05] result. So to get that [13:08] >> any MLflow contributors here are you [13:10] listening? [13:11] >> This is a problem for everybody. [13:12] >> Yeah. So to get that lineage information [13:14] you would do a tool hop. So multiple [13:17] engineers will actually have to find the [13:19] lineage information from multiple tools [13:21] and you would assume that MLflow solves [13:23] this problem but even yesterday I was [13:25] using three four tools to find out the [13:27] lineage of how the model was trained and [13:29] why did it give out this prediction. So [13:31] yeah I think a lot of our custom tooling [13:33] is very specific to the industry and uh [13:37] the problems that we get to solve within [13:38] our industry itself. So it has evolved [13:41] over the years uh uh on the basis of how [13:44] our developers so keeping our de [13:47] developers first uh while building out [13:49] the platform itself. [13:50] >> Sick. Okay. [13:53] Um what does your stack look like? Why [13:56] did you choose your tools? Yeah, I I [13:59] think a lot of the solutions have [14:01] already been covered and I think ours [14:03] would probably be very close to the [14:05] Pinterest stack where um it's been an [14:08] iterative process building out our ML [14:11] infrastructure. Initially, we started [14:13] having our training workloads managed by [14:16] Kubernetes-based [14:17] stacks. But then as these as the number [14:20] of models grew the number of users and [14:22] services scaled um the job orchestration [14:26] overhead was getting a lot and so we [14:28] started exploring Sage Maker for [14:30] instance as our managed solution for um [14:34] simplifying distributed training but [14:36] also giving our uh developers and ML [14:39] engineers the flexibility to uh train [14:42] any model agnostic of the framework. So [14:45] deep learning or gradient boosting [14:47] trees. Gradient boosting trees are very [14:48] popular at Lyft. Um but then for model [14:51] serving and deployment um we chose to [14:55] build our own internal platform and [14:57] tooling um on top of Kubernetes because [15:00] we had this latency and scale [15:02] requirements where we wanted to be more [15:04] in control of our deployment stack and [15:08] how we rolled out new models to [15:10] production. So this platform basically [15:13] gave engineers the ability to train and [15:16] s package models in whatever framework [15:18] they wanted to, but then deploy it using [15:20] this consistent interface which also had [15:23] other features like like basically any [15:26] Kubernetes managed service has like [15:28] autoscaling benefits or you have more [15:30] control over shadow and canary [15:32] deployments and roll out and roll back [15:34] logic and um that sort of control was [15:38] necessary for our serving [15:39] infrastructure. So and then on top of [15:42] that I think we also use MLflow for [15:44] observability to um track performance [15:47] metrics primarily across ML life cycles. [15:50] So yeah that's our stack. [15:52] >> That's an excellent answer. So actually [15:54] I I have a followup for you as well. Um [15:58] I it feels like like you were saying [16:00] your tech stack seems very very similar [16:02] to Pinterest and I'm actually willing to [16:05] wager it seems very similar to most [16:07] companies right now that are building an [16:09] ML. Um why do you think there is this [16:12] kind of singular tech stack that has [16:15] emerged in this space? [16:17] >> I I think it's because um like when you [16:21] have these sort of tried and tested out [16:24] solutions across the industry with sort [16:27] of like I guess with Pinterest and Lyft [16:29] both are trying to deal with [16:31] marketplaces like [16:32] >> well realtime market places. real time [16:34] marketplaces. And you have to have that [16:37] balance between control and leverage [16:39] where you're trying to use managed [16:42] solutions when they help you scale [16:45] efficiently without giving up on a lot [16:46] of your flexibility that you need. But [16:49] then you have to build out internal [16:51] platforms and tools when you want more [16:54] control like finer grain control and [16:56] visibility into some parts of like the [16:58] life cycle. [16:59] >> Okay, great. Let's actually um let's hit [17:01] both of you then with this question [17:03] considering you're not part of that [17:05] monolithic tech stack. You're going in [17:07] different directions. So uh there's kind [17:10] of a critical decision about [17:11] infrastructure which we've already hit a [17:14] little bit on whether it's like cloud or [17:16] onrem. How do you approach choices [17:18] around cloud versus onrem when you're [17:21] building infrastructure whether it's for [17:22] yourself or for clients? [17:24] >> So I'll go real quick. I think I [17:26] completely agree with a lot of the [17:27] statements you made. I I do think we [17:28] want to encourage that um that [17:31] flexibility for the development but [17:33] standardize the the training the uh [17:35] deployment and monitoring. So model [17:38] serving uh in a standardized you know [17:41] patternbased approach or configuration [17:43] based approach we actually uh in our [17:45] industry because of some data latency [17:47] and some data delivery we still do batch [17:49] in some of our predictions. Um so we [17:52] have some real-time workloads but it's [17:54] probably the minority of some of our [17:55] workloads. So um if you were to ask a [17:58] data scientist what they really care [17:59] about it's model training and you know [18:02] uh feature selection you know EDA all [18:04] that other stuff [18:05] >> they don't metrics [18:06] >> yeah they don't they don't care how the [18:07] model runs they care on uptime and and [18:10] monitoring so we try and standardize [18:12] those into kind of value added [18:14] activities versus undifferiated [18:16] activities um so I think that's what's [18:18] influenced it and that's what we use to [18:20] maybe select some of our architectural [18:22] solutions. [18:23] >> Awesome. Thanks. Go ahead Greg. Yeah. So [18:25] the I think the decision criteria Yeah. [18:28] I'll agree with a lot of what you just [18:30] said here, but it's um [18:33] a lot of the first step in the decision [18:35] comes from constraints that already [18:38] exist, right? So I'll diagram it out [18:40] that like three things if I had to stack [18:42] rank three things that really matter, [18:44] right? It's around constraints of the [18:46] problem, right? The use case that you're [18:47] trying to solve and constraints around [18:49] it. Uh second we need to talk about [18:52] capabilities and right the you know what [18:55] your people can actually do and what [18:57] that how they can manage stuff right so [18:59] that factors in a lot third would be [19:02] cost right and I would actually I would [19:04] you know it's debatable but I would off [19:06] the top of my head stack rank it in [19:07] those three uh three ways I think that [19:09] the like like for instance like you know [19:12] the specifics around where data needs to [19:15] be hosted where it comes from um you [19:18] know where does it need to be processed [19:21] and what types of sort of checks are [19:23] already, you know, already written and [19:25] available for your pipeline. Those are [19:27] the types of things that drive decisions [19:29] the most because in general we don't [19:31] want to have to rewrite those things and [19:32] we don't want to have to do duplicative [19:34] work, right? So, um that's especially [19:36] important at Genai, we deal with LLM, [19:38] right? We deal with generative AI. So, [19:41] so like my answer is going to be a [19:42] little different than than some of the [19:43] others here, but you're talking also [19:45] about just tools that don't always work, [19:48] right? You're talking about things that [19:50] are very new and you're talking about [19:51] data pipelines that have a very very big [19:54] impact on ultimately the quality the [19:56] downstream quality of your stuff and [19:58] you're you're dealing with individuals [19:59] and people talent who um for whom this [20:02] is all new stuff as well. So, you know, [20:05] I I' I'd put those considerations in, [20:07] but the biggest one is simply like what [20:08] problem are we trying to solve? And [20:10] what's the fastest path to value? And, [20:12] you know, that one level deeper is well, [20:15] then it's really stuff. It's processes [20:17] and and stuff you've already written so [20:19] far that drives so much of the decision [20:22] to just stay on prem or move something [20:24] on prem um you know, to basically figure [20:27] out what the easiest path is. Um I I'll [20:29] I'll also add that there's um some of [20:31] the other some of the other factors that [20:34] get into this is around collaboration as [20:36] well. So especially with data providers [20:38] um you know what happens when suddenly [20:40] you realize the building blocks of your [20:42] application data especially in the genai [20:44] if you're fine-tuning stuff uh if you [20:46] have external collaborators who are [20:48] contributing data sets to your pipeline [20:51] um this is yeah you know this is not [20:53] this does not look like traditional [20:54] software anymore right it means that [20:57] you're going to get into thorny problems [20:58] of well we use that cloud you you know [21:01] you guys use that cloud we don't use [21:03] that cloud therefore we need we need to [21:05] put all of this somewhere somewhere [21:06] else, right? Um, these are the types of [21:08] things that actually drive decisions u [21:10] much more than other factors. So, yeah. [21:12] >> Well, with that, let's get into a [21:14] valuable question here. Actually, this [21:16] is a question for the audience first [21:17] before we hit this. How many people here [21:18] by raise of hand work at a company that [21:20] has an MLOps platform offering where you [21:23] sell your platform to other people? [21:26] Zack, this is going to be really [21:27] valuable for you guys. Um, okay. So, for [21:30] you, what gaps do you see from MLOps [21:33] platform providers? like what gaps are [21:35] you trying to solve for and fill at [21:37] prove aai and what gaps do you see as [21:39] customers of those platforms? [21:42] >> Yeah, go ahead. [21:43] >> Yeah, sure. So, the biggest one that we [21:45] call out right now, right, I called out [21:47] the tool curation earlier, but it's [21:50] still performance evaluation, eval [21:52] especially in the Gen AI space as the [21:54] biggest one that hits people right in [21:56] the face when they get close to thinking [21:59] they're production ready, right? Right. [22:01] It's just because performance evaluation [22:03] in the generative AI space is is [22:06] horrendously difficult compared to [22:08] standard machine learning maybe. Right? [22:10] But um it is it's it's very difficult. [22:13] People are they don't see it coming. [22:14] They're not prepared for it. Um they [22:17] think they're 90% of the way done and [22:19] then they say uh-oh. Right. And it's [22:21] it's an immense amount of work. It's um [22:24] it's stuff that they typically the [22:26] talent in place doesn't have um really [22:28] enough information on and it it suddenly [22:30] creates this massive amount of time [22:32] right that says well we thought we were [22:34] going to launch in a month and now we're [22:35] going to launch in 6 months and nobody [22:37] nobody likes that we did prove ai did an [22:40] actual data survey where we surveyed a [22:42] bunch of companies and also um we I'm [22:44] using a statistic from IDC in a in a [22:46] March 2025 report right so 90% of of [22:50] generative AI projects don't make it to [22:52] production despite the fact that our [22:55] report shows that there's, you know, on [22:57] average 5 to10 million um invested in [23:00] generative AI stuff right now. So it [23:02] means people are pumping money into this [23:04] stuff is not getting to production. So [23:06] the question is what's the biggest thing [23:08] other than tool curation that's [23:10] preventing this? what we've seen from [23:12] our customers it's eval right now is the [23:15] problem and that's one of the things [23:16] right topline for prove aai right that [23:19] we help with um and it's that's a [23:21] combination of software yes because [23:22] there's plenty of good eval libraries [23:24] out there using them is hard creating [23:26] the data sets for those eval is hard and [23:29] the talent required to do that is is [23:32] missing right so proofi does all of [23:34] those things right we provide [23:35] handholding we provide services to get [23:37] started we provide open- source curation [23:39] of the best of best of breed tools that [23:41] help you run those AVLs, execute those [23:43] AVLs as well as create and manage right [23:46] on top of that in a much more easy to [23:48] use interface. [23:49] >> So I'll answer it twofold. I'll I'll [23:51] touch on the genai portion of what you [23:53] just mentioned. Completely agree [23:54] evaluation is very very difficult. It is [23:58] um it is a quickly changing ecosystem of [24:00] me. [24:00] >> You mean you're not satisfied with LLM [24:02] as judge? um in a blackbox nature. Yes. [24:06] But uh you know the explanability side [24:08] of it leaves that uh some some lacking [24:11] there. Um but I think I you know my data [24:14] scientists like the flexibility of being [24:15] able to look to the uh industry look to [24:17] the research and say you know what are [24:19] the metrics that are important to them [24:20] uh for for accuracy or precision and uh [24:23] having a modular based approach. So some [24:25] of the observability frameworks are [24:27] great from a blackbox service offering [24:29] or if you you know your square pegs fit [24:31] in their square holes but if you have [24:32] some other metrics and some custom [24:34] builds it's a little problematic. U and [24:36] that kind of leads into I guess I would [24:38] say the ML side as well. Um you know we [24:41] have SageMaker um we like the managed [24:43] nature of it fits well with that kind of [24:45] experimentation [24:47] uh for for training as well. Um but like [24:49] I said the the model serving is a little [24:51] bit lacking. Um but I think the piece we [24:55] find missing in SageMaker and and a [24:56] little bit in MLflow as well is the the [24:58] data model side of the intake the [25:00] registry and uh the the pieces needed [25:03] for kind of governance processes that [25:06] last mile that makes it unique to our [25:07] operational processes. You know they [25:09] have a if you can fit into the data [25:11] model paradigm of how they have [25:13] operations it works great. Uh but what [25:15] we found is most of these things always [25:17] require some sort of bolt-on, some sort [25:19] of add-on that fits our processes. You [25:22] know, sometimes our processes need to [25:23] change like any other company. Um but [25:26] you know, there are some examples where [25:27] we have some operational processes where [25:29] we want to stay relatively true and we [25:31] always have to do something after the [25:32] fact and you know do it outside of the [25:34] framework and then put it inside. [25:39] Um I think uh with respect to actual uh [25:43] different types of developers on the [25:44] team, you don't want your data [25:45] scientists to be coding YAML files but [25:48] rather them to be focusing on actual [25:50] models uh actual model production and [25:53] testing. So in that aspect I would feel [25:55] that today's gap is we don't make the [25:58] infra invisible enough. So if there is [26:00] an offering that would make the infra [26:02] invisible and if a DS is able to safely [26:05] test, deploy, build and even monitor his [26:08] changes in live production. So that [26:10] would be a great offering. And uh [26:12] talking from uh again like touching base [26:14] on a point I mentioned earlier, I think [26:16] there is a lack of like true lineage [26:18] when it comes to uh the different types [26:21] of problem that we solve. So if an [26:23] offering is able to give you true [26:25] lineage information, it saves several [26:27] debugging hours for an engineer who's [26:28] working on call and when he's solving [26:30] that incident. So it's again a added win [26:33] for several companies when you're able [26:34] to get that lineage information in in an [26:37] instant. [26:39] Um, in addition to all of these great [26:42] points, I think one thing that was sort [26:45] of unique to uh the kind of problems [26:47] that Lyft deals with is we had very [26:51] frequent training serving parity issues [26:53] where we would train and deploy models [26:56] but then very frequently there used to [26:59] be shifts in input data distributions or [27:02] um the residuals between predictions and [27:05] factuals would start increasing. And I [27:08] think that sort of gives rise to the [27:10] need to build internal tools. So we had [27:13] um we started building the centralized [27:16] feature serving infrastructure where it [27:18] gives the ability to both engineers and [27:21] DS to sort of selfservice realtime ML [27:24] features that they want to test out for [27:27] their new models and they basically just [27:29] plug in feature configs and definitions [27:31] and that gets deployed across batch [27:34] pipelines for training and then also [27:36] serving pipelines for realtime online [27:39] inference. Um so I think that we found [27:43] that lacking and and that's why we sort [27:46] of had to build our own internal [27:47] solutions for it. Even with respect to [27:50] the ML platform that I was talking [27:52] about, I think we wanted to cater to [27:55] like the tail latencies while serving [27:58] predictions to our users because with [28:01] with with Lyft, I think even like [28:04] millisecond level lags can cause can [28:06] write cancellations and sort of churn [28:09] our riders and cause them to go to our [28:11] competitors. And to avoid negative [28:13] business outcomes, we had to focus on [28:16] that tail latency aspect and not the [28:18] average latencies of of some of these [28:20] managed platforms. [28:23] >> That rocks. So um let let's bounce off [28:26] of that actually from your perspective [28:29] like what makes worldclass MLOps, right? [28:33] What what are you looking for in an [28:36] MLOps platform that makes it solve every [28:38] problem at Lyft or at Pinterest at the [28:41] company that you work at? [28:44] >> I I think the core aspect of these [28:47] platforms is that they should have these [28:49] reproducible workflows and highly [28:52] available self-service infrastructure [28:54] catering to different components across [28:57] the ML life cycle. So from data [29:00] ingestion to model training, deployment, [29:03] surveying and finally observability and [29:04] monitoring. So that as um as uh Balaji [29:08] mentioned that data like engineer data [29:11] scientists should focus primarily on [29:13] model prototyping and less on the [29:17] complexities of deployment and serving [29:20] and I think [29:20] >> data scientists should not be involved [29:22] in MLOps except at the very beginning. [29:24] >> Correct. Yes. Um so I I think tooling is [29:28] one aspect of it but then um there are a [29:31] lot of added layers on around [29:33] observability and monitoring which [29:34] should also be like managed by the [29:37] platform and abstracted. Um and I think [29:40] that's some of what we've tried to [29:42] tackle by having more comprehensive [29:45] observability and monitoring. Um and we [29:48] started off with very bespoke pipelines [29:51] where every team had their own hacky way [29:53] of deploying models but now we've moved [29:56] to a more unified workflow. So I I think [30:00] yeah it's it's an iterative process. [30:04] >> Uh according to me a worldass MLOps [30:07] product or team would actually consist [30:09] of first thing uh elastic performance. [30:13] So what I mean by elastic performance is [30:15] not how many nodes you're able to add [30:17] when you want to scale up or down but [30:19] rather with the available resources and [30:21] with a heterogeneous system of resources [30:23] like GPUs, CPUs, TPUs, is your system [30:27] able to perform well. So what that means [30:29] is do you have an idle GPU sitting [30:31] somewhere and are you using it or not [30:33] like fractional GPU scheduling and [30:36] topological aware scheduling etc. So by [30:38] making your system extremely uh [30:41] performance elastic I I feel like it'll [30:43] be a it'll make a great uh MLOps [30:46] addition to a truly world-class nature. [30:49] And the second thing is uh like what she [30:51] touched on it would be the actual [30:52] developer ergonomics. So if you make the [30:55] developer really comfortable when he's [30:57] working with the product I feel like it [30:59] would still uh again like is is a huge [31:02] value addition to the entire company [31:03] itself. So uh and if there are several [31:06] ways to do that. So once you uh for [31:09] example uh a simple thing to do is [31:11] something called as paved roads. So what [31:14] pave roads are you have a product you [31:16] just give a set of starter templates [31:17] saying that this is how you should start [31:19] documenting and when different teams [31:21] start seeing the heterogeneous solutions [31:23] and templates you've come up with it's [31:25] often easier to convince them and make [31:27] them adopt your product as well. And the [31:29] third thing is obviously good governance [31:31] and observability. like when you give [31:33] access to infinite experimentation there [31:35] is a chance of abuse there is a chance [31:37] of GPU cost getting uh depleted etc so [31:41] we with like good metrics good [31:43] monitoring systems I think like uh and [31:46] having transparency at its core making [31:48] sure the metrics are available and [31:50] you're able to consume it in real time [31:52] is is is what makes a great MLOps [31:54] platform [31:56] >> yeah again repeating many of the things [31:58] you said that elasticity but not [32:00] ultimate flexibility you know how do How [32:02] can you be flexible to the point but [32:04] then also find few reusable patterns so [32:06] that you're not ultimately flexible and [32:08] water down your ops. Um user experience [32:11] is very great. Um you know we we we tend [32:14] to align around a given tool stack. You [32:16] know they really like Jupiter as their [32:18] as their their core development [32:19] framework. So if we can deliver them a [32:22] Jupiter uh notebook that's integrated [32:24] into GitHub, we can do things like code [32:26] scanning and and and standards and [32:27] automated testing and make it relatively [32:30] frictionless for them. they don't know [32:31] the pieces that are happening behind the [32:33] scenes and we can kind of give them [32:35] those carrots to standardize their pro [32:37] procedures. But I also think you know [32:39] with ops capabilities and integrations [32:41] if you deliver to them ops metrics as [32:44] part of their deployment or their model [32:45] training they're less likely to go about [32:48] doing that in code themselves. Um so how [32:51] do you make that frictionless? How do [32:53] you give them the tools so they can [32:54] spend their time uh in the notebooks on [32:57] the models training gathering new data? [33:00] I'm I'm actually in our organization, [33:02] we're actually uh part of the data [33:03] organization as well. So how do we [33:05] position those uh data stores and data [33:08] products through feature stores in a way [33:10] that is frictionless as well. So don't [33:12] don't incentivize them to come in and [33:14] experiment to bring in a data set and do [33:16] local you know transformations uh that [33:18] water down the governance process. Give [33:20] them something that's already [33:21] pre-aligned to the data platform in a [33:24] well-governed and high highly quality [33:26] way. [33:28] >> Yeah. So I agree with everything. Uh the [33:31] only additional sort of comments I think [33:33] that might help is I can answer this [33:35] from a non-technical perspective as [33:37] well. So like when when customers come [33:39] to us and say I want to build an MLOps [33:42] pipeline and I want it to be world [33:44] class, right? And we say oh okay well [33:47] what does that mean? Well it it's [33:48] different for every customer, right? Is [33:50] is one problem. Um but I guess the best [33:53] non-technical definition we've gotten to [33:55] in this case is is one. Can you get to [33:58] production? Can you get to production [33:59] quickly without having unexpected spikes [34:02] in work? Right? That's one. I think [34:04] that's that's already been covered. [34:05] Like, do you have the observability you [34:07] need to do that? Right? Do do the tools [34:09] work together in order to do that? Are [34:12] you effective and are you efficient with [34:13] your compute resources? Can you scale? [34:16] Right? And all of those things [34:18] absolutely valid. The the one thing I'll [34:20] add though is that we say as a business [34:21] milestone world class means as well can [34:25] you iterate and can you change your [34:27] model? Can you refine your stuff faster [34:30] and faster and faster on each run? [34:33] Right? That's if you can do that then [34:34] you've built a worldclass AI pipeline. [34:36] Right? So the business outcome that we [34:38] we focus on is to say forget about the [34:40] tools. Let's stop talking about tools [34:42] because we're just going to end up not [34:44] knowing what we need yet because we [34:45] haven't solved the problem yet. Um, but [34:48] if you want that vision of what's the [34:50] worldclass AI pipeline, right, it's [34:52] really about how are you going to update [34:54] this model and refine this model on a [34:56] weekly daily basis if that's what you [34:58] need to do. And if you have an ML, you [35:01] know, if you have an ML kind of tool [35:03] pipeline that allows you to do that [35:04] without errors and without having to [35:06] call up, you know, additional resources, [35:08] then you've achieved that. So, let's [35:09] work backwards from that vision and [35:11] that's what'll give you that worldclass [35:12] uh pipeline. [35:14] >> Those are sick answers. Okay. So the the [35:17] reason I say that is cuz it feels like [35:19] you guys read my questions. You're just [35:21] going right into them. Um you talked [35:23] about people and responsibilities and [35:25] you talked about processes and [35:27] organizational policy and you're talking [35:28] about organizational change. All of [35:30] these are under MLOps unfortunately [35:33] right at this stage. So how do you guys [35:36] in your organizations approach change [35:38] management when you're scaling your [35:39] MLOps practices especially like tool [35:42] adoption? How do you identify early [35:44] adopters and and get it in their hands [35:46] as possible? How do you uh how do you [35:48] introduce new platforms, new ways of [35:50] working? [35:52] >> Yeah, you go first. [35:54] >> Yeah, sure. So, I I'll focus on two [35:56] things that I think are really [35:57] important. Uh so, one is the evals, [35:59] right? So in terms of organizational [36:01] change, right, finding somebody or a [36:04] collection of people who can dive deep [36:07] into what success and good really means [36:10] for generative AI is critical for us. [36:12] And that's probably the hardest thing [36:14] that we have, you know, that that we run [36:16] into. Um it's it's a it's a very [36:18] difficult thing to do and and as a [36:20] vendor, that's not really something we [36:22] can fully own, right? So, um, you know, [36:25] that's something we definitely lean on [36:27] our customers for to to really figure [36:30] out like, do you have somebody we can [36:31] work with to dive into, you know, not [36:34] only defining it, but then how would you [36:36] how would you actually test this? How [36:37] would you write the the tests for this? [36:39] How would you create the data set that [36:41] that does real evaluation and and how [36:43] can you ultimately get scientific about [36:45] that? That is that's definitely sort of [36:47] a capability to build within the team, [36:49] which is a big org change. Um the second [36:51] area that I'll pick on is around data [36:53] management. Um, so for generative AI [36:56] specifically, um, well, as well as any [36:58] any MMO really, but for generative AI, [37:00] especially when you get into fine-tuning [37:02] use cases, um, we we found that just the [37:05] the availability of data and being able [37:08] to think about like whether something is [37:11] solvable or not solvable or how well [37:13] it's solvable is is a is a really hard [37:16] thing to get discussed because [37:18] traditional software, right, is if you [37:20] go into a large organization that's [37:22] relatively early in their generative AI [37:24] journey or AI journey, they're probably [37:27] taught to think data is separate from [37:29] apps, right? And and this is not new, [37:31] right? This is, you know, for for those [37:32] who are practicing AI or any sort of [37:34] machine learning, this was a change that [37:36] organizations had to go through a decade [37:38] ago. And to realize, well, maybe we [37:40] shouldn't do that anymore because that's [37:41] a bad idea, right? The quality of your [37:43] data, the availability of your data [37:45] severely impacts your quality of your [37:48] solutions. Um, with generative AI, it [37:50] makes it even worse, right? and you have [37:51] all these other additional, you know, [37:53] problems with that. But yeah, we just go [37:55] into so many organizations that still [37:58] need help changing how they think about [38:00] their data assets, their data pipelines, [38:02] their data quality, their data [38:04] cleansing, right? And figuring out how [38:06] to treat your data as the same way that [38:08] you would treat an application, right? [38:09] Because fundamentally your data is the [38:11] application now. And that's that's a [38:13] very different way of thinking for a lot [38:14] of organizations, right? They're still [38:16] stuck in the no, I get it. Genai, it [38:18] uses data. So, you know, I get it. I get [38:20] it. I was like, "No, not really." Right? [38:22] Because it's it's part it is the [38:24] application. The data is the [38:26] application. So why aren't you treating [38:27] it like an application now? Right? [38:29] That's the thing to to get through. And [38:31] that's that's one of the hardest [38:32] organization changes that uh that we [38:34] confront. [38:37] >> Uh so for change management um how we've [38:39] approached this was you know the old [38:41] adage of people process technology. We [38:43] wanted to include the process side of [38:45] operations and governance into the [38:47] platform so that if they adopt the [38:49] platform or if they deploy their [38:50] solutions there then the barrier to [38:52] entry is lower. That was our goal. Um [38:55] but you know that takes time. It takes [38:57] uh change change management in their [38:59] models what my model won't run because I [39:01] don't do it the way you want it to do it [39:02] and we have to incentivize maybe a a [39:05] certain pattern so they get those [39:06] outcomes. Um so how we did that was we [39:09] separated some of that out as policy and [39:11] we said hey you know we're a regulated [39:13] industry we need to either already or in [39:16] the future prepare for this. So we need [39:18] to be armed with the information. We [39:19] need to collect the information as part [39:20] of observability. We need to have a [39:22] up-to-date registry that goes through uh [39:25] you know continuous review and updates [39:27] change management processes and all the [39:29] things that go with that. Um so you know [39:31] it's hard to argue uh against policy in [39:34] a regulated industry. Um you know we [39:36] didn't have to go there. We wanted to [39:37] incentivize the proper way of doing it, [39:39] but in some cases you have to. Um, but I [39:41] think we talked about this earlier too. [39:42] Make it frictionless so that it would be [39:44] more painful that for them to argue with [39:46] it uh than it would be for them to just [39:48] follow because we're going to do some of [39:49] the tasks for them automatically. [39:54] >> Um, I think uh uh it starts with [39:58] convincing the developer first. So if [40:00] you're able to convince a developer to [40:02] adopt your product, then that's a that's [40:04] your first win. And how how you can do [40:07] that is pretty much you we need to [40:09] understand that uh like I touched on [40:12] earlier that tool hopping introduces a [40:14] lot of friction. So if you have a single [40:16] CL plus UI which gives access to all of [40:19] your tooling and if you're able to [40:20] safely test deploy and monitor your [40:22] changes then the developer is happy and [40:25] by making sure then the next is [40:27] convincing different teams and to do [40:29] that uh uh the easiest thing that has [40:32] worked at least from my experience has [40:34] been quick wins. So what I mean by quick [40:36] wins is you should convince another team [40:38] that whatever you are uh implementing [40:41] has low risk but easy wins and we do [40:44] that in the case of uh let's say with [40:46] let's say a very large scale migration [40:48] change and to adopt that change first we [40:51] will run through the old code path run [40:53] through the new code path collect a set [40:55] of metrics basically a set of SLAs which [40:57] all of the teams are chasing and uh [41:00] pretty much announce that this is a [41:01] change we are making create on one pager [41:03] and these are the metrics that we will [41:05] be touching. Are you happy about it? And [41:07] if you get a no from them, then it's [41:09] probably a riskier change. So if you're [41:11] able to convince that with low risk, [41:13] you're able to get easy wins, then you [41:15] make teams happy and ultimately they [41:17] adopt your product. The third thing [41:19] which is very interesting is having a [41:21] shared literature. So what I mean by [41:23] shared literature is your ML teams, your [41:25] software engineering teams, your data [41:27] scientist team should work together to [41:29] probably create shared knowledge bases [41:31] like on calls on call even like sharing [41:34] on call rotations helps a lot with [41:36] building like uh the team culture itself [41:38] and the knowledge within the teams [41:40] itself and by understanding the pain [41:42] points of an S sur uh a software [41:45] engineer ML engineer you'll be able to [41:48] create the product that all of these [41:50] different developers are able to love [41:51] and use like for years to come. [41:56] >> Yeah. I I think when it comes to [41:59] adopting internal tools or even managed [42:02] solutions for that matter, um the first [42:04] thing that's important is understanding [42:08] what problem you're trying to tackle [42:10] with this particular tool. Um and more [42:13] specifically, how is the day-to-day of [42:15] of like an ML engineer becoming um more [42:18] frictionless or um how are they able to [42:21] scale their day-to-day work faster and [42:24] are they able to iterate more quickly [42:26] over things? Um when it for example like [42:29] taking the Sage Maker migration as an [42:32] example um we did try to uh bring about [42:36] bring some internal teams on board by [42:39] trying to have them uh train their [42:42] models on this platform and once that [42:45] was successful that was able to drive [42:47] internal adoption but also that helps us [42:50] uncover what gaps could arise in in [42:53] adopting the solution orwide or [42:55] companywide. So things like IM [42:57] permissions or like inconsistent logging [43:00] formats, those are sort of exposed when [43:02] you have um when you're trying to bring [43:05] few teams on board and once they have [43:07] successful use cases, you're more [43:09] confident about rolling it out across [43:12] across the company. Um I I think one [43:15] last thing that I wanted to touch upon [43:16] is how um this has to be a frictionless [43:20] transition as in you're not trying to [43:22] uproot the workflows of every team to [43:25] try and embed this one new tool into [43:27] their uh day-to-day but sort of more [43:30] seamlessly transition it by um [43:34] integrating it into existing pipelines. [43:36] So with with SageMaker we we tried to [43:39] integrate existing feature versions and [43:42] data sto data sources to this training [43:45] platform and and that way teams didn't [43:48] have to worry about like platform [43:51] specific setup and we tried to abstract [43:53] that complexity from them. So, and yeah, [43:56] finally supporting them through [43:59] extensive documentation, one-on-one [44:01] training sessions or office hours like [44:04] those kind of things also help ensuring [44:06] that you're getting teams to more [44:09] seamlessly transition to these [44:11] solutions. [44:12] >> Awesome. So, okay, thank you. Thank you [44:16] all so much for being on this panel. I [44:18] if we could go just one by one and re [44:20] say your name and what company you work [44:21] for so people can connect with you at [44:23] the end of this. Is that okay? Um I'm [44:25] Rajita Naik. I'm a machine learning [44:27] engineer at Lyft. [44:29] >> Balaji Muttaragan. I'm a software [44:31] engineer at Pinterest. [44:34] >> Kevin Kelly. I'm the director of a IML [44:37] enablement at American Century [44:38] Investment. And Greg Wellin, I'm the CTO [44:41] at Prove AI. Uh we have a booth. Feel [44:43] free to stop by. [44:45] >> All right. Thank you so much, everyone.