Mercedes Scraps C++ Rewrites to Automate AI Across Vehicle Fleet

Under the Hood: A Unified ML Framework Driving AI Experiences in Mercedes-Benz Vehicles — Ai4

Ai4

Standardizing regional projects into a single lakehouse monorepo and deploying models directly via TensorFlow Lite eliminated manual translation bottlenecks across two million connected vehicles.

AI-assisted summary of Under the Hood: A Unified ML Framework Driving AI Experiences in Mercedes-Benz Vehicles — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published by the Ai4 conference channel on YouTube on August 19, 2026, this session features Frankie Cancino, Senior Data Scientist at Mercedes-Benz Research & Development. The presentation outlines the architecture of Mercedes-Benz's unified machine learning framework designed to bridge data pipelines, model training, fine-tuning, and job tracking across millions of connected vehicles and internal enterprise users.

## TL;DR Operating across millions of connected vehicles and companion mobile devices powering features like Routines and Zero Layer, Mercedes-Benz previously suffered from regional silos and a painful manual Python-to-C++ code translation bottleneck. To overcome this, the engineering team unified their ecosystem around a Databricks Medallion architecture within Unity Catalog and modernized edge deployments using TensorFlow Lite alongside configuration-driven AWS cloud serving. The framework cleanly bifurcates scheduled batch processing from sub-second real-time inference guarded by strict evaluation gates and rate limits. The current roadmap focuses on expanding reusable agentic AI templates and dynamic multi-region auto-scaling across technical centers in Germany, India, Japan, and Korea.

## Atmosphere & style The presentation carries a pragmatic, engineering-focused tone that balances architectural rigor with candid reflections on the realities of enterprise modernization. Frankie delivers a measured, methodical walkthrough characterized by practical problem-solving rather than speculative hype, openly discussing the tedious friction of merging repositories and retiring legacy engineering handoffs. Watching the session feels like an authentic insider debrief from the trenches of large-scale automotive AI, tracing an emotional arc from legacy frustration to hard-won architectural coherence.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=4s) Mercedes-Benz operates at a massive digital and physical scale, treating over 2 million globally sold vehicles and millions of companion app users as active edge endpoints. - [01:39](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=99s) In-cabin intelligence spans features like 'Routines' for automated habit learning and 'Zero Layer' for dynamically predicting single next-step actions on the MBUX display. - [03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) Legacy development suffered from deep organizational silos and a major bottleneck requiring an entire team to translate Python models line-by-line into embedded C++. - [05:00](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=300s) The unified ML framework centralizes foundational primitives and documentation, shifting engineering focus from reimplementations to expanding AI across the vehicle fleet. - [06:43](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=403s) A Databricks Medallion Architecture organized in Unity Catalog processes multimodal streams into Bronze, Silver, and Gold layers to create reusable datasets for future model architectures. - [09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) Model deployments navigate distinct target environments—from constrained in-vehicle CPUs running TFLite binaries to scalable AWS cloud GPU clusters—enforced by automated evaluation gates. - [11:34](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=694s) Adopting Databricks Asset Bundles, automated CI/CD parity tests, and TensorFlow Lite packaging replaced the manual C++ rewrite process entirely. - [14:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=844s) Workloads are separated into scheduled high-throughput batch pipelines and low-latency real-time inference requiring sub-two-second responses and rate-limited API gateways. - [16:25](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=985s) Consolidating ten standalone repositories into a single monorepo required substantial refactoring into generic, modular base classes alongside executive buy-in. - [18:36](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1116s) The platform roadmap focuses on reusable agentic AI templates that give autonomous agents standardized access to internal enterprise tools and proprietary data. - [19:25](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1165s) Dynamic auto-scaling and telemetry manage unpredictable real-time inference workloads across distributed global engineering hubs in Germany, India, Japan, and Korea.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=4s) The Scale of Connected Intelligence at Mercedes-Benz

Managing machine learning across Mercedes-Benz requires operating at massive physical and digital scale. In the United States alone, Mercedes sold over 300,000 vehicles in the past year, while global sales surpassed 2 million units. In the eyes of the machine learning team, every single one of those vehicles acts as an independent edge device emitting data and running intelligent software. On top of the vehicle fleet, the companion mobile app accounts for millions of active installations, heavily promoted across dealerships to onboard drivers into personalized digital profiles and connected vehicle features.

Connecting these endpoints creates a vast distributed network where hundreds of thousands of concurrent nodes constantly interact with backend machine learning systems. Building AI products under these conditions means handling diverse streaming modalities, high concurrency, and varied compute constraints across distinct physical environments.

- Over 300,000 vehicles sold annually in the US; over 2 million sold globally. - Multi-million-download companion mobile application acting as a secondary edge platform. - Every vehicle and mobile installation functions as an active edge endpoint.

[01:39](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=99s) AI Products in the Cabin: Routines, Zero Layer, and Beyond

Mercedes-Benz deploys machine learning across a wide array of in-cabin systems and customer-facing touchpoints. Among the flagship features built and shipped by the ML team is the "Routines" application in newer vehicle generations. Routines observes repeated driver behaviors over time and automatically learns patterns to offer intelligent automation suggestions—such as adjusting cabin settings or initiating actions without manual intervention.

Another major production deployment is the "Zero Layer" interface on the MBUX home screen. Instead of forcing drivers to navigate through submenus or rely solely on voice commands, Zero Layer dynamically predicts and surfaces the single next action the driver wants on the primary screen. For instance, if a driver habitually calls a specific contact upon leaving work, the system places a one-touch call button directly on the display at that exact moment. Beyond these predictive interfaces, the ML team powers autonomous driving and ADAS capabilities, navigation routing, massage and comfort packages, multimodal voice assistants, and custom driving modes.

```mermaid flowchart TD accTitle: Mercedes-Benz Connected ML Architecture accDescription: Architectural pipeline showing data ingestion from edge devices through Databricks Medallion processing to deployment targets.

subgraph EdgeSources["Edge Data Sources"] V["Vehicles (2M+ Globally)"] M["Mobile App (Millions of Users)"] end

subgraph DataPlatform["Databricks Lakehouse & Unity Catalog"] B["Bronze: Raw Multimodal Ingestion"] S["Silver: Cleaned & Filtered"] G_AI["Gold: ML Features & Embeddings"] G_DS["Gold: Data Science & Analytics"] B --> S S --> G_AI S --> G_DS end

subgraph TrainingEval["Model Pipeline & Governance"] T["Fine-Tuning (LLMs, VLMs, Traditional ML)"] E{"Strict Evaluation Gate"} G_AI --> T T --> E end

subgraph ServingTargets["Deployment Environments"] EdgeDep["Edge: In-Vehicle CPU / TFLite"] CloudDep["Cloud: AWS Real-Time APIs / GPUs"] BatchDep["Cloud: Scheduled Batch Jobs"] end

EdgeSources --> B E -->|"Passed Criteria"| EdgeDep E -->|"Passed Criteria"| CloudDep E -->|"Passed Criteria"| BatchDep ```

[03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) The Pitfalls of Silos and the Python-to-C++ Rewrite Bottleneck

Before standardizing their workflows, the machine learning organization struggled with deep organizational and technical silos. Teams working across different geographical regions—such as the US and Germany—frequently recreated components from scratch without knowing identical tools already existed elsewhere in the company. Onboarding engineers to new initiatives required reinventing repository structures, tooling setups, interaction cadences, and project management processes.

The most severe bottleneck stemmed from the language barrier between research and production. The ML team developed experimental models in Python, but in-vehicle deployment required compiling to embedded C++. This handoff required an entire secondary engineering team dedicated solely to translating Python models line-by-line into C++. The translation process introduced massive miscommunications, parity bugs, and prolonged verification cycles. Downstream, data scientists investigating customer adoption and feature usage had to navigate entirely disparate telemetry formats and reporting mechanisms for every single project, asking "How do I do my job?" across siloed stacks and stretching delivery timelines to an agonizing pace.

[05:00](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=300s) The Vision for a Unified Machine Learning Ecosystem

To eliminate these bottlenecks, the engineering team pitched leadership on building a single unified ML framework founded on reusability, shared knowledge, and velocity. Instead of maintaining disconnected codebases, the team sought to centralize foundational machine learning primitives so that components built for one feature could immediately serve subsequent projects.

Crucially, this shift prioritized open experimentation and documentation. Historically, accessing the experimental results or design documents of an adjacent project required navigating formal access requests and administrative hurdles. Centralizing documentation, telemetry, and shared abstractions created organizational visibility. Accelerating development cycles directly translated into greater departmental reach, shifting engineering bandwidth from low-level reimplementations toward expanding intelligent features across the vehicle lineup.

[06:43](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=403s) Data Ingestion and the Medallion Pipeline in Unity Catalog

Data is the foundational backbone of the unified framework. The system ingests high-frequency multimodal streams directly from connected vehicles and mobile apps, including audio signals, computer vision streams, environmental sensor data, and mobile interaction logs.

To process these disparate formats systematically, Mercedes-Benz leverages a Medallion Architecture managed within Databricks: - **Bronze Layer:** Ingests raw, unfiltered streaming telemetry directly from edge devices and user interfaces. - **Silver Layer:** Cleans, validates, and standardizes records across disparate vehicle models and software versions. - **Gold Layer (AI & ML):** Generates structured feature tables, high-dimensional vision embeddings, audio representations, and fine-tuning datasets. - **Gold Layer (Data Science & Analytics):** Aggregates business metrics, customer funnel interactions, and usage statistics independently from model training pipelines.

All processed datasets are registered within Databricks Unity Catalog as an open, governed data lake. Preserving raw and intermediate data in a shared catalog ensures historical telemetry becomes a reusable "gold mine" for future initiatives—such as training emerging vision-language models on imagery collected months prior—without needing to engineer bespoke collection pipelines.

[09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) Model Deployment Across Edge Vehicles, Mobile, and Cloud

Deploying AI models across Mercedes-Benz requires navigating vastly different execution environments, each with unique computational constraints. The edge vehicle environment is strictly constrained: models execute primarily on in-cabin CPUs without current access to onboard GPU acceleration, requiring lightweight, highly optimized binaries. Conversely, cloud-hosted deployments provide scalable CPU and GPU clusters where memory and compute are flexible, though governed by infrastructure cost efficiency. Mobile devices represent a third distinct environment with platform-specific operating constraints.

The deployment framework orchestrates diverse model families across these environments, including Large Language Models (LLMs), Vision-Language Models (VLMs), Audio-Language Models (ALMs), traditional tabular machine learning models, recommendation systems, and autonomous agentic workflows. Continuous telemetry monitors infrastructure latency, inference throughput, and downstream user retention funnels. To ensure strict quality governance, no model reaches production without clearing rigorous automated evaluation gates, utilizing LLM-as-a-judge evaluation frameworks alongside deterministic accuracy and latency thresholds.

[11:34](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=694s) Platform Architecture: Databricks Bundles, CI/CD, and AWS Serving

The core technical platform relies on Databricks Asset Bundles to orchestrate end-to-end tasks, job runs, and parameter configurations directly within Python workflows. All engineering assets live in a centralized GitHub monorepo backed by comprehensive CI/CD pipelines and automated integration testing.

``` ┌─────────────────────────────────────────────────────────────────┐ │ GitHub Monorepo │ │ - Databricks Asset Bundles (Orchestration & Parameter Passing) │ │ - Automated CI/CD & Parity Testing Pipelines │ └────────────────┬───────────────────────────────┬────────────────┘ │ │ ▼ ▼ ┌─────────────────────────────────┐ ┌───────────────────────────┐ │ Databricks Data Lake │ │ AWS Production Serving │ │ - Unity Catalog Governance │ │ - Config-driven deploys │ │ - Medallion Ingestion Tables │ │ - Dynamic GPU/CPU nodes │ └─────────────────────────────────┘ └───────────────────────────┘ ```

A major engineering milestone was deprecating the error-prone manual Python-to-C++ translation handoff. The team modernized their edge pipeline by adopting TensorFlow Lite, packaging production-ready Python pipelines directly for target hardware. During the transition, automated GitHub Actions pipelines ran regression suites across both Python and C++ implementations to detect discrepancies, which routinely failed parity checks initially before full automation stabilized the stack.

For real-time cloud hosting, the architecture leverages AWS infrastructure fronted by custom configuration-driven deployment wrappers. Engineers define resource requirements in a simple configuration file—specifying instance quantities and GPU types—and a unified CLI deploys models, interactive applications, or autonomous agents behind standardized, abstracted security and authentication layers.

[14:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=844s) Serving Patterns: High-Throughput Batch vs. Low-Latency Real-Time Inference

The framework divides model serving into two distinct operating paradigms: scheduled batch processing and dynamic real-time inference.

| Dimension | Batch Processing | Real-Time Inference | |:--- |:--- |:--- | | **Execution Trigger** | Scheduled cron/event pipelines | Dynamic, on-the-fly user requests | | **Compute Profile** | Predictable, highly optimizable, retryable | Always-on, variable-load CPU/GPU clusters | | **Latency Tolerance** | High (minutes to hours) | Sub-second (predictions expire in 1–2 seconds) | | **Data Governance** | Abstracted behind cached APIs to protect lakehouse | Fronted by secure API gateways and rate limiters | | **Key Challenges** | Forward-looking predictions and data staleness | Traffic surges, GPU costs, and shared-service stability |

Batch serving handles scheduled, forward-looking predictions where jobs can be retried safely upon failure. To prevent external services from querying internal Unity Catalog tables directly, an abstraction layer synchronizes and serves batch predictions through structured APIs. In contrast, real-time inference powers time-sensitive vehicle and mobile interactions where an inference result loses all utility if not delivered within one to two seconds. Because real-time models are often exposed as shared internal services across multiple vehicle features, the platform incorporates strict rate limits and autoscaling policies to prevent unexpected request surges from cascading into system-wide outages.

[16:25](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=985s) Organizational Buy-In and the Reality of Monorepo Refactoring

Transitioning an enterprise automotive manufacturer to a unified machine learning framework requires navigating substantial organizational and technical friction. Shifting away from legacy practices—such as transitioning the dedicated C++ translation team to new responsibilities once the ML group began deploying production models directly via TensorFlow Lite—demanded deep, transparent alignment with executive leadership to justify the architectural evolution.

On an engineering level, merging ten separate project repositories into a unified monorepo generated extensive technical debt and tedious code refactoring. Standardizing fragmented codebases required creating generic base classes that could be extended via modular child classes for audio, vision, and sensor modalities. Building this modular foundation remains an ongoing, iterative process that demands patience and sustained cross-team collaboration across the engineering organization.

[18:36](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1116s) Roadmap and Agentic AI Templates

Driving large-scale adoption of an internal machine learning framework takes patience and deliberate alignment. It requires clearly mapping the long-term vision, understanding developer pain points, and communicating the value in a way that truly "inspires people to want to work on this with you." Looking forward, a primary focus area is leaning heavily into agentic workflows. Mercedes-Benz already has functional AI agents running in production that are equipped with access to internal tools, proprietary data points, downstream models, and core enterprise applications.

The immediate initiative is lowering the barrier to entry for building these systems across the enterprise. Just as the team engineered modular, generic classes across the foundational ML framework, they are now introducing standardized templates for autonomous agents. By packaging agent design patterns into reusable templates, internal engineering teams can deploy agents that interface seamlessly with enterprise systems, move faster, and cut down boilerplate development time.

```mermaid flowchart TD accTitle: Future Architecture and Enterprise Priorities subgraph AgenticFramework["Agentic Framework & Templates"] T["Generic Agent Templates"] --> A["Autonomous Agents"] A --> Tools["Internal Tools & Models"] A --> Data["Proprietary Company Data"] end subgraph GlobalOps["Global Infrastructure"] GH["Global Hubs<br/>(Germany, India, Japan, Korea)"] --> I["Real-Time Inference Layer"] I --> AS["Automated Dynamic Scaling"] I --> Mon["Telemetry & Awareness"] end subgraph OrgValue["Organizational Impact"] AgenticFramework --> OrgValueNode["Deep Enterprise Integration"] GlobalOps --> OrgValueNode OrgValueNode --> ROI["Demonstrable ROI & Cultural Change"] end ```

- **Current Agent Capabilities**: Agents already query proprietary corporate data points, trigger internal application APIs, and chain tasks across various specialized models. - **Agent Templating Goal**: Building generic base classes and scaffolding to ensure standardized security, tool-use integration, and rapid time-to-deployment across diverse business units.

---

[19:25](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1165s) Global Auto-Scaling and Enterprise Value Integration

Another major operational focus is refining auto-scaling for real-time inference workloads. Managing capacity across a global enterprise is challenging because infrastructure teams cannot easily anticipate what distributed teams in Germany, India, Japan, or South Korea are going to execute at any given moment. Central platform teams frequently receive little advance communication about whether an overseas colleague is casually running ad-hoc experimental tests or ramping up heavy, production-grade demand. Implementing intelligent, dynamic auto-scaling and heightened system observability ensures the platform adapts flexibly to unexpected traffic spikes without manual intervention.

Ultimately, the overarching objective is deepening integration across the broader Mercedes-Benz organization. Proving sustained value hinges on embedding this unified framework into more everyday business processes, which establishes a clear return on investment (ROI) and encourages positive organizational and cultural change around AI adoption. Wrapping up as the session reached its time limit, Frankie invited attendees to connect for offline discussions and questions.

- **Global Operational Hubs**: Serving real-time inference across geographically distributed technical centers spanning Germany, India, Japan, and Korea. - **Traffic Unpredictability**: Platform challenges stem from unannounced workload swings between early-stage sandbox testing and production-level queries. - **Strategic Vision**: Expanding integration deep into enterprise workflows to demonstrate quantifiable ROI and foster company-wide collaboration.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=4s) In the eyes of the machine learning team, every single one of those vehicles acts as an independent edge device emitting data and running intelligent software. — Frankie Cancino

> [01:39](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=99s) Instead of forcing drivers to navigate through submenus or rely solely on voice commands, Zero Layer dynamically predicts and surfaces the single next action the driver wants on the primary screen. — Frankie Cancino

> [03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) The most severe bottleneck stemmed from the language barrier between research and production. The ML team developed experimental models in Python, but in-vehicle deployment required compiling to embedded C++. — Frankie Cancino

> [03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) How do I do my job? — Frankie Cancino

> [06:43](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=403s) Data is the foundational backbone of the unified framework. — Frankie Cancino

> [06:43](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=403s) Preserving raw and intermediate data in a shared catalog ensures historical telemetry becomes a reusable 'gold mine' for future initiatives. — Frankie Cancino

> [09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) The edge vehicle environment is strictly constrained: models execute primarily on in-cabin CPUs without current access to onboard GPU acceleration, requiring lightweight, highly optimized binaries. — Frankie Cancino

> [11:34](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=694s) A major engineering milestone was deprecating the error-prone manual Python-to-C++ translation handoff. — Frankie Cancino

> [14:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=844s) Real-time inference powers time-sensitive vehicle and mobile interactions where an inference result loses all utility if not delivered within one to two seconds. — Frankie Cancino

> [18:36](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1116s) inspires people to want to work on this with you. — Frankie Cancino

> [19:25](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1165s) Proving sustained value hinges on embedding this unified framework into more everyday business processes, which establishes a clear return on investment (ROI) and encourages positive organizational and cultural change around AI adoption. — Frankie Cancino

## Mentioned Frankie Cancino, Mercedes-Benz, Mercedes-Benz Research & Development, Ai4, MBUX, Routines, Zero Layer, Databricks, Databricks Asset Bundles, Unity Catalog, AWS, GitHub, GitHub Actions, TensorFlow Lite, Python, C++

## Video description Frankie Cancino - Senior Data Scientist, Mercedes-Benz Research & Development

Managing AI at Mercedes-Benz scale requires a seamless bridge between data and deployment. This session breaks down our unified ML framework: a single ecosystem orchestrating data pipelines, model training, fine-tuning, and comprehensive job tracking. Learn how we architected this platform to support the dual demands of low-latency real-time inference and high-throughput batch processing for vehicles and internal users.

[Source](https://www.youtube.com/watch?v=Ok8BV5Gj128)

中文简报

奔驰放弃C++重写，实现车队AI自动化 通过将各区域项目规范整合为统一的湖仓一体单体代码库（monorepo），并直接借助 TensorFlow Lite 部署模型，消除了覆盖 200 万辆联网汽车的人工转换瓶颈。 梅赛德斯-奔驰（Mercedes-Benz）在数百万辆联网汽车及配套移动设备上支持 Routines 和 Zero Layer 等功能，此前曾深受区域孤岛以及繁琐的人工 Python 转 C++ 代码瓶颈困扰。为解决这一难题，工程团队围绕 Unity Catalog 内的 Databricks Medallion 架构统一了生态系统，并结合 TensorFlow Lite 与配置驱动的 AWS 云服务实现了边缘部署的现代化。该框架将定时批处理与受严格评估门控及速率限制保护的亚秒级实时推理清晰分离。当前的路线图重点在于德国、印度、日本和韩国的技术中心推广可复用的 Agentic AI 模板，并实现动态多区域自动伸缩。 梅赛德斯-奔驰将全球年销超 200 万辆的汽车及数百万伴侣 App 用户视作活跃边缘节点，支撑座舱内的 Routines 习惯学习与 MBUX“Zero Layer”零层级下一动向预测等智能化体验。 (0:04) 针对此前需专职团队将 Python 模型逐行手动转写为嵌入式 C++ 的严重瓶颈与组织孤岛，工程团队构建了统一 ML 框架，通过集中基础原语与共享文档加速全流程开发。 (3:06) 数据平台依托 Databricks Unity Catalog 建立多模态奖章架构（Bronze、Silver 与 Gold 层），将车载传感器与交互数据沉淀为统一资产，供未来各类模型直接复用训练。 (6:43) 技术架构借助 Databricks Asset Bundles、CI/CD 自动化校验与 TensorFlow Lite 打包彻底淘汰了 C++ 手动重写，同时将任务严谨区分为离线高吞吐批处理与要求 1 至 2 秒内极速响应的实时推理。 (11:34) 团队正进一步将已投产的 AI Agent 沉淀为可复用的标准模板以降低全司构建门槛，并优化面向德国、印度等多国研发中心的全球实时推理动态弹性伸缩。 (18:36)

Reading summary

To support connected features across millions of vehicles, Mercedes-Benz replaced regional engineering silos and manual Python-to-C++ rewrites with a unified machine learning framework spanning edge, mobile, and cloud environments. Distributed Edge Scale and In-Cabin Applications (0:04) Operating AI across Mercedes-Benz involves hundreds of thousands of concurrent endpoints. Global sales exceed 2 million vehicles annually—with over 300,000 units sold in the United States—supplemented by a companion mobile app with millions of active downloads. In the ML team's architecture, each vehicle functions as an independent edge node generating multimodal streaming telemetry while executing localized software. Production deployments target cabin automation and predictive driver interfaces. The "Routines" application analyzes recurring driver behaviors over time to automate vehicle adjustments. Similarly, the MBUX "Zero Layer" interface predicts driver intent to surface single-action buttons on the primary display at relevant moments, such as placing habitual phone calls upon leaving work. The ML organization also powers ADAS, navigation routing, comfort packages, and multimodal voice interactions. Resolving Silos and the Python-to-C++ Bottleneck (3:06) The engineering organization previously suffered from geographical fragmentation between regional hubs like the US and Germany, causing teams to repeatedly build duplicate components and bespoke telemetry formats. Onboarding was delayed by differing repository setups and administrative obstacles that restricted access to adjacent experimental findings. The most severe friction occurred in the handoff between research and production. Experimental models developed in Python had to be translated line-by-line into embedded C++ by a dedicated engineering team for in-cabin vehicle deployment. This manual translation cycle introduced persistent miscommunications, prolonged verification phases, and parity bugs that slowed feature release cadences. Unified Medallion Data Architecture (6:43) To streamline ingestion, Mercedes-Benz established a Databricks Medallion Architecture governed through Unity Catalog. Raw multimodal streams from vehicles and mobile devices enter a Bronze layer before being cleaned, validated, and standardized in the Silver layer across different vehicle models and software versions. The data then forks into two Gold layers: one generating feature tables, audio representations, and vision embeddings for model fine-tuning, and another tracking customer analytics and adoption metrics. Preserving raw and processed assets within an open lakehouse allows older sensor streams and camera feeds to be repurposed for subsequent initiatives, such as training new vision-language models, without establishing custom data pipelines for each project. Monorepo CI/CD and Edge-to-Cloud Serving (9:30) The engineering team consolidated ten separate project repositories into a single GitHub monorepo managed with Databricks Asset Bundles. To eliminate the C++ translation layer, the team introduced TensorFlow Lite to package Python workflows directly for in-vehicle execution, using automated GitHub Actions parity tests to catch regression discrepancies. Because vehicles run models primarily on in-cabin CPUs without onboard GPU acceleration, edge binaries must remain strictly optimized. Cloud serving handles both scheduled batch jobs and dynamic real-time inference on AWS. Batch pipelines process retried jobs behind structured caching APIs to shield the lakehouse, while real-time inference handles sub-second requests using auto-scaling clusters, configuration-driven wrappers, and strict rate limits to absorb traffic surges from shared vehicle services. Automated evaluation gates, including deterministic metrics and LLM-as-a-judge frameworks, govern production promotions. Agentic Templates and Global Auto-Scaling (18:36) Recent platform expansion emphasizes agentic AI workflows and global capacity management. The company currently runs production agents that access internal enterprise APIs, proprietary data repositories, and downstream models. To reduce boilerplate coding across business units, the team is deploying standardized agent templates modeled on the framework's modular base classes. Operationally, the platform must dynamically auto-scale to absorb unpredictable query volumes from distributed engineering hubs in Germany, India, Japan, and South Korea, where sandbox experiments and heavy production traffic can occur without advance notice. Mercedes-Benz's platform strategy centers on demonstrating measurable return on investment by integrating the unified ML stack into daily enterprise workflows, though engineering teams continue to manage the technical debt and code refactoring required to maintain modular, multi-modal monorepo foundations.

精炼阅读摘要

Mercedes-Benz 机器学习团队通过构建统一的端到端 ML 框架，消除了跨国开发孤岛与繁琐的 Python 到 C++ 手动重写流程，实现车端、移动端及云端 AI 体验的高效协同与规模化交付。 车端与移动端边缘智能的连接规模 (0:04) Mercedes-Benz 的机器学习系统面临庞大的物理与数字运行规模。该公司过去一年在美国售出逾 30 万辆汽车，全球年销量更超过 200 万辆，每一辆上路车辆在机器学习团队眼中都是持续产生流式数据并运行智能软件的独立边缘节点。与车辆配套的官方移动应用程序在经销商推广下拥有数百万活跃用户，构成了次级边缘平台。这些终端共同组成了一个由数十万个高并发节点组成的分布式网络，要求后端系统能够稳定处理多模态数据流并适配多样化的物理计算约束。 在座舱内部，团队已落地了多项核心功能。例如，较新车型中的“Routines”应用程序能够随时间观察驾驶员的重复习惯并自动生成智能化建议；MBUX 系统的主屏幕“Zero Layer”零层界面能够动态预测驾驶员下一步操作，在离开公司时直接在主界面浮出单键拨号选项，无需进入多级子菜单。此外，该体系还支撑着辅助驾驶（ADAS）、导航路径规划、座椅按摩舒适包、多模态语音助手及定制化驾驶模式等车载智能化场景。 打破研发孤岛与语言转换瓶颈 (3:06) 在统一架构确立之前，团队深受地域分割与技术孤岛的困扰。美国与德国等不同地区的团队频繁重复造轮子，工程师接手新项目往往需要重建仓库结构、开发工具与协作规范。最严峻的技术瓶颈在于研发与生产环境之间的语言鸿沟：算法团队在 Python 环境中完成实验与模型开发，而车载硬件部署要求将其编译为嵌入式 C++ 代码。这导致公司必须维持一支专职工程团队，逐行将 Python 模型翻译为 C++。 这种人工转译流程不仅耗费数周时间，还频繁引发代码逻辑不一致、功能对齐缺陷与漫长的交叉验证周期。下游的数据科学团队在分析用户采纳度与功能表现时，面对不同项目割裂的遥测格式与报告机制，数据获取困难，直接拖慢了新功能的落地节奏。为此，工程团队推动了由底层基础原语、共享知识库与标准化文档构成的统一平台转型。 基于 Databricks 与 AWS 的统一技术栈 (6:43) 统一框架的核心依赖于 Databricks 与 AWS 构建的端到端技术链路。针对来自车载传感器、车载视觉、音频信号及移动端的原始数据，团队在 Databricks 内采用奖牌架构（Medallion Architecture）实施治理：Bronze 层摄取原始流式数据；Silver 层完成跨车系及不同软件版本的数据清洗与标准化校验；Gold 层分别沉淀用于训练的特征向量与供数据科学分析的业务指标。所有数据集统一注册至 Databricks Unity Catalog 进行数据湖治理，使历史遥测数据无需重复采集即可复用于新型多模态模型的训练。 在工程落地层面，团队将十个独立的工程项目合并为单一 GitHub Monorepo，并利用 Databricks Asset Bundles 管理任务编排与参数配置。针对边缘交付，团队引入 TensorFlow Lite 直接打包 Python 生产管线并部署至车载 CPU，完全淘汰了手工 C++ 转译流程，并通过 GitHub Actions 自动化回归流水线校验数值一致性。云端推理则基于 AWS 构建，工程师仅需通过配置文件指定 GPU 或 CPU 规格，即可使用统一 CLI 自动化发布模型或交互式服务，底层集成了标准化的安全认证与网络隔离。 批处理与实时推理的双轨服务模式 (14:04) 该框架根据业务时延敏感度将模型推理明确划分为两套运行范式。批处理架构面向高吞吐、计划驱动的前瞻性预测，具备计算资源利用率高且允许失败重试的特性。为保护内部 Unity Catalog 数据湖不被直接调用，批处理生成的结果经过同步层抽象，通过结构化 API 安全地向外部系统提供缓存预测结果。 实时推理架构则专门承载车载交互与移动端应用等对时延极为苛刻的场景，推断结果通常必须在 1 至 2 秒内返回，逾期则丧失交互意义。由于实时模型常作为共享服务供多项座舱功能调用，系统配置了严格的 API 网关限流策略与自动化弹性伸缩机制，以防单点调用激增引发连带故障。所有模型在正式上线前，必须通过由传统准确率/时延阈值与大语言模型作为裁判（LLM-as-a-judge）组成的自动化质量评估门禁。 重构阻力与未来智能体框架演进 (16:25) 向统一框架演进的过程中面临着严峻的组织惯性与技术债挑战。放弃 C++ 手动转译需要妥善调整原团队的职责分工，并与高管层保持充分沟通；将分散的代码仓库整合进 Monorepo 带来了大量底层抽象工作，工程师必须提炼通用的基类接口，再针对音频、图像及传感器模态派生定制子类。 面向未来，Mercedes-Benz 正在加速推进企业级智能体（Agentic AI）体系建设。团队已在生产环境中部署了可自主调用内部工具、访问企业专有数据并串联下游模型的 AI Agent。为了进一步降低开发门槛，团队正将智能体设计模式抽象为标准化的脚手架模板。此外，针对德国、印度、日本和韩国等全球技术中心未提前报备的突发测试与生产调用，平台正在强化跨地域动态自适应伸缩能力，以展示明确的投资回报率并深化全公司的智能化融合。 Mercedes-Benz 通过统一代码库、标准化数据湖治理以及容器化边缘发布，克服了车载硬件资源受限与多国团队协作割裂的痛点。未来该平台能否在全球多中心高并发负载下维持成本与稳定性的平衡，并顺利推广标准化智能体模版，仍有待业务实践的持续检验。

Detailed notes — zh

## Background 本内容由 YouTube 频道 Ai4 于 2026 年 8 月 19 日发布，主讲人为梅赛德斯-奔驰研发团队的高级数据科学家 Frankie Cancino。视频简介指出，本次演讲深入剖析了奔驰如何构建统一的机器学习框架，打通数据管道、模型训练、微调与作业跟踪，以支撑数百万边缘车辆及内部用户的低延迟实时推理与高吞吐批处理需求。 ## TL;DR 面对全球数百万联网车辆与移动端 App 构成的庞大边缘计算网络，奔驰针对以往项目孤岛与 Python 算法需交由独立团队重写为 C++ 的痛点，搭建了全公司级的统一 ML 框架。该架构基于 Databricks 奖牌架构治理传感器、视觉与语音等多模态数据，通过 Monorepo、CI/CD 与 TensorFlow Lite 实现 Python 模型向车端边缘的直接交付，并设立了严格的自动化评估准入门槛。在服务模式上，框架精细区分了定时批处理与低延迟实时推理，未来将进一步向标准化 Agentic 架构、全球弹性扩缩容与跨业务深度集成方向演进。 ## Atmosphere & style 整场演讲节奏紧凑、干货密度极高，呈现出务实且极具工业落地色彩的技术分享风格。主讲人毫不避讳地分享了合并旧仓库的技术债、组织职责重组的阵痛等真实挑战，语气严谨而真诚。在短短 20 分钟的时间里，演讲以清晰的架构逻辑与扎实的工程实践，为观众勾勒出传统豪华车企在智能化转型中攻坚克难的生动图景。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=4s) 奔驰全球年交付超 200 万辆联网汽车，加上数百万移动 App 用户，共同构成了庞大的车端与移动端双重边缘计算网络。 - [01:31](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=91s) 车端已落地习惯学习系统 Routines 与主屏一键直达 Zero Layer 等代表性 AI 功能，并向座舱舒适配置、导航和 ADAS 全面渗透。 - [03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) 早期研发深陷烟囱式孤岛困境，且算法验证用的 Python 必须交由独立工程团队重写为 C++ 才能上车，导致交付周期漫长且逻辑偏差频发。 - [04:52](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=292s) 统一 ML 框架确立了四大核心支柱：提高代码复用性、促进跨组织知识共享、大幅压缩研发周期以及拓展团队的业务辐射半径。 - [06:40](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=400s) 依托 Databricks 奖牌架构与 Unity Catalog 统一多模态数据底座，确立多模态数据即便当下未用也规范清洗归档的长期资产策略。 - [09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) 构建覆盖轻量级车载 CPU、云端弹性集群和移动端的端云分级部署策略，并对大模型、多模态及传统算法提供全谱系支持。 - [09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) 设立严格的自动化模型评估门禁（Evaluation Gate）与 LLM-as-a-Judge 机制，模型关键指标未超 baseline 严禁推送至生产环境。 - [11:34](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=694s) 基于 GitHub Monorepo 与 Databricks Asset Bundles 打通工程底座，并通过车端集成 TensorFlow Lite 实现了 Python 算法端到端直达车端，彻底消除了人工 C++ 重写环节。 - [13:58](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=838s) 将服务范式清晰划分为定时计划批处理与实时低延迟推理，前者通过受控 API 隔离数据底座，后者通过常驻 GPU 与限流机制应对突发流量。 - [16:18](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=978s) 框架演进的核心挑战在于组织协同与技术债治理，不仅需要重新规划原 C++ 工程团队的职责定位，还要耗费大量精力将旧项目抽象重构为通用类库。 - [18:36](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1116s) 未来团队将聚焦三大核心方向：推行标准化企业级 Agent 模板、构建应对全球多国不可预测负载的智能 Auto-scaling 机制，以及深度打通业务以持续量化呈现 ROI。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=4s) 演讲背景与奔驰的万物互联规模 来自梅赛德斯-奔驰研发团队的高级数据科学家 Frankie Cancino 展开了这场关于车企级 AI 平台演进的分享。他谈到，自己的机器学习团队在奔驰经历了一段漫长且充满挑战的旅程，本次演讲的核心就是向大家拆解团队如何为整个奔驰构建统一的机器学习框架。这套框架绝不仅仅服务于单一团队，而是作为全公司的底层赋能平台，支撑从实时低延迟推理到定时批处理的各种复杂工作流。鉴于演讲时间只有短短 20 分钟，他热情地鼓励对该架构感兴趣的同行在会后直接找他进行深入交流。 谈及奔驰所面临的工程规模，数字本身就极具冲击力。仅去年一年，奔驰在北美市场的销量就超过 30 万辆，全球交付量更突破 200 万辆。在数据科学团队的眼中，每一辆奔驰汽车在本质上都是一台高度复杂的边缘计算设备（Edge Device）。除此之外，奔驰的官方手机应用程序去年也达到了数百万级的下载量，并且随着线下经销商在车主提车时积极引导下载、注册车主档案，这一用户基数还在快速膨胀。数百万辆上路行驶的车辆与数百万个活跃的移动端 App，共同构成了奔驰庞大的边缘计算与数据源网络。 - **全球新车出货规模**：美国年销量突破 30 万辆，全球年销量突破 200 万辆，每一辆车都是独立运行的边缘节点。 - **移动生态触达**：官方 App 拥有数百万级下载量，与车端共同构成双重边缘计算场景。 ### [01:31](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=91s) 车端 AI 功能全景：从 Routines 到零层级推荐 在如此庞大的硬件与用户网络之上，机器学习团队交付了极其丰富的 AI 产品与功能矩阵。其中一个代表性落地应用是“Routines（日常习惯例行程序）”，它已被部署到奔驰新一代车型的生产环境中。其核心逻辑在于：如果系统感知到车主反复在特定场景执行某种操作，模型便会自动学习这一习惯模式，并主动为车主提供智能化建议，甚至实现全自动触发。另一个极具辨识度的落地产品是已推出数年的“Zero Layer（零层级交互界面）”，它能够在车机主屏幕上实时预测车主接下来的潜在意图，实现“单键直达”。例如，若车主每天下班离开公司时习惯给某人打电话，系统无需车主翻找三四级菜单或唤起语音助手，便会直接在首页高亮展示拨号建议，真正做到一键呼叫。 除了 Routines 与 Zero Layer，AI 团队的技术栈还全面渗透到了智能座舱与整车的各个领域，包括路线导航优化、自动驾驶与高级驾驶辅助系统（ADAS）、座椅按摩等舒适性配置联动、车载语音助手以及即将推出的情景模式（Modes）等。所有这些面向车端和移动端的功能，与数百万台互联设备相乘，意味着团队每天都要在极高维度上应对海量的数据流与极其复杂的机器学习工程挑战。 ```mermaid flowchart TD accTitle: Mercedes-Benz AI Feature Matrix and Edge Footprint subgraph Fleet["分布式边缘硬件层 (Edge Footprint)"] V["200万+ 全球联网车辆 (Edge CPU)"] M["数百万级 移动端 App 客户端"] end subgraph Features["智能化体验矩阵 (AI Products & Features)"] R["习惯推荐系统 (Routines)"] Z["主屏一键直达 (Zero Layer)"] C["舒适与座舱体验 (座椅按摩 / 情景模式)"] D["行驶与交互系统 (ADAS / 导航 / 语音助手)"] end Fleet --> Features ``` ### [03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) 跨部门数据孤岛与 Python-C++ 重构的研发困境 然而在团队成立之初，工程协作模式完全处于严重的“烟囱式”孤岛状态（Silos），整体研发运转十分吃力。当时最突出的痛点是各团队之间频繁地“重复造轮子”——不仅跨国团队之间存在重复开发，甚至同一个组内的工程师由于分属不同项目，也会各自从零编写完全相同的底层逻辑。跨团队沟通成本极高，而且存在严重的技术断层。最典型的历史遗留问题是：机器学习团队在前期探索和算法实验时主要使用 Python，但当功能要真正部署进汽车嵌入式环境时，必须依赖另一个完全独立的工程团队用 C++ 将 Python 代码全盘重写一遍。 这种开发模式带来了极其繁重的反复沟通与联调开销。每次开启一个新项目，工程师都要重新摸索代码仓库放在哪里、依赖哪些系统环境、采用何种敏捷流程以及使用哪个 Jira 看板。更糟糕的是功能上线之后的维护与运营阶段：当下游数据科学家试图去分析用户行为、评估某项功能在各车型上的实际使用表现时，会沮丧地发现每一个独立项目的底层埋点和数据结构都大相径庭，连最基本的数据提取方式都无法复用。这种割裂的链路导致了超长的研发交付周期，从算法原型落地到车端生产环境往往耗时漫长。 - **代码二次翻译壁垒**：早期算法验证使用 Python，车端落地需完全交由独立工程团队重写为 C++，导致逻辑偏差频发且耗时巨大。 - **项目孤岛与环境异构**：不同项目之间的工具链、代码托管位置、敏捷协作机制互不相通，新项目冷启动成本极高。 - **数据分析标准割裂**：各产品上线后监控与日志规范不统一，数据科学家无法采用标准化流程评估用户行为。 ### [04:52](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=292s) 统一 ML 框架的四大核心愿景 为了打破这一僵局，团队决心向高层领导与各利益相关方提出建设“统一机器学习框架（Unified ML Framework）”的宏大愿景。该愿景建立在四大核心支柱之上：第一是**极高的代码复用性（Reusability）**。由于全公司处理的底层数据与机器学习任务具有高度相似性，必须将通用逻辑抽象沉淀到底层通用库中，让所有人都能在其上快速叠加业务逻辑。第二是**跨组织知识共享（Knowledge Sharing）**。过去每个项目在模型微调、特征工程和技术文档上的探索成果都被严格封锁在各自的项目权限内，查阅资料需要层层申请审批；新框架要求最大程度开放实验洞见与文档资产，使全体工程师都能直接借鉴既有经验。 第三是**大幅压缩研发周期（Faster Development）**。对于非汽车传统软件背景出身的工程师而言，汽车行业缓慢的部署节奏曾是最大的痛点之一，统一框架的目标正是通过标准化流程实现功能敏捷迭代。第四是**拓展团队业务辐射半径（Increased Reach）**。研发效率的大幅提升能够让团队释放精力去孵化更多前沿 AI 产品，深度赋能奔驰各业务条线，持续优化既有在线系统，从而彻底告别过去周而复始的低水平重复建设。 ### [06:40](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=400s) 数据基础架构：Medallion 奖牌架构与多模态共享资产 在搭建统一框架的过程中，最耗费心力且至关重要的第一步就是重构数据底座。遍布全球的奔驰车辆与移动 App 时刻都在源源不断地回传遥测数据，涵盖的模态极其多样——包括传感器时序序列、车内外摄像头视觉数据、语音音频信号以及 App 端的用户点击流和页面访问路径。在生成式 AI 爆发的当下，多模态数据的整合难度更是呈指数级上升。为此，团队深度依托 Databricks 平台，落地了标准的“奖牌架构（Medallion Architecture）”，构建起从原始数据到高质量业务资产的分层流转管道。 在这一架构中，所有设备收集的原始多模态数据首先无差别灌入青铜表（Bronze Tables）。随后，数据链路分流为两条并行主线：一条专门面向机器学习与生成式 AI 模型，对数据进行深度特征工程处理（如提取视觉 Embeddings 或音频表征特征）；另一条则经过清洗转换供给数据科学与商业分析团队，因为分析师所关心的业务指标维度与底层 AI 模型所需的特征矩阵截然不同。所有清洗提纯后的高价值数据均沉淀入白银（Silver）与黄金（Gold）层，并统一注册在 Unity Catalog 数据湖中。团队确立了一条铁律：即便某些图像或多模态数据在当前项目中未被立刻使用，只要未来可能对其他潜在产品具有价值，就必须规范清洗并永久归档，因为“这些多模态数据在未来 6 个月内随时可能变成金矿，一旦中途遗失，就再也无法挽回”。 ```mermaid flowchart LR accTitle: Medallion Multi-Modal Data Lake Architecture subgraph Ingestion["多模态数据摄入"] Raw["车辆遥测 / 视觉图像 / 语音 / 传感器 / App点击流"] end subgraph Medallion["Databricks 奖牌架构"] B[("Bronze 原始数据层")] S[("Silver 清洗转换层")] G[("Gold 业务聚合层")] end subgraph Consumption["下游消费与资产沉淀"] ML["AI/ML 特征工程 (Embeddings/多模态微调)"] BI["数据科学与业务分析 (漏斗分析/功能监控)"] UC[("Unity Catalog 数据资产湖")] end Raw --> B B --> S --> G G --> ML G --> BI ML --> UC BI --> UC ``` ### [09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) 端云多目标部署策略与严格的模型准入评估 在模型部署层面，由于硬件约束极其悬殊，团队构建了一套能够覆盖全场景的分级部署矩阵。车端作为典型的边缘计算环境，计算资源极其受限，主要依靠车载轻量级 CPU 运行，目前甚至无法直接调用车端 GPU 算力；而在云端部署环境下，算力资源几乎没有硬性上限，主要考量点在于云基础设施的成本控制；与此同时，移动端 App 又具备完全不同的操作系统与资源约束。目前，该统一框架能够支持包括大语言模型（LLM）、视觉语言模型（VLM）、音频语言模型、传统推荐系统机器学习算法以及自主 AI Agent（智能体）在内的全谱系模型交付。 在整个工程流中，团队将模型评估（Evaluation Gate）置于决定性的准入地位。无论业务端提出何种产品需求，还是算法工程师自行优化迭代，任何模型在进入生产环境之前都必须通过极其严格的定量评估红线。只有当新模型的关键性能指标明确超越上一代 baseline，或者完全契合新产品的准入标准时，CI/CD 流水线才会放行部署。与此同时，系统对在线运行状态进行全生命周期监控：数据科学侧持续追踪用户功能转化率与漏斗留存，模型运维侧实时监测端到端时延、吞吐量以及推理准确率，甚至引入了前沿的“LLM-as-a-Judge（大模型充当裁判）”机制来自动化评估生成式 AI 任务的表现。 - **边缘端计算约束**：车端依赖轻量级 CPU，资源极度紧俏；云端算力弹性充足但需严格管控成本；移动端则遵循独立的应用运行时规范。 - **全谱系支持能力**：涵盖传统分类/推荐模型、大语言模型（LLM）、多模态模型（VLM/音频模型）及智能体（Agents）。 - **硬性评估准入门槛**：代码与模型交付必须通过自动化基准测试，指标未达标严禁推送至生产环境。 ### [11:34](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=694s) 统一工程底座：Databricks Bundles、Monorepo 与自动部署 在底层技术实现上，奔驰团队打通了一套高度工业化的 MLOps 工具链。团队采用 Databricks Asset Bundles 来统筹编排所有数据管道、模型训练任务以及离线作业的工作流，通过配置文件统一注入运行参数，无缝对接底层的 Python 代码库。在代码治理方面，团队全面推行 GitHub Monorepo（单体大代码仓库）策略，将全公司所有 AI 资产收拢在同一代码底座下，并在其上构建了高度自动化的 CI/CD 持续集成与自动化单元测试流水线。 Cancino 提到，在早年需要依赖外部团队将 Python 算法翻译为 C++ 时，为了保证两套代码在车端的执行逻辑绝对一致，团队在 CI/CD 中设计了严格的跨语言测试套件，用海量测试场景数据同时驱动 Python 和 C++ 代码并比对输出。在早期，几乎每一次新功能交付两者的输出都会出现逻辑偏差。如今，团队通过将整套代码全面转向 Python 并在车端集成 TensorFlow Lite，实现了 Python 模型的端到端直接交付车端，彻底消除了中间的人工转译环节。在云端托管与实时 API 开发方面，团队主要基于 AWS 构建了统一的抽象基类，算法工程师只需在配置中指定所需的 GPU 型号或 CPU 资源规格，一条命令行即可一键完成安全合规的模型、应用或 Agent 的自动化容器化部署，将底层复杂的网络与安全策略对算法工程师完全透明化。 ### [13:58](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=838s) 落地服务模式对比：计划批处理 vs 实时低延迟推理 在将 AI 能力向车端、移动端及内部业务系统开放时，框架清晰地划分了两种服务范式，并对它们各自的架构权衡进行了精细设计： ```mermaid flowchart TD accTitle: Batch Processing vs Real-Time Inference Architecture subgraph Batch["定时计划批处理 (Scheduled Batch)"] B1["离线计算与批量预测"] --> B2["前瞻性预测与数据时效性校验"] B2 --> B3[("Unity Catalog 存储")] B3 --> B4["API 抽象隔离层 (只读副本同步)"] end subgraph RealTime["实时低延迟推理 (Real-Time Inference)"] R1["车端/App 实时请求触发"] --> R2["常驻 GPU 算力集群 (毫秒级响应)"] R2 --> R3["弹性伸缩与共享服务限流保护"] end ``` | 维度 | 定时计划批处理 (Batch Processing) | 实时低延迟推理 (Real-Time Inference) | |:--- |:--- |:--- | | **执行机制** | 预先定时调度执行，计算资源可充分优化削峰填谷。 | 随请求动态触发，要求算力资源 720° 常驻在线。 | | **时延容忍度** | 容错率较高，单次任务失败可从容重试。 | 极度严苛，预测结果往往在 1~2 秒内就会失效。 | | **核心挑战** | **前瞻预测的时效性（Data Staleness）**：必须确保提前计算出的结果在未来实际使用时依然准确有效。 | **高昂算力与流量冲击**：GPU 算力成本高昂；共享模型服务在遭遇突发并发流量时易出现崩溃风险。 | | **数据与安全架构** | **强制 API 抽象层隔离**：严禁外部业务方直连 Unity Catalog 数据底座，统一通过受控 API 下发同步数据副本。 | **严格鉴权与过载防护**：建立多层安全防护与流量整形机制，防止单一应用的流量洪峰影响共享大模型的可用性。 | ### [16:18](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=978s) 框架演进中的核心教训：组织协同与代码重构的技术债 在回顾统一框架的建设历程时，Cancino 分享了极为深刻的组织与工程落地经验。首先是**组织层面的全面认同（Organizational Buy-in）**。推动系统整合不仅仅是技术重构，更是对长达数年工作习惯的颠覆。管理层必须深刻理解技术统一所带来的商业与研发价值，才能下定决心进行资源重构与投资。最典型的组织变革案例是：当算法团队改用 Python 结合 TensorFlow Lite 直推车端生产环境后，管理层必须妥善重新规划原先负责编写车端 C++ 代码的整个工程团队的定位与职责，这需要极具魄力的沟通与组织协同。 其次，**代码重构与数据汇聚过程充满痛苦与技术债**。Cancino 直言将十几个分散在不同 GitHub 仓库、历经多年演进的独立项目合并为一个通用基类库是一项极其枯燥、繁重且耗时漫长的工作，极度考验团队成员的耐性与执行力。目前，该框架仍在持续演进中：团队正在全面推进基础通用类（Generic Classes）的标准化建设，例如在模型评估模块上，先封装顶层的通用 Evaluate 抽象基类，再由此派生出针对音频、视觉、传感器时序等不同模态的子类（Child Classes）。这种统一的面向对象继承架构，正让全公司的数据科学家能够以模块化的方式复用既有能力，以前所未有的速度推动奔驰下一代车端 AI 体验的规模化落地。 ### [18:36](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1116s) 统一 ML 框架的未来核心规划与发展愿景 构建一个规模化的机器学习生态系统往往需要耗费大量时间，因为团队不仅需要勾勒出清晰的技术愿景、深入理解现有工作流中的各项痛点，更需要以一种极具说服力的方式向内部展示这些价值，从而“真正激发大家与你并肩协作的热情”。针对平台的下一步演进，团队确立了三大核心优先事项：推进 Agentic（智能体化）架构、完善实时推理的自动扩缩容（Auto-scaling），以及进一步深化与企业内部各业务线的集成。 在智能体化方面，团队目前已经在生产环境中部署了部分 Agent。这些 Agent 拥有调用特定内部工具、访问特定数据点、串联其他模型以及集成公司内部现有应用程序的能力。为了进一步降低使用门槛，团队正致力于将此前在整体框架中推行的“通用类（Generic Classes）”与标准化理念复现到 Agent 开发中。通过构建可复用的 Agent 模板，内部工程师能够极其便捷地部署出能够安全访问奔驰内部数据与资源的新智能体，从而大幅加快全公司的开发与部署节奏。 ```mermaid flowchart TD accTitle: Mercedes-Benz ML Framework Future Priorities subgraph S1["优先事项 1：Agentic 架构拓展"] A["标准化 Agent 模板 / 通用类"] --> B["连接内部数据 & 专用模型"] B --> C["调用内部工具与业务应用"] C --> D["加速企业级 Agent 部署"] end subgraph S2["优先事项 2：弹性扩缩容"] E["全球多地用户<br>(德国/印度/日本/韩国)"] --> F["不可预测的实时推理负载"] F --> G["智能 Auto-scaling 机制"] end subgraph S3["优先事项 3：全公司深度集成"] H["打通更多内部业务线"] --> I["量化呈现业务价值与 ROI"] I --> J["推动组织协同与文化变革"] end ``` 针对实时推理（Real-time Inference）场景，实现精准的弹性自动扩缩容至关重要但也极具挑战。梅赛德斯-奔驰的研发与业务人员遍布全球（如德国、印度、日本、韩国等地区），各分支团队在不同时段发起的操作与负载往往无法提前预知，而且各方并不会每次都向平台团队报备当前是在进行轻量测试还是上线了大规模的实际生产业务。因此，系统必须具备更强的全局感知能力与自适应自动扩缩容能力，以平稳应对难以预测的高峰与低谷流量。最后，通过不断加深与全公司业务的融合，平台将能更清晰地向管理层与各部门证明其投入产出比（ROI），进而推动更深层次的组织协同变革。演讲最后，讲者坦言时间已略微超时，并热情邀请有进一步疑问的与会者在会后私下交流。 - **Agent 模板化加速**：通过提供标准化模板与通用抽象，降低各团队构建可访问内部数据和工具的 Agent 的门槛，实现快速敏捷上线。 - **全球化负载治理**：应对来自德国、印度、日本、韩国等多国团队无法预估的实时推理请求，强化监控感知与 Auto-scaling 机制。 - **业务价值与 ROI 闭环**：通过深度集成奔驰内部更多业务应用，直观展现平台价值与投资回报率，激励更多团队参与到 AI 驱动的生态变革中。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=4s) 每一辆奔驰汽车在本质上都是一台高度复杂的边缘计算设备（Edge Device）。 — Frankie Cancino > [01:31](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=91s) 如果系统感知到车主反复在特定场景执行某种操作，模型便会自动学习这一习惯模式，并主动为车主提供智能化建议，甚至实现全自动触发。 — Frankie Cancino > [01:31](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=91s) 若车主每天下班离开公司时习惯给某人打电话，系统无需车主翻找三四级菜单或唤起语音助手，便会直接在首页高亮展示拨号建议，真正做到一键呼叫。 — Frankie Cancino > [03:06](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=186s) 机器学习团队在前期探索和算法实验时主要使用 Python，但当功能要真正部署进汽车嵌入式环境时，必须依赖另一个完全独立的工程团队用 C++ 将 Python 代码全盘重写一遍。 — Frankie Cancino > [06:40](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=400s) 这些多模态数据在未来 6 个月内随时可能变成金矿，一旦中途遗失，就再也无法挽回。 — Frankie Cancino > [09:30](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=570s) 只有当新模型的关键性能指标明确超越上一代 baseline，或者完全契合新产品的准入标准时，CI/CD 流水线才会放行部署。 — Frankie Cancino > [11:34](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=694s) 通过将整套代码全面转向 Python 并在车端集成 TensorFlow Lite，实现了 Python 模型的端到端直接交付车端，彻底消除了中间的人工转译环节。 — Frankie Cancino > [16:18](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=978s) 当算法团队改用 Python 结合 TensorFlow Lite 直推车端生产环境后，管理层必须妥善重新规划原先负责编写车端 C++ 代码的整个工程团队的定位与职责，这需要极具魄力的沟通与组织协同。 — Frankie Cancino > [16:18](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=978s) 将十几个分散在不同 GitHub 仓库、历经多年演进的独立项目合并为一个通用基类库是一项极其枯燥、繁重且耗时漫长的工作，极度考验团队成员的耐性与执行力。 — Frankie Cancino > [18:36](https://www.youtube.com/watch?v=Ok8BV5Gj128&t=1116s) 真正激发大家与你并肩协作的热情。 — Frankie Cancino ## Mentioned Frankie Cancino, Mercedes-Benz, Databricks, Unity Catalog, Databricks Asset Bundles, GitHub Monorepo, TensorFlow Lite, AWS, Jira, Python, C++, Routines, Zero Layer, Modes, ADAS, LLM-as-a-Judge ## Video description Frankie Cancino - Senior Data Scientist, Mercedes-Benz Research & Development Managing AI at Mercedes-Benz scale requires a seamless bridge between data and deployment. This session breaks down our unified ML framework: a single ecosystem orchestrating data pipelines, model training, fine-tuning, and comprehensive job tracking. Learn how we architected this platform to support the dual demands of low-latency real-time inference and high-throughput batch processing for vehicles and internal users. [Source](https://www.youtube.com/watch?v=Ok8BV5Gj128)

Source quotes

[0:04] Frankie Cancino: 每一辆奔驰汽车在本质上都是一台高度复杂的边缘计算设备（Edge Device）。 [1:31] Frankie Cancino: 如果系统感知到车主反复在特定场景执行某种操作，模型便会自动学习这一习惯模式，并主动为车主提供智能化建议，甚至实现全自动触发。 [1:31] Frankie Cancino: 若车主每天下班离开公司时习惯给某人打电话，系统无需车主翻找三四级菜单或唤起语音助手，便会直接在首页高亮展示拨号建议，真正做到一键呼叫。 [3:06] Frankie Cancino: 机器学习团队在前期探索和算法实验时主要使用 Python，但当功能要真正部署进汽车嵌入式环境时，必须依赖另一个完全独立的工程团队用 C++ 将 Python 代码全盘重写一遍。 [6:40] Frankie Cancino: 这些多模态数据在未来 6 个月内随时可能变成金矿，一旦中途遗失，就再也无法挽回。 [9:30] Frankie Cancino: 只有当新模型的关键性能指标明确超越上一代 baseline，或者完全契合新产品的准入标准时，CI/CD 流水线才会放行部署。 [11:34] Frankie Cancino: 通过将整套代码全面转向 Python 并在车端集成 TensorFlow Lite，实现了 Python 模型的端到端直接交付车端，彻底消除了中间的人工转译环节。 [16:18] Frankie Cancino: 当算法团队改用 Python 结合 TensorFlow Lite 直推车端生产环境后，管理层必须妥善重新规划原先负责编写车端 C++ 代码的整个工程团队的定位与职责，这需要极具魄力的沟通与组织协同。 [16:18] Frankie Cancino: 将十几个分散在不同 GitHub 仓库、历经多年演进的独立项目合并为一个通用基类库是一项极其枯燥、繁重且耗时漫长的工作，极度考验团队成员的耐性与执行力。 [18:36] Frankie Cancino: 真正激发大家与你并肩协作的热情。

Source quotes

[0:04] Frankie Cancino: In the eyes of the machine learning team, every single one of those vehicles acts as an independent edge device emitting data and running intelligent software. [1:39] Frankie Cancino: Instead of forcing drivers to navigate through submenus or rely solely on voice commands, Zero Layer dynamically predicts and surfaces the single next action the driver wants on the primary screen. [3:06] Frankie Cancino: The most severe bottleneck stemmed from the language barrier between research and production. The ML team developed experimental models in Python, but in-vehicle deployment required compiling to embedded C++. [3:06] Frankie Cancino: How do I do my job? [6:43] Frankie Cancino: Data is the foundational backbone of the unified framework. [6:43] Frankie Cancino: Preserving raw and intermediate data in a shared catalog ensures historical telemetry becomes a reusable 'gold mine' for future initiatives. [9:30] Frankie Cancino: The edge vehicle environment is strictly constrained: models execute primarily on in-cabin CPUs without current access to onboard GPU acceleration, requiring lightweight, highly optimized binaries. [11:34] Frankie Cancino: A major engineering milestone was deprecating the error-prone manual Python-to-C++ translation handoff. [14:04] Frankie Cancino: Real-time inference powers time-sensitive vehicle and mobile interactions where an inference result loses all utility if not delivered within one to two seconds. [18:36] Frankie Cancino: inspires people to want to work on this with you. [19:25] Frankie Cancino: Proving sustained value hinges on embedding this unified framework into more everyday business processes, which establishes a clear return on investment (ROI) and encourages positive organizational and cultural change around AI adoption.

Original transcript

[0:04] What we're going to talk about today is how my  team, the machine learning team at Mercedes, [0:09] builds our AI products, features, everything.  Um it's been a journey. Uh we're going to [0:15] talk about the scale of Mercedes-Benz, the  vehicles, and everything we have to deal with. [0:20] Um this unified framework my team came up with  for the company. It's not just like for our team, [0:25] but it helps deploy things for the  company. Uh how we serve this out with, [0:31] you know, real-time versus scheduled kind of  batch workflows, and then some lessons we learned, [0:36] what we're looking forward to in the future,  and hopefully we'll have questions at the end. [0:42] I only have 20 minutes, less than 20 minutes,  so if you're interested or want to talk more, [0:46] please please find me. We can have an open  discussion or I'll answer any questions I can. [0:52] So, to get started about the scale, um so last  year, Mercedes in the US alone sold a little [1:00] over 300,000 cars. Uh globally, a little over 2  million. Uh think of those each as its own device. [1:09] So, they're basically like edge devices in our  eyes. Uh and then we also have this mobile app. Uh [1:15] I don't know, like 2 million downloads  as of last year. There's a lot more now. [1:19] If you go to a dealership, you'll notice that  they're trying to get you to download it, [1:22] create a profile, and all that. That's, you know,  coming from us and uh Germany a bit, so that way [1:27] we can get more and more people kind of on board  to all of these cool features that we're building. [1:31] And again, think of those as edge devices for  everything we do. And then from the product side, [1:39] we have so many. These are just a subset. So,  we have this app called routines that my team [1:43] built out and deployed it to production. It's in  newer generation cars. The idea is that if you do [1:50] something over and over and over again, uh we can  learn that and provide it to you as a suggestion. [1:54] You can just have it automatically do these  things that you typically do. Um we also have [1:59] a thing called zero layer. If you're familiar with  Mercedes cars, that's been around for a few years. [2:03] Again, my team deployed that. It's essentially  on the home screen, it will try to recommend the [2:09] next action or the next thing you want. So that  way it's a one click of a button. So for example, [2:13] if you typically call a person at this time  or when you leave work or whatever may be, [2:18] instead of going through three, four, five  clicks to go find that person, make the call, [2:23] um or you don't want to use the voice assistant,  it just shows up. You click one button on the [2:26] screen and then it just does the call. Um I'm  not going to go through all of these, but yeah, [2:30] we do stuff for navigation. Of course,  we have Thomas driving and ADAS systems, [2:35] um comfort functions you might have we have  massages in our cars for some of the packages, [2:41] um entertainment, the voice assistant like I  mentioned, uh modes is a new thing coming out, [2:46] and then we also like I said have the mobile  app, and there's a lot more features. So [2:52] multiply this by all those devices I just  mentioned and we our team deals with all kinds of [2:59] issues, uh machine learning type  of questions, data, etc. etc. So [3:06] when I first started though, we definitely worked  in silos. And it wasn't working out super well. [3:12] Uh some of the the reasons it  wasn't working out are listed here. [3:17] One of the biggest ones we're always reinventing  what someone else did or someone on on your same [3:23] team is working on this project with some  other team, maybe even not even in the US, [3:27] and they're recreating everything. Um there's  miscommunication constantly. For example, uh [3:34] historically our team would write kind of more  experimental machine learning code in like Python, [3:39] and then eventually when it gets deployed  to production, it was in C++. So we had an [3:43] entire different team that would re-implement  everything we did in C++ to put it in the car. [3:48] Uh it it's just a lot of back and forth, and every  project meant someone had to onboard, you had to [3:55] figure out where's the code going to be, what are  the systems they use, what's the interaction like, [4:01] what's the way of working, you know, Agile,  are you using Jira, what everything. Um [4:07] The other thing too I want to mention  here is after everything got deployed, [4:13] our data scientists who look at usage and  try to understand what our customers are [4:18] doing and what's working what's not, if they  went to each of these projects and asked them, [4:23] "How do I do my job?" it was completely different  between the projects, which made that part also [4:29] very difficult. Um and if you add all  this together, the last point there, [4:36] and super long development cycle. Just getting  something out to production took forever. [4:40] We had to rewrite code like I mentioned with  a completely different team. They had to sit [4:44] with our team, and then they had to understand  everything, and then the data scientists were [4:47] like, "We don't It's going to take forever to  also monitor and understand our customers." [4:52] So, we want to essentially pivot to a unified  framework, obviously the title of the talk, [5:00] and we had to create the vision for  leadership and for everyone involved. [5:06] So, the main thing was reusability. If we're  making all of these changes across the company and [5:14] things are probably pretty similar, especially  dealing with machine learning and data, [5:18] we want to be able to put that code somewhere  and just reuse it and it build on top of it. [5:23] The other thing is knowledge sharing. There  is so much experimentation happening for [5:28] each project about what machine learning models  are working, what features are working, what um [5:33] documentation there was around it, and it  was basically that was also siloed off with [5:38] each project. You wanted to go learn about what  someone did in this project, well guess what, [5:42] you had to go get requests and permissions to  go view all the documentation, and multiply [5:47] that by the number of projects, and you probably  don't know what's happening across the company, [5:52] even though you're on the same team as some of  the engineers that were working on that. So, [5:57] try to share as much as possible. All of that  would lead to faster development. That was [6:02] one of my biggest headaches, especially joining.  I was not coming from automotive, but it was just [6:06] super slow to deploy anything. And then, able  to develop faster meant more features, more [6:14] you know, products, whatever you want to call  them, and that increased our reach as a team [6:18] and as a department. Which meant that we can  then help out Mercedes-Benz in different ways, [6:25] and improve our systems that were already running,  instead of just, you know, recreating everything [6:29] over and over again. So, that is the vision. I'm  going to go through the framework kind of bit by [6:35] bit, and then piece it all together for you. Uh  so, the first part was huge. This took a Uh data. [6:43] Data was the back Going, like I mentioned,  think of all the vehicles in the mobile app [6:47] and everything as like edge devices emitting data  constantly. Well, not only that, we have a ton of [6:54] different data sources. We had the modalities,  especially now with like GenAI is getting huge. [6:59] So, there might be audio, there might be vision,  there might be sensor data, there might be um [7:04] you know, the mobile app type of data coming, like  clicks and and the pages you're you're viewing. [7:08] Uh so, that was challenging to wrangle all  that together. And then, what we did is we [7:15] did this medallion architecture. I won't get into  that too much, because that's a Databricks thing. [7:19] We use Databricks heavily. And essentially, the  point of that is there's this bronze, silver, [7:24] and gold flow. Uh all your raw data comes in at  the bronze level, and then you kind of clean it up [7:31] for, you know, the necessary projects, whatever it  may be, as you go through, and that's the silver, [7:35] and then the gold. So, here you'll see we get, you  know, all kinds of data coming in, going to the [7:41] bronze tables for ML and AI, we split it out for  features and things like you can think of vision [7:48] embeddings or, you know, audio type of stuff or  whatever may be. And then we have data science and [7:53] analytics separately because what they care about  is different than what the AI models cared about. [7:58] So, we're all reading from the same raw  data, but the point of view are different. [8:06] And then in fine-tuning, of course, we have  traditional ML, uh recommendation systems, [8:12] etc. So, we have training on that data. We  have, of course, GenAI fine-tuning now because [8:17] LLMs, vision language models, um audio language  models, we have all these different large models [8:22] now, so we do fine-tuning based off that data.  And I'll make a point about this later, too, but [8:29] evaluation is very key. Like, we take evaluation  very seriously because that determines whether or [8:33] not we're going to push something to production.  And then one key point is that we want to share [8:40] all that data. So, for example, let's say one  product um is using image data. That image data [8:47] could be useful for another project that may not  have even existed or started yet, but let's say 6 [8:53] months in the future, which I'm I'm saying this  from experience right now, I'm dealing with it, [8:57] uh that data is a gold mine. And if we don't  collect it or clean it up or save it along the way [9:07] and no one does, you end up with no data  along the way. So, now we made it a point to, [9:12] hey, if this data might be useful in the future  for a different product, we want to save it and [9:17] put it in, I'll mention it later, Unity Catalog,  which is, again, a Databricks thing, but it's [9:22] basically our data lake. Um and then we do,  you know, massaging, cleaning, etc. after that. [9:30] Model deployment, not trivial. Um  this one, not as difficult because [9:34] there wasn't much there to begin with, but  yeah, it was a journey still nonetheless. [9:39] Uh We have a variety of deployment options, so we  have the vehicle, which is like an edge device. [9:44] The constraints are different there. They're you  lightweight, you have a CPU. I don't think we even [9:50] have access to a GPU right now in the  vehicle. Um versus cloud. The constraints are [9:57] not really there. Like we can use as much as we  want. If needed, cost obviously is a concern. [10:03] And then we have the mobile app. So deployment to  that as well. It has there's completely different [10:06] constraints for that. And right now we're able  to do deployment of LLMs, visual language models. [10:14] I should have added to maybe some audio language  models, traditional applications, traditional [10:19] machine learning models, and then also agents.  So this is the realm of our deployment strategy. [10:26] And then we monitor everything along the  way. So like I mentioned, we split things [10:30] out for data science and analytics. So we look at  customer feature usage, funnel analysis, you know, [10:35] just what's working, what's not. The AI models  obviously, we look at the latency, the throughput, [10:41] how well they're performing. I know before  the talk or before this talk someone [10:46] the presenters talking about LLMs as judges.  We also evaluate those and then also use those [10:51] judges to help evaluate the other stuff. Um  Then we monitor the internal workflows as well. [10:57] And then this would be the  workflow stitched together. So you [11:05] pipes into model training and fine-tuning, but  you'll also notice that nothing gets deployed [11:09] without evaluation. We have criteria, whether  it's a product from like product or from the [11:15] machine learning engineers saying that hey, this  model is, you know, X% doing this. We're not going [11:21] to deploy unless it beats that or if it matches  type of new product requirement. And then again, [11:28] data insights and monitoring on the bottom.  Every step of the way, we monitor everything. [11:34] And then, just to get into it a little  bit, uh again, we use Databricks heavily. [11:40] Um we use Databricks bundles to  orchestrate all the workflows, [11:43] all the running of the jobs, all the tasks. Um we  pass in configurations and parameters this way. Uh [11:49] goes right into our Python code. And  we also use a GitHub monorepo. So, that [11:56] evaluation or not the evaluation, uh the  unit test framework I showed before, the [12:05] Is this still working? Okay. Uh it's one massive  monorepo where we have CICD, automated testing. [12:12] Um like I mentioned before, too, we used to, not  anymore thankfully, uh have a team translate our [12:18] Python code to C++ code. Well, guess what? We had  to make sure that it did what it was supposed to. [12:24] So, we would run scenarios and test data through  both the Python and C++ code, and we had automated [12:30] test for that to see, did they match? And I'm  sure it's not a shock that in the beginning, the [12:36] first time for each of these products, it never  matched. It was always different somehow. So, [12:41] we just automated everything um through  GitHub actions and through our CICD process. [12:47] Uh and then, I hinted at earlier, UniKalg is our  data lake. Uh we store everything there. It's [12:52] reusable by everyone who has access to it from  our team, including some other teams. You can [12:57] then share some of that data to other external  teams, or if you're working with someone else, [13:02] they can go look at that as well. Uh for hosting,  we use AWS primarily for the real-time, uh [13:10] you know, API development. Um we can use GPUs  or CPUs. This is just a screenshot of a generic [13:16] class that we made, and it basically takes in a  configuration. Uh I don't know if I can point at [13:21] it, but basically, there's just like a little  if statement there where in the configuration, [13:24] if If a GPU, it can just deploy it. You just  pass it what kind of GPU you want and it'll [13:28] deploy as many as you want for that. And then  if it doesn't need it, it just goes to CPU, [13:32] but in general, it's one command line command  and it will just deploy everything you need. Um [13:39] Uh it says there, yeah, models, apps, or agents,  we can deploy any of those. Um it's all, you know, [13:46] secure. We have all these layers of abstraction  security, which is nice. Um it's abstracted away [13:51] from most of the machine learning engineers  and it's yeah, pretty stable, which is good. [13:58] And then how we serve this out to customers or,  you know, internal teams. Um so there's batch [14:04] processing, which much easier. It's scheduled,  you can optimize it. You know exactly what you're [14:10] getting out of it and if you're planning ahead  and something goes wrong, you just kind of retry. [14:16] The thing about that, a lot of these predictions  needs to be forward-looking. So for example, [14:19] if you do a batch, let's say predictions, most  likely they're going to be used in the future, [14:25] which means you need to consider whether or not  your data will be stale. [snorts] Is prediction [14:31] actually going to be useful, let's  say, tomorrow or even further out? Um [14:37] Also, we added a layer of abstraction for all  of our batch processing uh model predictions or [14:43] generations because you don't want people directly  accessing your data. So if we put this, let's say, [14:49] in Databricks and Unity Catalog somewhere, we  don't want to give people access to directly call [14:53] it, so we would add layers of abstraction such  as like APIs or whatever may be, where they would [14:58] request it and they would get a copy of the data  and they would always be synced up and, you know, [15:04] refreshed accordingly. The other way, real-time  inference, um this is on the fly. You don't know, [15:11] necessarily, how many requests you're going  to get, but you have to serve them right away. [15:15] So all the resources need to be up and running at  all times and then these are time-sensitive. So [15:22] a prediction or generation, it may not  be useful even 1 or 2 seconds from now. [15:32] Which means this compute is very necessary and  hence GPU. Uh GPU costs went up a lot in the [15:38] last couple years for obvious reasons. Um but  compute and speed and latency here are vital. [15:45] Uh and then one thing here it says a shared single  service, but if you think about it, there are like [15:50] security layers layers and whatnot, but people can  access your service directly essentially, and so [15:57] too many people might crash it. So, you have to be  aware of that and you have to scale accordingly, [16:01] and it's kind of hard to when you can't predict  it. And if you're sharing or reusing models um [16:08] like we do, you have to just kind of hope that  people don't all of a sudden have a surge of [16:13] requests and then might break it for other  people, too, that are using the same model. [16:18] So, a little about the road ahead, but I'm going  to get to some of the lessons that we learned. Uh [16:25] it requires organizational buy-in. Um not everyone  may be on board to bring everything together and [16:33] work a certain way that might be different  than maybe years of doing the previous way. [16:38] And also management leadership need to  understand the benefit of this if in order to [16:44] invest, let's say, even in the resources, the  technology, the like I said, we we stopped using [16:52] a team that wrote C++ code and then did everything  in Python and made it more production ready using [16:57] um TensorFlow Lite. So, our code now is directly  going to production. So, now we had to have frank [17:03] discussions with the management about what will  that other team do now? Right? So, it changes [17:09] things up and so you need organizational  buy-in to understand what's the benefit [17:13] of unifying some of your systems, and a team  together, and reusing everything you're doing. [17:19] Uh also, the second one, personal experience.  Refactoring code, consolidating data is a pain. [17:24] It's super tedious, and there's always tech debt.  If you're taking, let's say, 10 different GitHub [17:29] repositories from 10 different projects, trying to  merge them together, and make generic classes out [17:34] of all of them, it it takes forever, and it's  not fun. Uh so, you need to have like a team [17:40] uh or coworkers that are willing to do this, and  are not going to complain the whole time. Luckily, [17:46] at my None of my coworkers really complain to too  much. Um but it definitely took a lot of time. And [17:53] unifying requires patience. It really does. I Even  today, we're still building out these classes. So, [17:59] for example, evaluation uh piece of the models,  we're making generic classes for evaluation [18:06] period, and then we have like subclasses or  child classes for audio modality, vision, [18:12] uh sensor, right? Like, we we want to make  every single component of that framework [18:18] have a generic class where people can just build  from it using child classes, and work faster, [18:25] and reuse what other people have already built.  And So far, it's going well, but we're still not [18:30] done. We're still building all of that out. And  particularly vision, that's also another one that [18:36] takes time, because you have to build out  the vision, and understand your pain points, [18:40] and present it in a way that it basically  inspires people to want to work on this with you. [18:46] So, right now, our top priorities are you  heard about agentic, I'm sure. Uh we want [18:52] to make things more agentic. We have agents  deployed. They have access to certain tools, [18:57] and to certain data points, and to other models,  and um [snorts] applications within the company. [19:03] But we want to make it easier. So, like I just  mentioned, we're making generic classes for the [19:09] entire framework. We want to do the same thing for  the agent approach. So, we want to make templates, [19:14] make it easy to deploy agents that can access the  data, the you know, anything inside the company, [19:20] especially. So, that way we can work faster and  deploy faster. Then, there's auto scaling. So, [19:26] for the real-time inference especially, it's  like I said, it's hard because we don't know [19:32] what everyone in the company, whether they're in  Germany, they're in India, they're in Japan or [19:36] Korea, where they're from, what they're going to  do at all times. And they don't always communicate [19:40] to us like what they're doing or how much  they're going to do it or if they're just testing [19:44] something out or if this is a real thing. So, we  want to be able to auto scale and and be aware of [19:49] what's going on more so with our systems. And then  the last part is integrate more into the company. [19:55] Um once we can integrate [snorts] more and more,  it'll show the value and then we will be able to [20:01] um have an ROI kind of attached to everything  and you know, hopefully inspire more, you know, [20:08] people change and they can help Yeah. That's all I  have. Thank you. And if anyone has any questions, [20:15] feel free to catch me after. I think  I'm already over time, but thanks again.