Ford, TikTok AI Leads Shift Data Prep Upstream to Cut Deploy Times

福特与TikTok的AI负责人将数据准备前移以缩短部署周期 将数据清洗从孤立的Notebook转移到上游工程流水线，可将模型交付周期缩短高达70%，并消除算法调优无法解决的根本性缺陷。 在Chetana Katpally的主持下，来自Caterpillar、Navy Federal Credit Union、Ford和TikTok的跨行业领袖阐述了为何高质量的数据准备是企业AI成功的核心驱动力，而不仅是枯燥的基础管道工作。嘉宾们展示了将数据画像与转换向工程上游转移至集中式流水线和特征库中，如何将投产周期压缩高达70%并防止灾难性的模型故障。他们深入探讨了现代架构模式，包括单元格级访问治理、LLM辅助元数据标记、数据网格（Data Mesh）原则以及合成数据生成。讨论最后强调，应在上游建立技术与业务协同的数据管理机制，以避免团队在下游陷入对低质量数据无休止“打地鼠”式的被动应对中。 高质量的数据准备是一项重要的运营倍增器，能够将机器学习的投产耗时压缩60%至70%，且在提升模型精度方面的效果比单纯的超参数调优高出2至5倍。 (2:07) 现代工程架构可以利用Apache Airflow等工具使60%到70%的数据准备生命周期实现自动化，同时保留人机协同（human-in-the-loop）验证检查点，并追踪清洗前后的数据漂移。 (7:46) 数据治理正从粗粒度权限向动态的单元格级访问控制演进，并由数据接入阶段基于机器学习的自动PII分类和端到端数据血缘提供支撑。 (11:21) 组织必须调整管理层的激励机制，以奖励前期的数据清洗和异常值剔除，而不是仅仅激励基于未加工数据快速交付模型。 (20:17) 将探索性画像、异常值检查和标准化工作从孤立的Jupyter Notebook前移至集中式工程流水线和特征库中，可将建模周转效率提升3至10倍。 (21:54)

Enterprise AI practitioners argue that model performance and production velocity depend far more on rigorous data preparation, automated upstream governance, and domain stewardship than on algorithmic sophistication. Citing McKinsey research, moderator Chetana Katpally notes that clean data accelerates decision-making fivefold and curbs model errors, while cross-industry panelists emphasize that unrefined inputs systematically compromise downstream machine learning. Elevating Model Precision Over Algorithmic Complexity (2:07) Across industrial anomaly detection and consumer platforms, panelists argue data hygiene yields greater performance returns than hyperparameter tuning. Santosh Vasudevan notes that clean preparation directly eliminates false positives in heavy machinery diagnostics, saving operational time for downstream analysts. Tajinder Singh Chahal warns that feeding unrefined data into models is like pumping crude fuel into jet engines, observing that resolving foundational data defects can improve accuracy by 2x to 5x. Similarly, Madhura Raju points out that when TikTok A/B experiments severely underperform projected metric lift, annotation and data representation flaws—rather than model architectures—are almost universally the root cause. To justify the 70% to 80% of project time consumed by data preparation, teams track empirical improvements in precision and recall between raw baseline models and cleaned iterations. Rahul Zende adds that establishing curated 'gold tier' datasets compresses end-to-end data science production cycles by 60% to 70%. Replacing Local Notebooks with Standardized Pipelines (21:54) Shifting exploratory profiling and cleaning out of siloed Jupyter notebooks into standardized engineering pipelines optimizes turnaround times by 3x to 10x. Isolated notebook experimentation duplicates computational effort and introduces conflicting schemas, such as mismatched datetime formats or customer identifier types across business units. Standardized pipelines counter this by executing automated statistical profiling and outlier filters at ingestion before raw data reaches downstream workflows. Centralized feature stores further accelerate iteration cycles by providing pre-aggregated, verified variables. Zende explains that while simple projects require two to three iterations, complex enterprise models require five to seven; shared feature stores prevent teams from re-engineering transformations during pivots and ensure models in regulated banking environments avoid non-compliant or predatory variables. Enforcing Cellular Governance and Ingestion Safeguards (11:21) In regulated environments like consumer banking, pipelines must satisfy strict compliance Service Level Agreements (SLAs) to guarantee auditability and responsive grievance resolution. Chahal notes organizations are transitioning from broad table-level permissions to dynamic cellular controls that combine row-level security and column masking. Machine learning models scan incoming records at landing points to automatically classify and mask Personally Identifiable Information (PII) like Social Security numbers and birthdates. Lineage platforms such as Collibra, Alation, and Dataplex trace provenance across tables to maintain consumer trust. Furthermore, early collaboration between data engineers and data scientists during vendor ingestion prevents the inadvertent loss of predictive features. When engineers discard vendor fields deemed redundant for basic reporting, high-value machine learning signals are permanently lost. Accelerating Scale with LLMs and Synthetic Data (29:45) Emerging workflows increasingly automate manual data engineering. Zende and Chahal report that large language models are now deployed to generate automated metadata tags, propose baseline cleaning rules, and surface cross-table correlations across thousands of tables in cloud environments like GCP. Although engineers retain oversight through human-in-the-loop review, LLM-assisted onboarding sharply reduces the manual effort needed to prepare new data sources. Simultaneously, teams are addressing projected data constraints. Raju cites industry discussions and research in Nature projecting that public human-generated internet training data may be exhausted by 2028. To counter data scarcity and resolve representation gaps, enterprises are adopting generative simulation to create high-fidelity synthetic data, expanding a strategy originally popularized in autonomous vehicle development. Adopting Data Mesh and Paired Stewardship Models (40:16) To avoid the reactive cycle of constantly fixing corrupted data downstream, panelists advocate for Data Mesh architectures that treat data as a domain-owned product. Chahal underscores the creation of 'certified data products' backed by schema guarantees and CI/CD validation at entry—even within real-time streaming telemetry—ensuring downstream trust without requiring unnecessary physical data relocation. Addressing governance overhead, the panel cautions against placing data stewardship on isolated individuals, which induces bottlenecking and burnout. Instead, they recommend a paired stewardship model combining a technical steward (focused on pipeline and schema health) with a business steward (focused on domain context). This paired structure enforces upstream quality standards, such as eliminating null-value propagation, directly at the source. While upstream data preparation and centralized governance significantly improve model outcomes, enterprise incentives frequently work against them by rewarding rapid feature deployment over data hygiene. Furthermore, panelists emphasize that even meticulously prepared models face inevitable decay from shifting real-world conditions, making continuous data observability and drift monitoring an ongoing operational requirement.

在企业人工智能实践中，高质量的数据准备不仅是缩短研发周期的核心，也是决定下游模型准确率与业务投资回报率的决定性前提。跨行业专家指出，企业必须将数据清洗从孤立的下游分析脚本转向工程化、标准化的上游治理体系，并通过技术与业务双重管理机制保障数据资产的长期可靠性。 数据清洗的业务回报与杠杆效应 (2:07) 虽然数据准备通常占据数据科学项目70%至80%的时间，但专家们强调前期清洗工作具有极高的杠杆效应。Caterpillar首席数据科学家Santosh Vasudevan指出，在异常检测系统中，整洁的数据清洗对消除误报和提升精度的贡献往往超过机器学习算法本身，能够直接缩短质量控制分析师排查根因的时间。Navy Federal Credit Union的Rahul Zende则表示，建立经过严格审查的“黄金级”建模就绪数据集，可使模型投入生产的周期缩短60%至70%。 福特汽车的Tajinder Singh Chahal与TikTok的Madhura Raju分别将低质数据比作直接注入喷气发动机的未经提炼的原油，以及用腐烂番茄做出的意面。Madhura提到，当线上A/B测试或生产模型未能达到预期的指标提升（仅产生0.1%的预计效果）时，排查结果几乎都指向底层数据质量或代表性缺陷，而非算法错误。Tajinder进一步补充，许多团队盲目调整超参数或复杂的模型架构，而实际上解决基础数据质量问题即可将准确率提升2倍到5倍。 流水线自动化、动态合规与访问控制 (7:46) 在实施层面，流水线编排和自动化程度直接影响交付效率与合规表现。Caterpillar采用NumPy和pandas进行特征工程，借助Apache Airflow进行工作流编排，实现准备生命周期60%到70%的自动化，同时利用MLflow和Weights & Biases追踪清洗前后的数据漂移。在金融强监管背景下，Navy Federal Credit Union面临严格的客户申诉服务水平协议（SLA），要求端到端管道高度整合，避免跨系统数据出入带来的延迟与合规风险。 面对GDPR和CCPA等隐私监管，传统粗粒度的表级权限已向基于行级安全和列掩蔽的细粒度“单元格级”动态访问控制演进。福特汽车等企业在数据接入点部署机器学习模型，实时自动扫描并标记社会安全码、信用卡号和出生日期等个人可身份识别信息（PII）。配合Collibra、Alation及Google Cloud Dataplex等血缘追踪工具，企业能够在整个生命周期内记录审计日志，防范非授权查询与数据缺陷。 数据准备工程化与特征库沉淀 (21:54) 传统的单兵作战模式通常让数据科学家在隔离的本地Jupyter Notebook中进行探索性分析与离群值处理，不仅导致大量重复劳动和算力浪费，还拖慢了交付速度。将数据分析与特征提取上移至标准化的数据工程流水线，可以在原始记录触发核心作业前完成标准差偏离检测和极值过滤，推动端到端建模速度实现3倍到10倍的提升。 统一架构能够解决不同业务线在数据类型与时间戳格式上的冲突（例如不同团队将客户ID分别存为整型与字符串）。Rahul主张构建集中式特征库，预先聚合关键预测变量，这不仅让算法工程师免去从头构建转换逻辑的负担，加速模型迭代，更能在金融业务中通过清晰的特征血缘防范使用涉嫌掠夺性或违规的预测特征。此外，在湖仓一体架构中，若数据工程人员在引入外部供应商数据时缺乏算法团队介入，容易主观剔除部分统计特征，导致关键预测信号永久丢失。 大模型辅助工程与合成数据探索 (29:45) 新兴技术正在重塑底层数据准备模式。Rahul指出，大型语言模型（LLM）具备语境理解能力，正在将工程师从繁重的元数据打标、底层转换逻辑编写等枯燥任务中解放出来，由人工仅负责复核验证。福特汽车的Tajinder补充道，借助Google Cloud Platform等云生态工具，LLM能够跨越销售、车辆和客户等数千张业务表展开全域关联分析，捕捉人类分析师无法穷尽的跨域潜在关联。 在数据架构层面，企业正逐步采纳去中心化的数据网格（Data Mesh），将治理责任下放至深谙业务背景的领域团队，借助逻辑集成层实现按需访问。针对未来数据要素供给，TikTok的Madhura Raju引述行业讨论与《自然》（Nature）期刊的研究指出，互联网上人类生成的训练数据可能在2028年前后耗尽，企业必须借助生成式AI和高保真仿真技术合成高质数据集，以填补特殊场景下的训练数据缺口。 上游协同治理与成对数据管理机制 (40:16) 从概念验证（POC）过渡到企业级生产，最常见的痛点往往是组织协作脱节与指标体系缺失。Santosh指出，跨团队标准缺失会在规模化推广时放大细微偏差，导致巨大的返工代价；Madhura强调，团队必须建立贯穿算法层、生产环境与业务经营的多层次指标体系作为量化评估门槛。此外，Tajinder强调必须构建“认证数据产品”，即便在实时车辆遥测流等高吞吐原始接入层，只要具备严格的模式（Schema）保障，即可确立数据可信度。 针对观众提出下游无休止修补脏数据的问题，专家一致认为数据治理必须溯源至产生源头。单纯依靠某一位数据管理员容易造成管理瓶颈并引发职业倦怠。企业应当推广由“技术管理员”与“业务管理员”搭档的成对管理模式：技术人员保障管道稳健与模式完整，业务专家校准数据内涵与应用价值，以此在上游入口彻底消除过度空值等基础缺陷。 该讨论表明，高质量的数据准备依赖于上游工程标准化、自动化治理及跨职能的成对管理机制。然而，在互联网公开人类数据预计将在2028年前后面临枯竭的行业背景下，如何通过生成式AI可靠地合成边缘场景训练数据，并在去中心化数据网格中平衡各领域的执行成本，仍是企业在规模化应用中亟待解决的挑战。

## Background 该内容由行业组织 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台上。视频记录了一场题为“掌握数据准备：打造干净、可用数据的策略”的专家研讨会，来自梅西百货（Macy's）、卡特彼勒（Caterpillar）、Navy Federal Credit Union、福特汽车（Ford Motor Company）和 TikTok 的多位数据与技术负责人共同探讨了 AI/ML 模型开发中数据清洗、转换、治理以及自动化管道构建的最佳实践与落地挑战。 ## TL;DR 研讨会深入剖析了数据准备在 AI/ML 项目中占据 70% 到 80% 工作量的本质原因，指出高质量的底层数据是模型性能倍增的核心杠杆。各位跨行业专家分享了如何将数据清洗与特征工程从单机 Notebook 孤岛前置至自动化工程流水线与特征库（Feature Store），并探讨了大语言模型（LLM）在自动化元数据打标及异常检测中的革新作用。讨论还涵盖了强监管行业中的细粒度数据治理、Data Mesh 架构下的领域所有权、合成数据趋势，以及通过构建层级化指标体系与双轨数据管治（Stewardship）从源头上打破数据清洗的“打地鼠”循环。 ## Atmosphere & style 整场研讨会充满务实、严谨且极具行业穿透力的技术交流氛围，五位来自不同领域的资深专家毫无保留地分享一线实战经验与踩坑教训。交流中穿插了诸如“未经提炼的原油灌进喷气发动机”和“厨房烂西红柿做意面”等生动贴切的比喻，既有针对底层架构与算法指标的深度推演，也有对组织文化、跨部门博弈与合规治理的深刻洞察。现场节奏明快而富有共鸣，尤其在观众问答环节直击数据管治的痛点，呈现出一种兼具战略视野与实战落地价值的专业思想碰撞感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=4s) 数据准备通常占据数据科学项目 70% 至 80% 的工作量，麦肯锡研究表明善用干净数据的企业决策效率可提升约 5 倍，且高质量数据往往比算法调参对模型性能的提振更具决定性。 - [02:07](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=127s) 企业沉淀‘黄金标准’（Gold Standard）分层数据资产可将模型从开发到上线的时间缩短 60% 至 70%，并通过大幅减少误报降低业务分析师的排障人力成本。 - [02:07](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=127s) 未经治理的脏数据输入模型就像未提炼的原油直接灌进喷气发动机；TikTok 等团队实践表明，线上 A/B 测试中模型泛化不及预期的根本原因几乎无一例外指向底层数据质量缺陷。 - [10:25](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=625s) 在金融银行业等强监管场景下，数据管道技术选型必须优先满足低延迟、高容错、合规 SLA 以及端到端审计要求，尽量减少数据在跨系统出入（Data Ingress/Egress）时的管理开销。 - [11:21](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=681s) 现代数据治理正在从粗粒度的表级权限向结合行级安全与列掩码的‘单元格级别’（Cellular Level Access）访问控制演进，并利用 AI/ML 实时识别敏感 PII 数据与打标。 - [15:23](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=923s) 在敏捷交付压力下，团队应通过模块化代码复用、LLM 辅助生成样板代码以及‘足够好即可’的渐进迭代策略来平衡数据准备深度与上线速度。 - [17:27](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1047s) 统一特征库（Feature Store）通过提前聚合统计指标与预测因子，使数据科学家能够即取即用，不仅缩短基线模型构建周期，也便于在合规限制下迅速剔除偏见特征并重新迭代。 - [22:10](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1330s) 数据特征工程正在经历从单机 Notebook 孤岛向工程流水线（Pipeline）前置的范式转移，在入库前自动化校验离群点与标准差可带来 3 到 10 倍的效率提升。 - [25:58](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1558s) 在跨异构供应商系统的湖仓架构中，数据科学团队必须早期介入数据摄取，防止具有预测价值的关键字段在入库时被工程师误当作无用冗余剔除。 - [28:14](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1694s) 说服管理层投资数据清洗的最佳方式是用硬核对比实验说话（如对比清洗前后 v0 与 v1 的精确率与召回率），同时必须建设可观测性体系以应对数据漂移与模型衰退。 - [29:45](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1785s) LLM 正在通过自动元数据打标（Metadata Tagging）与跨业务域异常特征关联发现重塑数据工程，在人机协同（Human-in-the-loop）下免除重复繁琐的手动标注。 - [33:35](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2015s) 数据网格（Data Mesh）倡导将数据作为产品（Data as a Product）下放至具体业务领域，打破集中式壁垒；而合成数据（Synthetic Data）正成为缓解高质量人类数据枯竭的核心手段。 - [36:03](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2163s) 将数据科学项目从 PoC 试点推向企业级规模化时，最容易忽视的瓶颈是跨团队数据规范不一致（人）、数据孤岛以及生产层与模型层的层级化指标体系（Hierarchy of Metrics）缺失。 - [40:16](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2416s) 要终结下游工程反复修补数据的‘打地鼠’循环，必须在数据流水线最上游建立由‘技术管治者与业务管治者’结对的双轨数据管治（Data Stewardship）机制。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=4s) 研讨会开场与跨行业嘉宾介绍 研讨会主持人、来自梅西百货（Macy's）的机器学习工程高级经理 Chetana Katpally 宣布会议正式开始。本次研讨会的核心主题是“掌握数据准备：打造干净、可用数据的策略”（Mastering Data Preparation: Strategies for Clean, Usable Data）。Chetana 在开场时指出，充分准备的高质量数据能够显著加速企业决策、降低模型错误率，并释放可扩展的 AI 商业价值与投资回报率（ROI）。麦肯锡（McKinsey）的研究数据表明，善于利用干净且经过妥善准备的数据的企业，其决策制定效率能够提升约 5 倍。 在正式讨论前，四位来自不同行业的资深专家依次进行了自我介绍： - **Santosh Vasudevan**（卡特彼勒 Caterpillar，首席数据科学家）：目前专注于大规模异常检测系统（Anomaly Detection at Scale）的构建与落地。 - **Rahul Zende**（Navy Federal Credit Union，首席数据科学家）：负责该机构的数据科学与人工智能（AI）战略规划及落地。 - **Tajinder Singh Chahal**（福特汽车 Ford Motor Company，数据工程与治理负责人）：重点聚焦于将大语言模型（LLMs）与生成式 AI（GenAI）能力深度融入数据治理与数据工程框架中。 - **Madhura Raju**（TikTok，首席产品经理）：当前主要攻克的核心问题是通过核心推荐系统优化信息流（Feed）质量，推动上线多款机器学习模型以识别低质内容，提升全球用户的产品体验。 整个专家团队横跨了零售电商、重型工程机械、金融与银行、汽车制造以及媒体传播等五大关键行业领域。 --- ### [02:07](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=127s) 数据质量对模型性能的决定性影响与跨部门沟通策略 业界普遍认为数据准备占据了数据科学项目 70% 到 80% 的工作量，但这一环节往往属于组织中“最隐蔽、最不易被看见的工作”。针对如何向容易低估此环节价值的利益相关者（Stakeholders）阐明其重要性，嘉宾们展开了深入探讨。Santosh 首先指出，在实践中，高质量的数据准备往往比模型算法本身的选择更重要。以异常检测系统为例，干净的数据准备能直接降低误报率（False Positives）并提高精确率（Precision）。在向业务团队沟通时，他主张将数据质量直接与业务痛点挂钩：更高的精确率意味着质量保障（QA）分析师和业务分析师无需将时间浪费在海量误报上，能更快定位故障根因，从而节省大量人力成本。 Rahul 补充道，数据科学本质上是一个高度迭代的过程，通常需要多轮调整才能满足业务需求。如果企业内部沉淀有“黄金标准”（Gold Standard）级别的高质量数据，模型从开发到上线生产环境的时间可以缩短 60% 至 70%。他建议企业建立完善的数据治理系统，对数据进行分层治理（如建立 Gold Tier 黄金数据层），为下游的分析与建模提供开箱即用的资产。 ```mermaid flowchart LR accTitle: 数据准备与业务价值传达闭环 subgraph DataEngineering["数据治理与工程"] A["原始数据 / 脏数据"] --> B["自动化清洗与特征工程"] B --> C["分层治理 (Gold Standard)"] end subgraph Modeling["模型开发与评估"] C --> D["高精度模型训练"] D --> E["减少误报 / 提升泛化能力"] end subgraph BusinessValue["业务价值呈现"] E --> F["缩短 60-70% 上线周期"] E --> G["分析师排障提速 & 降低人力成本"] E --> H["核心业务指标增长 (收入/留存)"] end ``` Tajinder 从组织架构与工程安全角度提出，数据准备绝非为了数据而数据的表面工作，而是一场需要深植于企业的组织变革。他用了一个生动而极端的工程比喻：“如果你把刚开采出来、未经提炼的原油直接灌进喷气发动机，不仅发动机会熄火故障，整个制造挑战甚至会导致整架飞机直接报废。”向管理层汇报时，他主要从三个维度量化价值：一是数据输入质量对模型预测精度的直接提振；二是对比手动清洗与自动化机器学习处理流程所带来的 ROI 与成本节约；三是建立把关机制，防止脏数据流入生产管线。 Madhura 随后分享了一个贴近生活的“厨房做饭”比喻：“如果你进厨房做意面，拿出的却是变质的橄榄和烂西红柿，那你做出来的意面必然糟糕透顶——这就是经典的‘垃圾进，垃圾出’（Garbage In, Garbage Out）。”她强调提升数据质量绝非选配项，而是业务发展的“力量倍增器”（Force Multiplier）。在 TikTok 的 A/B 实验与生产部署中，团队常遇到模型预期指标与实际表现大相径庭的情况——上线前预期能显著提升指标，上线后却发现实际涨幅连预期的 0.1% 都不到。每次召集数据科学家与标注人员复盘，根因几乎无一例外指向数据质量缺陷：模型无法捕捉真实的表征分布导致泛化失败。 - 数据准备是模型准确率提升 2 倍甚至 5 倍的核心杠杆，而非单纯依赖调参。 - 数据团队常陷入盲目调整超参数（Hyperparameters）或套用高阶算法的误区，往往忽略了基础数据质量缺陷才是限制模型效果的根本瓶颈。 --- ### [07:46](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=466s) 自动化数据管道工具链与强监管行业的合规挑战 在探讨数据准备的技术栈选择与自动化落地程度时，Santosh 详细介绍了卡特彼勒所采用的技术实践。在初期的数据摄取、清洗与特征工程阶段，团队高度依赖 Python 生态的核心库（如 NumPy 和 pandas）。为了确保整套流程具备高可维护性、透明度与可复现性，他们使用 Apache Airflow 等编排工具管理数据流。目前卡特彼勒能够实现数据准备流程约 60% 至 70% 的自动化，但依然在关键节点保留“人机协同”（Human-in-the-loop）的反馈机制。同时，团队引入了 MLflow 和 Weights & Biases（W&B）等实验跟踪工具，严格记录与比对数据清洗前后的指标变化。Santosh 强调，构建健壮的数据管道时应保持“工具无关性”（Tool-Agnostic），以便未来根据需要灵活切换底层工具。 对于身处强监管金融银行业务场景的 Rahul 而言，数据准备与数据访问面临着严格的合规与 SLA 挑战。金融机构必须在严格规定的时限内响应会员疑问、投诉以及监管要求。为了满足合规 SLA，技术架构必须将数据摄取、清洗、治理合规、数据精炼以及面向分析用例的数据呈现（Surfacing）整合成端到端的统一数据管道，确保数据流转在全生命周期内均能达到严格的合规时效与审计标准。 ### [10:25](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=625s) 银行业对数据管道的合规与低延迟要求 Rahul 解释了银行业在构建数据管道时的严苛标准。在金融机构中，技术选型必须首要保证**合规性（Compliance）**与**垂直整合（Vertical Integration）**能力，尽量选择高度集成、减少碎片化的生态系统。尽管开源工具能帮助团队搭建灵活的数据管道，但在银行业务场景下，系统的**低延迟（Low Latency）**和**高容错性（Fault Tolerance）**同样不可妥协。此外，整个技术栈需要尽量减少数据在不同系统之间频繁出入（Data Ingress / Egress）的管理开销，以确保数据安全与系统稳定性。 ### [11:21](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=681s) 现代化数据治理：从粗放管理到单元格级别的精细权限 主持人将话题引向 Ford Motor Company 的 Tajinder Singh Chahal，探讨数据治理（Data Governance）流程如何支撑数据转换与落地。Tajinder 回顾指出，早年间数据治理常被忽视，但随着 AI、机器学习以及大数据湖（Data Lakes）的迅猛发展，如果不遵循严格的标准，企业将面临巨大风险。特别是在 GDPR 和 CCPA 等隐私法规的约束下，合规已成为生命线。 业务团队总是希望在数据目录（Data Catalog）中查到数据后能“立刻”获取，而数据治理团队的原则绝非“不给数据”，而是确保“在明确的业务用例（Use Case）下，将正确的数据提供给正确的人”。针对 AI/ML 场景下数据在吞吐速度（Velocity）与多样性（Variety）上的爆炸式增长，Ford 采用了结合基于规则的策略（Rule-based Policies）、列掩码（Column Masking）以及行级安全（Row-level Security）的治理方案，最终实现**单元格级别（Cellular Level Access）**的精细化访问控制，而不再局限于粗粒度的表级或对象级权限。 ```mermaid flowchart TD accTitle: AI-Driven Data Governance Pipeline subgraph Ingestion["数据接入与智能检测"] A["原始数据接入生态"] --> B["AI/ML 算法自动扫描"] B --> C["识别 PII 敏感特征<br>(SSN / 信用卡 / 出生日期)"] end subgraph Governance["治理策略与细粒度控制"] C --> D["自动打标签与风险评级"] D --> E["单元格级访问控制<br>(行级过滤 + 列掩码)"] end subgraph Monitoring["可信度与审计追踪"] E --> F["端到端血缘追踪<br>(Data Lineage)"] E --> G["访问日志与可观测性审计<br>(Observability Logs)"] F --> H["高可信度数据产品<br>(供模型训练消费)"] end ``` 为高效识别 PII（个人身份信息），团队利用 AI/ML 模型在数据落地的瞬间自动打标签，精准定位社保号（SSN）、信用卡号、出生日期等敏感字段，并评估对终端用户与组织的潜在风险。Tajinder 还强调了**数据血缘（Data Lineage）**与**可观测性（Observability）**的重要性： - **血缘追踪**：帮助终端用户判断数据来源是否可靠，避免下游模型消费空值率高达 80% 的不可用数据。常用工具包括 Collibra、Alation 以及 Google Cloud 的 Dataplex（支持表级并逐步扩展到列级血缘）。 - **访问审计日志**：实时监控谁在访问数据、是否契合报备的业务用例、是否存在越权批量拉取等异常行为。 主持人总结，这一整套治理机制标志着企业内部**数据访问契约（Data Access Contracts）**的演进与成熟。 ### [15:23](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=923s) 交付速度与数据质量的平衡之道 针对主持人提出的“数据科学团队面临快速交付洞察的压力，如何平衡深入数据准备与敏捷速度”的问题，Caterpillar 的 Santosh Vasudevan 坦言这是一场“持续的平衡博弈（Balancing Act）”： 1. **敏捷迭代而非追求初期完美**：在项目初期阶段，数据准备管道无需一步到位做到完美，只要“足够好以产出洞察并指导下一步行动”即可。 2. **模块化代码复用（Modularized Code）**：跨项目沉淀处理缺失值、异常值（Outliers）以及数据去噪的标准模块，避免重复造轮子。 3. **引入 LLM 编程助手提效**：将 GitHub Copilot 等大模型工具引入数据管道构建，快速生成诸如从 Amazon S3 搬运数据至 Snowflake 等模板化样板代码（Boilerplate Code），让数据工程师集中精力攻克核心业务逻辑。 4. **建立“数据准备是迭代过程”的业务文化**：引导业务利益相关者（Stakeholders）理解数据准备绝非“一锤子买卖（One and Done）”，必须随业务逻辑和需求的变化持续演进与无缝集成。 ### [17:27](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1047s) 模型迭代周期与特征库（Feature Store）的核心价值 Rahul 针对模型达到业务方验收（Stakeholder Acceptance）所需的迭代轮次进行了深入剖析。他指出迭代次数具有高度主观性，完全取决于模型复杂度与业务交付周期： - **简单模型**：若预期在 1 个月内完成构建与部署，通常基于初始基线（Baseline）经历 2 到 3 轮迭代。 - **高复杂度模型**：需要大量业务协同与审批时，经历 5 到 7 轮迭代十分常见。 Rahul 强烈赞同“垃圾进，垃圾出（Garbage in, Garbage out）”的铁律：如果底层数据根本没准备好，无论怎么“把海水煮干（Boil the ocean，指做无用功）”，也无法交付有实际业务价值的方案。 高质量的元数据（Metadata）能大幅缩短模型构建时间，而 Rahul 从切身经验出发，强调了构建**特征库（Feature Store）**的重要性。特征库提前将各类统计指标和预测因子（Predictors / Variables）聚合好，使数据科学家能够即取即用以构建基线模型。这不仅大幅缩短了初始构建周期，而且在银行业务中由于合规限制需要调整策略（Pivot）或规避掠夺性/偏见特征（Predatory data）时，能让团队快速重构并重新迭代模型。 ### [20:17](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1217s) 数据去噪的长期价值与企业文化激励 在机器学习与预测建模中，数据的合法合规与质量是模型的生命线。预测性成果必须依赖于正确、健康的状态与数据；如果使用了错误的数据，不仅模型效果大打折扣，甚至可能引发具有“掠夺性”（predatory）或违规伦理的问题。因此，完备的元数据（metadata）治理与易于获取的特征存储（Feature Store）至关重要，它们能帮助团队快速选取正确特征，大幅缩短模型构建和部署的周期。 面板嘉宾补充强调，数据去噪（denoising）和清洗在短期内确实会拖慢研发节奏，但从长远来看却能带来更高、更持久的商业影响。现实中团队往往面临两难抉择：是图快用劣质数据粗制滥造三款模型，还是花上整整一个月进行去噪、剔除异常值、制定清洗策略，进而打造一个高影响力、经得起考验的长期模型？这本质上是一种“企业文化”。企业高层领导必须将数据质量视为核心投资领域，并在绩效机制上予以倾斜——不能仅仅激励员工“快速上线功能或模型”，更应建立机制鼓励团队花时间去清洗数据、剔除“垃圾数据”，从而沉淀出高价值资产。 - **短期速度 vs 长期影响**：前期投入时间进行数据清洗与去噪会暂时放慢上线速度，但能换来长期稳定的模型表现。 - **管理层激励转向**：企业需要改变只看重“交付速度与功能数量”的 KPI 导向，激励工程人员投入时间打磨高质量底层数据。 --- ### [22:10](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1330s) 从 Notebook 孤岛到数据工程流水线：特征工程的范式转移 针对数据准备工作向数据工程（Data Engineering）转移的趋势，Tajinder Singh Chahal（福特汽车）给出了高度认同。过去，数据科学团队普遍在各自独立的 Jupyter Notebook 里单打独斗，分别进行前期数据分析、异常值排查、特征构建，最后再训练模型。这种各自为政的“数据孤岛”模式导致针对同一数据源重复造轮子，既浪费算力成本，又严重拉长交付周期。如今，行业正在发生根本性转变：将数据剖析与清洗直接前置并集成到数据流水线（pipeline）中，能够带来 3 到 10 倍的速度与效率优化。 在这种新架构下，流水线在被触发执行前，就会自动检测数据中的异常值、标准差及各项统计指标，确保只有达标的数据才能流入下游。与此同时，统一的特征存储（Feature Store）与企业级数据标准化机制有效解决了数据类型混乱的痛点。例如，统一制定日期时间格式（datetime format）或客户 ID（Customer ID 是整型还是字符串），避免不同团队因规范不一而产生摩擦。数据工程团队与数据治理团队提前规范这些复杂数据类型并注入特征库，使数据科学团队能够彻底从繁琐的数据清洗中解脱出来，专注于模型算法本身的产出。 ```mermaid flowchart LR accTitle: 数据准备从 Notebook 孤岛转向数据流水线与特征库 subgraph S1["传统模式：单打独斗（低效/孤岛）"] A["原始数据"] --> B["Notebook 孤岛清洗"] B --> C["重复特征工程"] C --> D["各自训练模型"] end subgraph S2["现代模式：流水线与标准化（提速 3x-10x）"] E["数据源"] --> F["流式/批处理流水线<br/>(自动统计指标/离群值拦截)"] F --> G["特征存储 (Feature Store)<br/>(全局统一数据标准)"] G --> H["模型训练与推理"] end ``` - **效率飞跃**：将清洗逻辑前置至工程流水线，相比单机 Notebook 处理可实现 3x 至 10x 的性能与时间优化。 - **自动化质量把关**：在触发下游处理前，系统自动计算标准差、均值等统计量，拦截异常数据。 - **格式与类型统一**：在底层统一 Customer ID 等核心字段类型及日期格式，消除不同团队间的标准冲突。 --- ### [24:14](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1454s) 数据产品生命周期中的跨部门协同契机 关于数据工程团队究竟应该在何时介入支持数据科学团队，Tajinder 强调，成熟企业的最佳实践是在“构建数据产品之前”就将所有利益相关者召集到同一张桌子上。这包括数据科学家、分析团队、机器学习负责人、数据工程以及数据治理专家。在立项会议上，团队共同敲定核心模式：接入哪些数据源、采用何种标准化规范、追踪哪些核心指标，以及模型预测的目标是否符合全公司的统一标准。 这种协同不是一蹴而就的单次行动，而是伴随 MVP（最小可行产品）推进的持续迭代过程。当团队推出第一个 MVP 版本时，数据科学团队便能即时验证：这是否是正确的数据产品？特征是否完整可用？模型的首批预测结果是否符合预期？随着业务演进，一旦引入新数据源或发生 Schema 字段变更，数据科学团队就必须再次与数据工程、数据治理团队紧密联动，确保全流程处于受控且高水准的状态。 --- ### [25:58](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1558s) 银行业多供应商架构下的数据摄取与在途转换挑战 Rahul Zende（Navy Federal Credit Union）结合自身在银行业的经验，深入剖析了异构系统集成中的隐患。金融机构通常依赖众多外部供应商平台：抵押贷款系统、信用卡处理系统以及消费信贷系统可能来自完全不同的厂商。当这些分散的数据通过摄取链路汇聚到企业级 Lakehouse（湖仓）时，前期的跨部门沟通显得尤为关键。 由于各平台数据结构迥异，工程师在入库时往往会做出主观的选择与取舍。Rahul 强调，如果没有数据科学团队的早期介入，极大概率会发生高价值特征被误当作“无用字段”直接丢弃的情况；一旦这些关键元素没有进入 Lakehouse，下游就彻底丧失了挖掘某些预测能力的可能。此外，如果能在数据流入湖仓的“在途阶段”（in transit / stream processing）直接完成特征清洗与转换，就能一劳永逸地惠及整个下游，避免不同分析师在搭建报表、物化视图和构建算法时各自进行无休止的重复计算。 - **多供应商异构环境**：银行核心业务往往跨越不同的供应商系统（如信用卡、房贷、消费贷系统），接入链路极为复杂。 - **避免特征误删**：缺乏算法团队参与的粗暴数据摄取，往往会导致具有高预测潜力的字段在入库前被当作冗余丢弃。 - **流式在途转换（Stream Processing）**：在数据流向湖仓的过程中直接执行标准化转换，能为下游仪表板与模型研发节约巨大的重复算力与人力成本。 --- ### [28:14](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1694s) 用数据说服决策层：从 A/B 测试到数据与模型漂移监控 Madhura Raju（TikTok）从产品经理的视角分享了跨部门推动数据质量建设的“元策略”（meta concept）——**用数据本身来说服决策者**。每次走进领导层的会议室争取资源时，唯有硬核的数据对比最能打动人心。例如，为了证明在数据清洗上的投入是值得的，团队可以设计严谨的对比实验：一组采用未经清洗的 v0 基础版本训练数据，另一组结合主动学习（Active Learning）等去噪策略产出 v1 训练数据，然后通过离线评估对比两者在精确率（Precision）和召回率（Recall）上的显著差距。这种实实在在的性能提升，正是推动管理层批准项目预算的最佳利器。 随后，对话自然延伸到了可观测性（Observability）这一保障环节。不仅数据本身会发生数据漂移（Data Drift），模型在线上的表现也会随着时间发生衰退（Model Decay）。“现实世界永远在变”，任何上线发布时的优秀模型，一个月后都可能失去原有的水准，因为现实中的用户行为与业务模式每天都在发生动态演变。因此，持续的数据监控与模型表现跟踪是维系系统长期健康的底层基础设施。 - **用实验证明数据价值**：通过对比清洗前（v0）与清洗后（v1，引入主动学习等技术）的模型精确率与召回率，为数据质量工程争取管理层资源。 - **模型衰退的必然性**：现实世界模式的动态变化导致线上模型不可避免地发生性能衰减，无法实现“一劳永逸”的部署。 - **数据可观测性**：必须对底层的输入数据漂移（Data Drift）与顶层的预测效果衰退建立端到端的实时监测机制。 ### [029:45] 大语言模型（LLM）与自动化：重塑数据工程与元数据治理 针对数据准备流程中正在发生的技术变革，Navy Federal Credit Union 的 Rahul Zende 首先分享了战略层面的观察。他指出，随着大语言模型具备了强大的上下文自动化理解能力，数据工程流水线正迎来深远转变。过去，为了让下游应用充分释放数据价值，数据工程师必须进行大量繁琐且手动的元数据打标（Metadata Tagging）；如今，无论是利用第三方厂商平台还是自研解决方案，LLM 都可以自动接管元数据标注，并在流水线中设定数据清洗、转换与处理的底层规则。Rahul 强调，这并不意味着人工环节被彻底取代，数据工程团队依然需要对模型标注的字段与表结构进行校验（Human-in-the-loop），但这将极大免除工程师在接入新数据源或记录系统时那些“极其枯燥、重复的工作”，使企业吸纳新数据库、实现数据资产价值的速度显著提升。 ```mermaid flowchart TD accTitle: LLM 赋能的数据工程与准备流程 subgraph Ingestion["数据摄入与理解"] A["新数据源 / 记录系统 (System of Record)"] --> B["LLM 自动化上下文解析与模式识别"] B --> C["自动元数据打标 (Metadata Tagging)"] end subgraph Transformation["规则清洗与校验"] C --> D["自动拟定清洗与转换规则"] D --> E["数据工程师校验 (Human-in-the-loop)"] end subgraph Governance["跨域治理与资产化"] E --> F["Data Mesh 领域数据产品 (Data as a Product)"] F --> G["企业级下游分析与 AI/ML 模型训练"] end ``` 福特汽车的 Tajinder Singh Chahal 进一步拓展了 LLM 在数据质量审查和异常值探测中的作用。在传统统计模型中，人工很难在成千上万张表中跨属性梳理关联性（人类在忙碌时或许只能比对数个或十几个特征），而大模型能够通过图分析与跨表计算快速定位异常。在汽车制造与销售场景中，企业通常拥有客户域（Customer Domain）、车辆域（Vehicle Domain）和销售域（Sales Domain）等高度复杂的业务线，LLM 能够自动跨数据集乃至跨云平台项目（例如 Google Cloud Platform 跨 GCP 项目级别）探索各领域之间的潜在关联与相关性，帮助团队直接输出分析洞察与业务问答。 - **元数据自动打标与清洗**：LLM 大幅压缩了传统数据摄入时的手动规则配置周期，自动生成数据转换逻辑。 - **跨项目级特征关联发现**：AI 辅助工具能够在成千上万张业务表之间自动化发现隐藏的统计关联与离群点。 - **Schema 结构的重要性**：LLM 进行精准标注与处理的高度依赖于底层元数据与结构化模式（Schema）的规范表达。 --- ### [033:35] Data Mesh 架构演进与合成数据（Synthetic Data）的新趋势 在谈及架构演化趋势时，Tajinder 重点强调了数据网格（Data Mesh）和数据融合（Data Fusion）对大型企业的战略价值。Data Mesh 倡导将“数据即产品”（Data as a Product）理念融入各业务领域，打破传统的中心化数据治理壁垒，将治理责任与所有权分散至各个具体的领域专家、业务团队和专属数据工程师。这种架构的核心优势在于：企业无需跨物理节点大规模搬运底层数据，只需在全局建立统一的逻辑层（Logical Layer），并配套精细的行级与列级安全访问权限控制（Row/Column Level Security），即可在各业务域之间实现端到端的数据可观测性（Data Observability）。 TikTok 的 Madhura Raju 则从更高维度的宏观数据供给切入，指出了“合成数据（Synthetic Data）”正在成为缓解数据荒的关键路径。她援引了 Elon Musk 与 Y Combinator 总裁 Garry Tan 的对谈，以及《Nature》杂志的相关研究预测：到 2028 年左右，互联网上由人类生成的高质量训练数据可能面临耗尽危机。为了避免模型训练陷入停滞并提升样本代表性，利用生成式 AI（GenAI）合成高质量训练数据已在自动驾驶等行业广泛应用，并正在快速渗透到其他 AI 业务场景中。 - **去中心化治理（Data Mesh）**：数据所有权回归领域团队，避免数据物理搬迁，依托逻辑层与行列级权限构建数据产品。 - **数据供给瓶颈**：随着高质量人类文本与交互数据的枯竭，合成数据技术正成为前沿 AI 训练不可或缺的补充手段。 --- ### [036:03] 从试点到企业级落地：规模化推广中最容易被忽视的隐患 在圆桌讨论“将数据科学项目从 PoC 试点推向全企业级推广时最容易忽视的因素”时，各位专家结合自身一线经验给出了核心洞察： Caterpillar 的 Santosh Vasudevan 率先指出，最容易被低估的是**组织协作中的“人的因素”**。即使企业采购并搭建了顶级的技术栈，但如果不同团队之间缺乏统一的数据准备规范与共享标准，各团队在数据清洗、转换上的微小不一致在单点试点时或许不明显，一旦向企业级规模化放大，这些不一致性就会被成倍放大，导致后续大量的返工与流程内耗。 Rahul Zende 紧接着补充，除了组织间的人员孤岛，**数据孤岛（Data Silos）**更是模型落地的致命伤。在模型最初开发时，很多极具预测价值的信号往往被锁死在各部门孤岛中而未被纳入，导致模型上线后效果不佳，不得不频繁推出 V2、V3 版本进行补救。因此，打破孤岛、在全企业范围内构建统一的预测信号与语义层（Semantic Layer）至关重要。 Tajinder Singh Chahal 最后强调了**受认证数据（Certified Data）**与信任机制的构建。数据从原始形态流向下游终端用户，必须经过严格的合规治理与工程流水线检验。值得注意的是，受认证的数据产品并不一定局限于最终的高聚合分析层；例如在福特汽车的网联车场景中，直接从车辆端流式采集的原始遥测数据流（Raw Streaming Telemetry），只要在接入时建立起完备的质量与安全认证，同样可以在原始层作为经过认证的高可信数据产品直接对下游提供服务。 | 核心阻碍维度 | 常见痛点表现 | 应对策略与最佳实践 | |:--- |:--- |:--- | | **团队协同与规范（人）** | 团队间数据准备策略不一，各自为战 | 制定跨团队统一的数据工程标准与共享操作指南 | | **数据资产流通（数据孤岛）** | 关键预测特征被隔绝，模型上线后频繁返工迭代 | 构建企业级逻辑层与语义层，开放全局特征信号 | | **数据可信度（治理与认证）** | 缺乏数据质量背书，下游模型不敢用、不可靠 | 实施全流程流水线治理，打造多层次的“受认证数据产品” | ### [39:13](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2353s) 认证数据产品与从 PoC 走向企业的核心盲区：指标体系构建 在探讨如何定义“认证数据产品”（Certified Data Product）时，嘉宾指出，只有当数据源自真正单一且权威的可信源头（Single Source of Truth）、明确定义了领域专家（SME）并遵循严格的领域规范与工程实践时，团队才能安心直接调用，并将其正式称为认证数据产品。 ```mermaid flowchart TD accTitle: 从 PoC 到企业级落地的指标与数据认证流 subgraph S1["数据认证与可信源头"] A["业务领域规范与实践"] --> C["认证数据产品 (Certified Data Product)"] B["明确领域专家 (SME) 责任"] --> C end subgraph S2["PoC 到 Enterprise 跃迁"] C --> D["PoC 验证阶段"] D --> E["层级化指标体系 (Hierarchy of Metrics)"] E --> F["模型层指标 (Model-level)"] E --> G["生产层指标 (Production-level)"] F & G --> H["企业级规模化上线 (Enterprise Production)"] end ``` 针对数据准备和模型落地过程中的盲区，TikTok 首席产品经理 Madhura Raju 补充强调，企业最容易忽视的核心盲区是**指标体系（Metrics）的量化与分层**，特别是在将项目从概念验证（PoC）推向企业级（Enterprise）生产环境的过程中： * **指标缺失现状**：在与许多小型企业交流时，极少有人会主动规划系统性的指标；大家往往只关注 PoC 是否跑通，却忽略了迈向规模化上线所必须跨越的标准。 * **层级化指标体系**：从 PoC 演进到企业级系统，必须建立一套自上而下的指标层级结构（Hierarchy of Metrics），明确在什么阶段达成何种硬性标准才能推进上线。 * **多维度量化评估**：指标不能仅停留在算法层面，既要涵盖模型层指标（Model-level metrics），也要定义生产与工程层指标（Production-level metrics），清晰量化目标与业务推进的进度。 --- ### [40:16](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2416s) 观众问答：通过数据管治（Stewardship）从上游终结“打地鼠”循环 在小组讨论进入现场问答环节后，观众 Lara Guridge 提出了一个犀利而关键的观点：尽管大家深入讨论了数据清洗与处理，但整个对话中最为核心却未被充分强调的一环是**数据管治（Stewardship）**。业界常说的“70% 时间被耗费在数据准备上”，要想真正缩减这部分成本，就必须追溯到整个数据流水线的**最上游（Upstream）**去建立管治机制。如果不从源头提升全组织对数据管治的认知素养（Data Literacy & Stewardship），工程团队就只能陷入反复修补、机械化清洗数据的“打地鼠”（Whack-a-mole）死循环。 ```mermaid flowchart LR accTitle: 双轨数据管治模型 subgraph DataAsset["数据资产 / 数据集 (Data Set)"] direction TB DS["统一数据源"] end subgraph DualStewardship["双轨数据管治体系 (Dual Stewardship)"] Tech["技术管治者 (Technical Steward)<br/>- 架构与流水线维护<br/>- 空值校验与数据质量"] Biz["业务管治者 (Business Steward)<br/>- 业务价值定义<br/>- 领域规则与语义校准"] end Tech --> DS Biz --> DS ``` 面对这一提问，嘉宾们展开了积极的共鸣与深化补充： * **领域归属即管治**：嘉宾澄清并指出，此前讨论的“领域所有权”（Domain Ownership）本质上就是数据管治。无论是数据工程师、数据所有者还是领域所有者，他们最了解数据的真实价值，其核心职责就是确保数据来自受信任的源头，并从源头把关属性质量，杜绝大量空值（Null values）或噪声流入下游。 * **双轨管治架构（Technical + Business）**：针对具体落地方式，小组嘉宾特别强调，单一数据集绝不能仅绑定一个数据管治专员（Data Steward），否则极易导致个人职责过载。最理想的模式是采用**技术专员与业务专员结对**的双轨制——由技术人员保障流水线规范与工程质量，业务人员把控数据语义与商业价值，两者协同才能最大限度释放数据资产的潜力。 * **全员集体责任**：数据管治不仅是某个岗位的专职工作，更是贯穿整个组织数据生态的“集体责任”（Collective Responsibility）。 ## Notable quotes > [02:07](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=127s) 如果你把刚开采出来、未经提炼的原油直接灌进喷气发动机，不仅发动机会熄火故障，整个制造挑战甚至会导致整架飞机直接报废。 — Tajinder Singh Chahal > [02:07](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=127s) 如果你进厨房做意面，拿出的却是变质的橄榄和烂西红柿，那你做出来的意面必然糟糕透顶——这就是经典的‘垃圾进，垃圾出’（Garbage In, Garbage Out）。 — Madhura Raju > [11:21](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=681s) 在明确的业务用例（Use Case）下，将正确的数据提供给正确的人 — Tajinder Singh Chahal > [15:23](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=923s) 足够好以产出洞察并指导下一步行动 — Santosh Vasudevan > [15:23](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=923s) 数据准备绝非‘一锤子买卖（One and Done）’，必须随业务逻辑和需求的变化持续演进与无缝集成。 — Santosh Vasudevan > [17:27](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1047s) 如果底层数据根本没准备好，无论怎么‘把海水煮干（Boil the ocean，指做无用功）’，也无法交付有实际业务价值的方案。 — Rahul Zende > [28:14](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1694s) 用数据本身来说服决策者 — Madhura Raju > [28:14](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=1694s) 现实世界永远在变 — Madhura Raju > [36:03](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2163s) 最容易被低估的是组织协作中的‘人的因素’。 — Santosh Vasudevan > [40:16](https://www.youtube.com/watch?v=uM4TGd_-ZrI&t=2416s) 如果不从源头提升全组织对数据管治的认知素养（Data Literacy & Stewardship），工程团队就只能陷入反复修补、机械化清洗数据的‘打地鼠’（Whack-a-mole）死循环。 — Lara Guridge ## Mentioned Chetana Katpally, Santosh Vasudevan, Rahul Zende, Tajinder Singh Chahal, Madhura Raju, Lara Guridge, Elon Musk, Garry Tan, Macy's, Caterpillar, Navy Federal Credit Union, Ford Motor Company, TikTok, McKinsey, Y Combinator, Nature, Apache Airflow, Python, NumPy, pandas, MLflow, Weights & Biases (W&B), Collibra, Alation, Dataplex, Google Cloud Platform (GCP), GitHub Copilot, Amazon S3, Snowflake, GDPR, CCPA ## Video description Chetana Katpally, Senior Manager - Machine Learning Engineering, Macy's Rahul Zende, Principal Data Scientist, Navy Federal Credit Union Santosh Vasudevan, Lead Data Scientist, Caterpillar Tajinder Singh Chahal, Data Engineering Manager, Ford Motor Company Madhura Raju, Principal Product Manager, TikTok Inc. Abstracts This session will focus on the critical process of preparing data for AI and machine learning models, exploring best practices for cleaning, transforming, and structuring data. Panelists will discuss how to handle common challenges like missing values, noisy data, and outliers to ensure the data is accurate and ready for modeling. Learn about the tools and techniques that can automate and streamline data preparation workflows, and gain insights into how proper data prep can dramatically improve model performance and reliability. [Source](https://www.youtube.com/watch?v=uM4TGd_-ZrI)

[2:07] Tajinder Singh Chahal: 如果你把刚开采出来、未经提炼的原油直接灌进喷气发动机，不仅发动机会熄火故障，整个制造挑战甚至会导致整架飞机直接报废。 [2:07] Madhura Raju: 如果你进厨房做意面，拿出的却是变质的橄榄和烂西红柿，那你做出来的意面必然糟糕透顶——这就是经典的‘垃圾进，垃圾出’（Garbage In, Garbage Out）。 [11:21] Tajinder Singh Chahal: 在明确的业务用例（Use Case）下，将正确的数据提供给正确的人 [15:23] Santosh Vasudevan: 足够好以产出洞察并指导下一步行动 [15:23] Santosh Vasudevan: 数据准备绝非‘一锤子买卖（One and Done）’，必须随业务逻辑和需求的变化持续演进与无缝集成。 [17:27] Rahul Zende: 如果底层数据根本没准备好，无论怎么‘把海水煮干（Boil the ocean，指做无用功）’，也无法交付有实际业务价值的方案。 [28:14] Madhura Raju: 用数据本身来说服决策者 [28:14] Madhura Raju: 现实世界永远在变 [36:03] Santosh Vasudevan: 最容易被低估的是组织协作中的‘人的因素’。 [40:16] Lara Guridge: 如果不从源头提升全组织对数据管治的认知素养（Data Literacy & Stewardship），工程团队就只能陷入反复修补、机械化清洗数据的‘打地鼠’（Whack-a-mole）死循环。

[0:04] Chetana Katpally: organizations utilizing clean, well-structured data can accelerate decision-making by roughly five times, while simultaneously reducing model errors and maximizing ROI. [2:07] Tajinder Singh Chahal: feeding unrefined data into machine learning models is like pumping crude, unrefined fuel directly into a jet engine—it doesn't just fail; it risks catastrophic damage that could scrap the entire airplane. [2:07] Madhura Raju: if you cook with stale olives and rotten tomatoes, your pasta will turn out terrible regardless of cooking technique. [11:21] Tajinder Singh Chahal: right data to the right people for the right use case. [17:41] Rahul Zende: garbage in, garbage out [20:17] Companies typically incentivize engineers and data scientists to ship shiny features and launch models at breakneck speed, but organizations must actively incentivize employees to spend dedicated time eliminating "garbage" and raising raw data quality. [21:54] Shifting profiling and feature preparation upstream into standardized data engineering pipelines can optimize turnaround times by 3x to 10x. [35:17] Madhura Raju: the tech industry is on trajectory to exhaust human-generated training data across the internet by 2028. [40:16] Lara Guridge: whack-a-mole game

[0:04] Hello everybody. Thank you for joining [0:05] us on this session of mastering data [0:07] preparation strategies for clean and use [0:10] and usable data. So a well-prepared data [0:13] accelerates decision making. It reduces [0:16] model errors and also unlocks scalable [0:19] AI value and return on investment. Now [0:22] according to McKenzie um it highlights [0:24] that companies leveraging clean and [0:27] well-prepared data can improve decision [0:29] making by about five times. [0:32] I'm Chetna Katpali. I am the senior [0:34] manager of machine learning at Macy's [0:37] and we have a fantastic panel joining us [0:40] today. So before we dive in, let's go [0:42] around for a quick round of [0:43] introductions. Panel members, please uh [0:46] say your name, your role, and in one [0:49] sentence, what are you working on today? [0:51] Uh thank you all for coming. Uh thanks [0:52] Chetna. Uh I'm Santos, a data scientist [0:55] at Caterpillar. Uh I focus on building [0:58] anomaly detection at scale. Just anomaly [1:00] detection systems at scale. [1:02] >> Hi everyone, I'm Rahul. I'm a principal [1:04] data scientist at Navy Federal Credit [1:06] Union. Um and I do data science and AI [1:09] strategy over there right now. [1:12] >> Hello everyone. Thanks for joining us. [1:14] I'm Tajenda Singh Chahal. Um I work for [1:16] Ford Motor Company as a data [1:18] engineering, data governance leader over [1:19] there and the current focus which we do [1:21] is basically building all the LLMs and [1:24] geni capabilities into the geni world [1:27] into the machine into the governance and [1:29] the data engineering framework and [1:31] that's where my skills are. [1:34] Hey [1:34] >> everyone, super excited to be here. I'm [1:36] Adura Raju. I'm a product manager at Tik [1:38] Tok. Uh the biggest problem that I'm [1:40] solving for Tik Tok right now is [1:42] improving the feed um um improving the [1:45] quality of the feed through the core [1:47] recommendation system. For this we ship [1:48] a bunch of machine learning models that [1:50] identify lowquality content and you know [1:53] make the user experience much better for [1:55] the users across the world. [1:57] >> Thank you everybody. So we have a [1:59] diverse panel here today construction [2:01] finance and banking uh manufacturing and [2:04] media and communication. Okay. So the [2:07] first question let's do a round robin [2:09] here. So it's often said that data [2:11] preparation takes about 70 to 80% of the [2:14] work for data science projects. Now from [2:16] your experience, how much impact does [2:19] high quality data preparation really [2:22] have on your model performance and how [2:25] do you communicate its value to [2:27] stakeholders who might underestimate [2:29] this part of the pipeline knowing that [2:32] that is the most um unseen work. by the [2:36] by the organization. [2:39] >> I can I can go first. [2:40] >> Sure. Yes, please. [2:41] >> Uh so in my experience, uh high quality [2:44] data prep is often more important than [2:46] the algorithm you choose uh to build [2:49] your model on and especially in in terms [2:52] of anomaly detection. Uh highquality [2:54] data prep results in reduced number of [2:57] false positives and increased precision [2:59] in in the model. And the way uh I [3:02] communicate this value with the [3:03] stakeholders is by tying the outcome of [3:06] the uh model uh to the high quality [3:09] data. For example, if uh model has uh [3:12] less false positive and high precision, [3:14] that would mean uh the quality assurance [3:17] analysts and business analysts who [3:19] review for example the anomalies uh and [3:22] alerts. They would have to spend less [3:24] time and they would be able to get to [3:25] the root cause of the issue a lot [3:28] faster. uh that's how I convey the uh [3:31] the benefits of a high quality data prep [3:34] pipeline. [3:36] >> Yeah, I can add to that. Um yeah, [3:39] definitely as Santo said u data science [3:42] is definitely an iterative process. You [3:44] have multiple iterations before you [3:45] reach a complete model that satisfies [3:48] business requirements. Uh thinking of [3:51] that, uh I think u there's definitely a [3:54] lot of value if you have data that's [3:56] gold standard ready for data science. uh [3:58] model building and usage you can cut [4:01] down on the time to production by almost [4:04] uh 60 to 70% per my experience. Uh so [4:09] from saying that um I would say uh [4:11] there's a lot of value in organizations [4:13] having different tiers of data [4:15] especially the gold tier of data that [4:17] that could be considered the most ready [4:18] for uh usage in analytics and especially [4:21] data science models having that ready [4:23] definitely helps. uh and from a from a [4:26] technical standpoint, I think every [4:28] organization should have a system uh [4:30] data governance system that enables this [4:33] in in in within the company. [4:35] >> I totally agree with that. Right. And [4:37] when I look into this data preparation, [4:40] it's not a data preparation done just [4:42] for a sake that the data should be good. [4:44] It's an organizational change which has [4:46] to be embedded because data is the key. [4:49] It's the raw element for any of our data [4:52] products. If your raw element is bad [4:55] then your whole product will be bad. [4:57] Right? That's how we look into it. I [4:58] think the important questions which she [5:00] asked was like how do you how do you [5:01] factor this uh data prep to your key [5:04] stakeholders and I think uh when I when [5:06] I need to articulate this to my [5:08] stakeholders or my leaders right uh the [5:10] very first thing is uh we start with how [5:13] the model is predicting. So how much [5:15] effort did we take in regards like where [5:17] the input was and how how exactly the [5:19] model accuracy the uh results has been [5:22] improved. So there's an increase with [5:23] that uh model accuracy which has to be [5:26] articulated with the help of the data [5:28] preparation. Second, I see ROI and also [5:32] the cost associated with some of the [5:34] manual process versus AML machine [5:36] learning processes, right? And the third [5:38] which I also think is the data prep is [5:40] like as I said it's a raw element, [5:42] right? And if you if you give a raw [5:46] curated fuel which just extracted to a [5:49] jet engine, it will basically not just [5:53] fail but it will never it will begin in [5:56] a big distract like big challenges and [5:58] big manufacturer that your whole [6:00] airplane has to discard it. So that's [6:01] how we look into uh data prep as a key [6:05] element before it gets into any of the [6:07] production pipelines in the future. [6:10] >> That was a scary example. Let me give [6:12] another one. Let's say you're going to [6:13] the kitchen and you have like a bunch of [6:15] ingredients and if you have like stale [6:17] olives and like you know stale tomatoes, [6:18] your pasta is going to be shitty, right? [6:20] So that's that's it. Like garbage in, [6:22] garbage out. So improving data quality [6:24] is not really an option. It's a force [6:26] multiplier for improving your business. [6:28] There is a direct link between you know [6:31] your data quality, your model precision [6:33] recall and then your business metrics [6:35] like engagement, subscription, revenue, [6:37] whatever. for in my experience we've [6:39] shipped so many models in like you know [6:41] for AB experiments and also to [6:42] production and we expect that okay this [6:44] particular model should move this metric [6:46] by X and then you look at the AB [6:48] experiments and you look at the [6:49] production models and you're like it's [6:51] like barely X it's like you know like [6:52] 0.1% of X or something and then I I [6:56] bring the team together to like data [6:57] scientists and annotation you know like [6:59] labelers and look into the actual data [7:02] and almost always it is a data quality [7:05] issue like the model is not able to [7:07] capture [7:08] um you know all the all the different [7:10] it's not generalizing well you're not [7:12] able to capture all the representations [7:14] that your model should capture. So you [7:16] go back to the drawing board and you [7:18] figure out you know how to improve the [7:19] data. So it's absolutely critical and [7:21] it's not an option. [7:23] >> Just to add to one more point over [7:25] there, right? When you build a model, I [7:26] think most of the times the data science [7:28] team what they think is the model is not [7:30] generating the right output, right? So [7:31] they start tweaking the hyperparameters [7:34] and all those advanced machine learning [7:36] algorithms rather than looking into [7:38] there could be a simple data quality [7:40] issue which can really increase the [7:41] model accuracy by 2x or 5x as you [7:44] mentioned. [7:46] >> That segus into my next question. So [7:48] there's a growing ecosystem of tools and [7:51] frameworks for data cleaning, [7:52] transformation and validation. So what [7:55] tools or technologies have you [7:57] integrated into your data preparation [7:59] workflows and how much of your process [8:02] have you been able to automate versus [8:04] relying on manual intervention? Santos, [8:08] >> it's a really good question and I would [8:10] say uh we leverage a lot of uh the [8:14] Python libraries for initial uh data [8:17] prep and data processing for example [8:19] numpy and and pandas uh for everything [8:22] starting from data ingestion uh to [8:25] cleaning creating uh features of feature [8:27] engineering that the model can monitor [8:29] and learn from. uh and uh we want to [8:33] implement this as uh a maintainable [8:37] transparent uh reproducible pipeline uh [8:40] and this is achieved by leveraging [8:42] orchestration tool data orchestration [8:44] tools like airflow uh we able to uh auto [8:48] automate I would say almost 60 to 70% of [8:51] the process but it's important to have [8:53] uh human in the opportunity for human in [8:55] the loop feedback uh we also leverage [8:58] some experimentation and logging [9:00] libraries like MLflow weights and biases [9:02] to track pre and post uh data cleaning [9:06] metrics which is very important. Uh but [9:08] I think the the philosophy should be to [9:12] at the end of the day to build uh robust [9:14] uh data prep pipelines and be more uh [9:18] tool agnostic so that you can swap out [9:20] tools uh if if needed. [9:23] >> Got it. And so Rahul so in the regulated [9:26] industry uh what are some of the [9:28] challenges you have in accessing data [9:30] and how have you resolved them? [9:32] >> Yeah I think uh from a regulation [9:34] standpoint uh in the banking industry we [9:37] have uh a lot of requirements uh in [9:40] order to stay compliant we have to [9:41] respond to certain uh member uh [9:44] questions feedback uh grievances uh in a [9:47] timely manner. There are sort of SLAs's [9:50] uh from that standpoint. So having the [9:53] right of uh right set of technologies to [9:55] support us uh in in doing all of that [9:58] also ties back to having the right tech [10:01] stack uh from a data standpoint being [10:04] able to uh automate uh data ingestion [10:07] processes but also data governance [10:09] processes that involve cleaning uh [10:11] curation and surfacing for analytical [10:14] use cases. All of that is technically a [10:17] one single pipeline that can effectively [10:20] be uh subject to that same SLA that I [10:23] was talking about. So uh definitely very [10:25] important I think u I in the banking [10:28] sector we I would say we specifically [10:30] choose solutions that help us stay [10:33] compliant in that uh aspect. Having [10:36] having vertical integration definitely [10:38] helps. um you would definitely opt for [10:40] an ecosystem that is as best fragmented [10:43] as possible. Uh like Santos was talking [10:46] uh there's a lot of open- source tools [10:48] available for use which which can [10:50] definitely help you build a pipeline [10:52] that supports all of these needs. Uh but [10:54] also the latency aspect is very [10:56] critical. um having uh systems that are [10:59] not just functional but also fault [11:02] tolerant to a very high extent and also [11:05] uh can run uh within a certain tech [11:08] stack without having to manage the data [11:12] ingress and outgress out of different [11:13] systems is very critical. So uh those [11:16] are the things we prioritize within the [11:18] banking space in order to meet these [11:20] requirements. [11:21] >> Got it. So Tajinder and in relation to [11:24] what Santosh and Rahul had explained how [11:27] have the data governance processes and [11:30] supported through uh these data [11:32] transformations and what are some of the [11:34] tools you use to support those pipelines [11:36] >> right so previously I think the [11:38] governance was not that key elements [11:41] long back right but recently the [11:43] governance the way the AI and machine [11:45] learning and that especially starting [11:47] from the big data leaks is taking a [11:50] curve over here And if we are not [11:53] following the standards, I think we we [11:54] can be in a big trouble, right? And [11:56] that's specifically when a privacy acts [11:58] comes in, whether it's from a GDPR [11:59] perspective or CCPA perspective, we need [12:02] to make sure that what is are we really [12:04] compliant enough? So we need to take all [12:06] the required actions in regards of that. [12:08] Now how do we do that? Right? I think [12:10] the teams basically they want the data [12:12] as soon as possible. They want it right [12:15] now whenever they look into the data [12:16] catalog. And the thing is like how [12:19] exactly how fast can you give this data [12:22] to the right people. So from a [12:24] governance perspective I think uh the [12:26] any of the governance team members can [12:27] agree with me. We never say that we are [12:29] not going to give the data right. But we [12:31] want to make sure that we give the right [12:32] data to the right people for the right [12:34] use case. I think unless the use case is [12:36] specified for that particular uh product [12:39] it's very difficult for us to handle the [12:41] data. Now when the data comes in the AI [12:44] machine learning perspective, right, it [12:45] flows in tremendous velocity, variety [12:48] and everything. How exactly can you make [12:50] sure that you are giving the right PI [12:52] elements to the right teams? I think [12:54] that's where this rulebased uh policies [12:57] integrated with the column uh column [13:01] masking as well as the role level [13:02] implementation. So it goes all the way [13:04] to a cellular level access, not just a [13:07] object level or a column or a table [13:09] level, right? And how do you do that? [13:11] Basically we use the AML machine [13:12] learning algorithms to do that like it [13:14] can easily the moment the data gets [13:15] landed into your ecosystem. You [13:17] immediately get it tagged where exactly [13:19] the SSN number resides where exactly the [13:21] social security resides the credit card [13:23] information is there a date of birth any [13:25] of the PI elements you train them train [13:27] the model in such a way that it can [13:29] easily identify where the data resides [13:30] and what what risk can it be for the end [13:34] users as well as the whole [13:35] organizations. So the moment you comes [13:38] you can tag it you can provide the right [13:40] risk access to the right people. Also [13:43] important thing to note over here is [13:44] when you do this data governance you [13:46] should be knowing the lineage of the [13:47] track like where's the source where the [13:49] data is coming from is it a right source [13:51] because at the end of the day the end [13:53] user has to trust whether this is the [13:56] right data product he can trust on is [13:58] there any kind of a like as I said like [14:00] the data preparation too right is there [14:02] more null values if they're getting a [14:04] table which has like an 80% of null [14:06] values it doesn't make any sense for [14:07] their models so they want to make sure [14:09] that the data product which we are [14:10] building for them is more trustworthy [14:12] they can rely on it. They can really uh [14:15] have their model output as expected. [14:18] >> Do do you have like tools for capturing [14:20] the lineage? Yeah, we basically I think [14:22] there are a lot of governance tools [14:23] which are there right now in the market [14:24] say especially Calibra Alian as they one [14:27] of them and I think all the cloud [14:29] platforms they have their inbuilt uh [14:31] tools which you can use as especially in [14:33] the Google's there's a very good tool [14:34] called data plex uh that basically gives [14:36] you the whole lineage uh it's basically [14:38] goes on the tableto table level right [14:40] now but I think they're building it for [14:41] the column level uh lineage too and on [14:44] regards of that on top of that I think [14:46] the important thing is the logs right [14:48] you need to track the logs observability [14:50] where exactly who is using that [14:52] particular table for what use case is [14:54] the use case aligning with that how much [14:56] data is he pulling something which is he [14:57] was not authorized to pull it into it [14:59] right so all that kind of a logs can [15:01] really help you out is there some other [15:02] team members who is not a part of that [15:04] team but he is trying to access that [15:05] data so I think all that no I think most [15:08] of the cloud providers they build that [15:09] kind of a logs inside that but on top of [15:11] that definitely uh you can get into [15:13] third party tools to to get the logs [15:16] >> so so what I'm hearing there is uh [15:18] evolution of the data contract data [15:20] access contracts [15:23] >> and so Santosh so data science teams [15:26] often have the pressure to deliver [15:28] insights quickly. So how do you balance [15:30] the need for thorough data preparation [15:33] with the demands for speed and agility [15:36] and have you developed any shortcuts or [15:39] best practices that help maintain [15:42] quality without slowing you down? [15:44] >> Yeah, it's it's always a balancing act, [15:46] right? uh so I would say in the initial [15:49] phase of the project the data prep [15:51] pipeline they need not be perfect uh it [15:55] can be an iterative process it should be [15:57] good enough to generate insights and [16:00] guide next steps another thing that [16:02] helps is if you have uh modules [16:05] modularized codes that you can leverage [16:07] from different projects in your [16:09] organization for example when it comes [16:11] to building a data prep pipeline you [16:13] would be uh running similar kinds of [16:16] steps to address missing values, [16:19] outliers, uh removing noise from your [16:21] data. Uh so having modulariz modularized [16:25] uh code helps uh like everything LLMs [16:29] have made their way into the data prep [16:31] part of the stack as well. So uh [16:34] leveraging uh coding assistants like uh [16:36] GitHub copilot, it can improve the [16:39] productivity of the data engineer who is [16:41] building the pipeline. uh it can be used [16:43] to leverage it can can be leveraged to [16:46] uh write code uh to perform trivial task [16:49] like for example getting data from uh S3 [16:53] to snowflake and and a really [16:55] boilerplate trivial code can be [16:57] generated while the data engineer can [16:59] focus on solving core business logic and [17:03] also it's important to create a culture [17:06] and uh understanding with the [17:07] stakeholder that uh data prep is not a [17:11] oneandone [17:12] part of a process. It's an iterative [17:14] process. It should be monitored and it [17:17] should evolve uh through interaction [17:19] with the business. If there is any [17:20] business logic change or change in [17:22] requirement that you should be able to [17:24] integrate that seamlessly in your [17:26] pipeline. [17:27] >> Yes. And you brought up the iterative [17:28] processes. So how many iterations Rahul [17:31] now this question is to you. So how many [17:32] iterations do you go through before an [17:35] optimum stakeholder ac optimum state for [17:39] stakeholder acceptance is achieved? [17:41] Yeah. Uh thanks for asking that [17:43] Chaitnam. I think this this really is [17:46] highly subjective. Uh given any data [17:50] science model, you would have uh [17:51] different complexities that come with [17:53] it. Uh some models uh if they're easy [17:57] and you're expected to uh deploy the [17:59] model, complete the build, deploy the [18:01] model within uh a one-mon time frame, [18:04] then you would expect to go through [18:05] maybe two three iterations of the model [18:07] from the reference baseline that you [18:09] built initially. Um or if it's a super [18:12] complex model, something that [18:14] specifically needs a lot of stakeholder [18:16] buyin and uh approval of uh you you can [18:20] easily expect maybe five or six or seven [18:22] iterations of that model as well. It [18:24] really depends. Uh but going back to [18:27] what we were initially discussing as [18:28] well, I think the iterations that we [18:30] take to build models, they're always [18:33] highly subjective to the data quality [18:35] that's available as well. Very very [18:37] highly subjective. uh what mother is [18:39] saying uh garbage in garbage out. If you [18:42] if you have data that's not even ready [18:43] to build that model that you set out to [18:45] build today uh no matter what you do you [18:48] boil the ocean uh you still are nowhere [18:50] close to building your model and [18:52] deploying it to some useful uh business [18:54] solution or problem statement [18:56] eventually. So uh and tying back to what [18:59] Tajjinder was saying as well uh having [19:01] quality metadata always definitely helps [19:05] uh but curating data that's good enough [19:08] to build models from the onset that can [19:10] help you reduce time that it takes to [19:12] build the models. I would even go one [19:14] step further u this is something that I [19:17] uh have benefited a lot from uh private [19:20] uh or personal practice and experience [19:22] as well. Not just having gold standard [19:25] data that's cured enough to build [19:27] models, but also having uh a feature [19:29] store uh of uh predictors or features. [19:34] In machine learning, we call them [19:35] features. Some of you might hear of them [19:37] as predictors and variables. having a [19:40] feature store that's readily uh [19:42] aggregating a bunch of insights from [19:45] your data store um and having them [19:48] readily accessible for data scientists [19:50] to build baseline models can also go a [19:52] long way uh in helping you cut down time [19:55] uh it might help you cut down time uh [19:57] for the initial build the reference [19:59] build that I was talking about but it [20:01] can also help you cut down time in the [20:03] future iterations especially if you [20:05] decide to pivot this happens a lot in [20:07] banking as well um that's where metadata [20:09] data is useful as well. We uh we try to [20:12] use data that is not uh predatory in [20:15] terms of predicting uh predictive [20:17] features uh to build models. So any [20:20] predictive outcome, it has to use the [20:22] right state right data to be able to [20:24] predict the eventual outcome. If you're [20:26] using the wrong data that could be [20:27] looked upon as uh predatory or just [20:30] plain wrong for use of that uh you know [20:34] model, you don't want to do that. So [20:36] having the right metadata helps you do [20:37] that. Having the feature store uh [20:40] readily available helps you pick the [20:42] right features that can speed your model [20:45] deployment uh and build timeline and [20:47] therefore lead to more productive [20:49] outcomes. [20:51] >> You have something to say? [20:52] >> Yeah, I just wanted to add on like I [20:54] think it's it's a fact that uh denoising [20:57] and taking time to improve your data is [20:59] going to slow you down but it is going [21:01] to slow you down in the short term not [21:03] in the long term. It's going to give you [21:05] longer impact. So it's it's like almost [21:07] comparing do I want to you know ship [21:09] like three models really quick with like [21:11] really bad data or should I spend a [21:13] month you know dnoising removing [21:15] outliers you know doing some cleaning [21:17] strategy and then build a model for the [21:20] long term that would be higher impact [21:23] than you're doing uh then you're [21:24] launching a model without all the due [21:26] diligence and this this is almost like [21:28] you know like Santo said it's a cultural [21:30] thing I think leaders company leaders [21:32] should understand that this is an [21:33] investment area and you have to incentiv [21:35] vise your employers to spend time [21:38] building data. Like all of us are [21:39] incentivized to ship features and you [21:41] know like launch models really quickly [21:44] but are you incentivizing you know [21:46] employers to actually spend time [21:47] cleaning up the data and removing the [21:49] garbage and you know making it high [21:51] quality right [21:54] so Tajinder then speaking about the [21:56] feature store here so do you think that [21:58] there is a shift in data preparation [22:00] practices where data engineering is able [22:02] to pick up some of those tasks in [22:05] cleaning up the data or even exploring [22:08] those initi [22:09] data points. [22:10] >> That's a very great question. I believe [22:12] and I definitely agree with this because [22:14] if you see right previously all the data [22:17] science teams they were leveraging their [22:18] own notebooks and they were looking into [22:21] this uh pre- data analysis and how the [22:23] outliers are then they are trying to [22:24] bring build the features to and then [22:26] they are trying to build the model. So [22:29] for one particular source or one [22:31] particular data product it takes like [22:32] you know different teams are doing this [22:33] in silos and that's basically adding a [22:36] cost as well as a time when you shift [22:38] that model from a data science [22:40] perspective directly integrating with [22:42] the pipelines you are basically saving [22:44] the time and you are basically [22:46] optimizing with 3x or 10x speed over [22:48] there. Now how exactly before like uh [22:50] previously the profiling results or [22:52] whatever they used to do basically [22:53] resides on the notebook. Right now what [22:56] the shift is coming is like you know [22:58] before the pipeline even triggers it [23:00] looks into those kind of outliers. It [23:01] looks into those kind of a standard [23:04] deviations values or all the standards [23:06] values spec statistics values I'll say [23:09] and then it basically filters down or [23:12] basically pushes the right data into the [23:14] right pipeline right and when it comes [23:17] in regards of the other thing is the [23:18] feature store right the feature store uh [23:20] or and the standards the two two [23:23] important things so when you build say [23:25] one team has built a data science [23:26] product that's based on some different [23:28] data type and then other team has built [23:30] on some other different data type Right. [23:31] So basically the standardization across [23:33] the uh company or the organization is [23:36] not there. Right. So it basically [23:38] integrates when the data engineering [23:39] sets the standards with the governance [23:41] integration over there that okay this is [23:44] the datetime format. This is what uh [23:46] customer ID for a company should look [23:48] like and we are getting into integer [23:50] values or string values and whatn not [23:52] and it gets multiply when it goes to a [23:55] complex data types right. So the feature [23:57] store can directly integrate into the [23:59] table as a values over there. so that [24:01] the data science team can spend their [24:04] value towards the model output rather [24:07] than playing again and again with the [24:08] data to get the right standard and the [24:10] get the right uh feature integrated over [24:13] there. [24:14] So today in your experience at what [24:17] point after data preparation with the [24:19] current methods that you are seeing in [24:21] the industry at what point do you think [24:24] data engineering team should engage with [24:26] the data science teams in supporting [24:28] some of these processes? [24:30] Well uh I have seen organizations when [24:32] before they create a data product right [24:34] the very first thing they include all [24:36] the stakeholders and when I mean with [24:38] stakeholders they include all the data [24:39] science teams analytical teams even the [24:41] machine learning leaders over there that [24:43] this is what the data product we are [24:44] going to build up and these are the [24:46] features and this is what the data we [24:48] required from are we so data governance [24:51] data engineering every all the different [24:52] teams get together and set up a meeting [24:55] and in that meeting they decide all the [24:58] key patterns right what data sources [24:59] they're going to bring is the [25:00] standardization what key metrics they [25:01] are going to bring what is the main [25:04] thing is what this model is going to [25:05] predict is that aligning with all the [25:07] standards across the across the company [25:09] right so all that thing waiting really [25:12] happens and this does not happens like [25:13] in a one go like let's say this is a [25:15] very pre pre-task but I think the main [25:18] uh point comes when you launch with the [25:21] first version of the MVP uh minimal [25:24] viable product that's where the data [25:25] science teams really starts indicating [25:27] whether is this the right data product [25:29] is are they getting the right values out [25:30] of it. Are they getting the right [25:31] features over here? Uh is the model [25:33] predicting as per their uh MVP as a very [25:35] first go and this is not a one-time [25:38] process. This is iterative process. So [25:40] data science team whenever there's a [25:41] change with the additional data sources [25:44] or there's a change in the columns or [25:46] everything. I think the data science [25:47] teams has to closely integrate with the [25:49] data science governance and the data [25:51] engineering to make sure that everything [25:53] is up to the mark. [25:55] >> Can you shed some uh light on your [25:57] experiences? [25:58] >> Yep. Yep. I can add to that. So uh from [26:00] my experience working in banking, [26:01] there's often a lot of vendor platforms [26:05] that you're using to support different [26:07] uh platforms across the enterprise. Uh [26:09] you could be using one system for [26:11] mortgage, you could be using one system [26:13] for credit card processing, you could be [26:15] using something else for consumer loans [26:17] and things like that. So from a uh from [26:20] a data ingestion standpoint, right at [26:22] the onset, what the gender was saying [26:24] makes perfect sense. having everyone [26:25] involved from the onset is key because [26:29] uh what I've seen often happens because [26:32] uh this is key uh that's why I mentioned [26:34] the vendor standpoint. You're bringing [26:36] in data from different vendor platforms [26:38] uh and I'm supposing it comes into a [26:41] lakehouse that's integrated uh and then [26:44] is the platform that data science and [26:45] analytical use cases are built on top [26:47] of. when you're bringing those there are [26:50] often a lot of subjective choices that [26:52] are made uh to bring in data elements uh [26:56] in the shape or form but also uh [26:59] sometimes they just get omitted uh if [27:01] you if you don't have data science uh [27:04] partners and other partners as well [27:06] involved in the process there is a high [27:09] chance uh and I've seen this many times [27:11] there is a high chance that some very [27:12] high quality uh data elements might be [27:15] dropped thinking that they're not useful [27:17] for downstream use and you just lose the [27:20] potential of uh some kind of predictive [27:22] outcomes because you didn't bring them [27:24] into your lakehouse uh for use. Uh [27:27] another thing uh that comes to mind is [27:29] um not just bringing in the right [27:31] elements uh rather than dropping them. [27:33] But there's also a very uh good chance [27:36] of conducting transformations on the [27:38] same elements while you're bringing them [27:40] into your lakehouse. If you do those [27:43] transformations in transit to the [27:45] lakehouse, it saves time downstream for [27:48] any analytical uh counterparts uh to [27:51] then having to repeatedly do them over [27:54] and over again for building dashboards [27:57] uh for building views as well as data [27:59] science models. So you're you're really [28:01] saving a lot of time if you have [28:02] stakeholder uh buy in and uh viewpoints [28:06] being expressed in the ingestion stage [28:09] uh as a part of the engineering [28:10] pipeline. saves a lot of time and effort [28:12] for everyone down the way. [28:14] >> Right. So, uh stream processing suggest. [28:17] >> Yes. [28:17] >> Right. Okay. Um so, uh Madra, what have [28:20] been some of your experiences [28:24] in engaging with different stakeholders [28:27] when working with data or cleaning or [28:29] shaping the data. [28:30] >> I think the most important like meta [28:33] concept is that data itself. Like every [28:36] single time I walk into a meeting with a [28:38] leadership to convince them to make a [28:39] decision is always data right like for [28:42] example like how do you convince your [28:44] team to invest in you know improving the [28:46] data run an AB experiment with like you [28:49] know like v0ero version of the training [28:51] data and then work on you know some [28:53] strategies like active learning or other [28:54] strategies and then launch another model [28:56] like v1 and then you can see the [28:58] difference in the in the in the offline [29:01] evaluation in like model precision and [29:03] recall and then that is Right. Cuz [29:05] you're like going to use that to make [29:08] every decision to convince every [29:09] stakeholder on how critical you know [29:12] this is. Yeah. [29:13] >> Right. And observability is an important [29:15] aspect of that because you are um [29:18] monitoring the data drift and also the [29:20] model performance drift over time. [29:22] >> Yes. And and then and it keeps changing [29:24] like models always almost always almost [29:27] always decay because the real world [29:29] changes. So what you test and ship is [29:31] not going to be the same model like a [29:33] month later cuz all the patterns that [29:35] the model is seeing is almost changing [29:36] every every day. So yeah. [29:40] So looking ahead so what trends or [29:42] innovations do you see emerging in data [29:45] preparation that could change how we [29:47] approach this critical part of the [29:50] pipeline? This question is to Rahul and [29:52] Tajinder. [29:52] >> Yeah, happy to go first on this. Uh I [29:54] think uh with with the dawn of I sit in [29:57] strategy so I get to see this a lot. uh [29:59] discussions about this with the dawn of [30:02] LLMs being able to uh automatically form [30:06] uh contextual understandings of things. [30:08] Uh that applies a lot to data [30:10] engineering pipelines today as well. [30:12] Some of the stuff we were talking about [30:14] um you might want to have metadata [30:16] tagging that is accurate in order to [30:19] have downstream usage that is uh [30:21] realizing the full potential of your [30:23] data. Well, metadata tagging it was [30:25] being done manually before. LM can do [30:28] that today. uh things like that uh you [30:30] know right from metadata tagging but [30:32] also to setting ground rules for data [30:35] transformations data cleaning data [30:37] treatment uh in the data engineering [30:39] pipelines uh we're seeing a lot of that [30:42] uh come through as a part of uh both uh [30:44] vendor products platforms but also [30:47] homegrown solutions so I think uh that's [30:49] that's one key interesting trend I see [30:52] coming in uh less focus on some of the [30:56] repetitive um some of So very boring [30:59] honestly uh things that data engineering [31:01] teams would have to do every time you're [31:03] ingesting a new data source uh or a [31:05] system of record less focus on that [31:08] because you can have LLMs do that the [31:10] process doesn't go away don't get me [31:12] wrong uh you would still have some [31:14] someone from your data engineering teams [31:16] validating what the LLMs are uh tagging [31:19] data elements and uh tables for uh but [31:22] yeah that that should significantly cut [31:24] down time in terms of u bringing new [31:27] data systems on board bringing new [31:29] tables, databases uh and realizing their [31:32] potential faster [31:34] >> and working with LLMs. Clearly, [31:35] schematic representations also become [31:37] very important because LMS depend on [31:40] those schemas in order to tag and [31:43] process that information. Right. Right. [31:45] Okay. [31:46] >> More importantly, what this LMS can do [31:48] is uh finding an outliers sometimes, you [31:51] know, uh exploratory through graphs and [31:53] everything. I think the LLMs are doing a [31:55] greater job in finding this outliers and [31:57] especially all the data quality checks [31:59] right uh because sometimes when you see [32:03] uh when you go into the stats model [32:04] right there is a correlation between one [32:06] product versus other product as a human [32:09] it's very difficult uh to find maybe you [32:11] can find with one or two attributes or [32:13] maybe 10 attributes based on how how how [32:15] busy your day is right but when you look [32:17] into across the whole portfolio on an [32:20] org level there are thousands and [32:22] thousands of tables and especially When [32:23] it goes into uh classification with the [32:26] domain modeling coming the picture, it [32:29] really depends upon how exactly are this [32:31] domain interacting like let's say you [32:33] have a customer domain, you have a [32:34] vehicle domain and you have a sales [32:36] domain, right? How exactly are the [32:39] machine learning models interacting with [32:41] only with the sales domain what is the [32:42] correlation between the sales versus the [32:44] customer? you might have find five [32:45] attributes but when you give it to the [32:47] LLM I've seen like you know the Google [32:49] has building a new uh new feature around [32:52] this and what it does is basically it uh [32:54] looks into your different uh data sets [32:56] level and find the correlation among [32:58] these tables and it gives you the [33:00] outputs and it can give you the [33:01] questions like what exactly uh can this [33:03] question answer and also the results [33:04] based on that but it does not scales [33:06] only at the data set level it goes [33:08] across the project level and when you go [33:10] in the project level basically you can [33:11] integrate across all the Google uh [33:13] projects or all the GCP projects inside [33:15] your org and basically you can interact [33:17] with one uh how the how how can you [33:19] correlate between one project versus the [33:21] other project and you can get the right [33:23] results and also on top of that what you [33:25] mentioned uh I believe uh the important [33:27] trend which is going towards the data [33:28] governance and the data engineing [33:31] framework is more with uh data [33:35] data mesh architecture that's a key [33:38] aspect if you want to look into it it's [33:39] a very good architecture which is coming [33:41] and lot of orgs are moving towards does [33:44] some it depends upon use case to use [33:45] case. Maybe it's not a use good fit for [33:48] smaller companies but big orgs [33:49] definitely find a lot of value with data [33:52] mesh and also there is data fusion also [33:55] is one of them. So when you look into [33:57] this they call data as a product right [33:58] each and every data they call it as a [34:00] product and they do a domain [34:01] architecture around that and make sure [34:04] that the not just the governance team [34:06] owns it basically uh the ste who is a [34:09] domain owner who is a theme for that [34:11] data he he also is has ownership on that [34:14] data right so it's not a central [34:15] governance it's basically you are uh [34:17] spreading the governance across all the [34:19] different data engineers and the uh [34:21] business business team members too right [34:24] how it works is basically You don't even [34:26] have to basically move the data from one [34:29] location to another location. What you [34:30] do is basically you just integrate a [34:32] logical layers across your whole or can [34:35] basically charge uh change it across the [34:37] domain modeling or whatever the data [34:39] sets you have built up across and [34:41] basically let's say that was the way I [34:42] giving example right there's one like a [34:45] uh sales or sales organization so you [34:47] call it as a sales domain and all the [34:48] sales attributes or the sales tables [34:50] basically follow that and anyone who [34:52] gets access to the sales like he can get [34:53] access to the sales with the right right [34:56] row level and the column level security [34:57] but when it goes to when you scale this [34:59] across Ross multiple domains you can [35:01] really build up the whole data mesh [35:03] architecture in much more uh useful use [35:06] cases and you can build the use cases [35:07] which can be a onetoone data [35:09] observability for the end users as well [35:11] as for the whole uh or or level [35:15] thank you [35:17] >> yeah um I know it's like pretty in the [35:19] weeds but a little highlevel trend that [35:21] I'm seeing is the use of synthetic data [35:23] I don't know if you recently saw Elon [35:24] Musk talk to you know like the Y [35:26] cominator president Gary where he said [35:28] that we are technically running out of [35:30] you know like training data and and and [35:32] I there's also like nature did a [35:34] research that by 2028 you're not going [35:36] to have enough human human trained or [35:39] human written training data like [35:41] internet is basically running out of [35:42] data so like synthetic data is like some [35:44] strategies that is like propping up like [35:46] a lot especially in like autonomous [35:48] driving it's kind of common but even in [35:50] other fields where you know you want to [35:52] add more representation into the models [35:54] you're like starting to use gen AI and [35:56] other technologies to synthetically [35:58] produce is you know like training data [36:00] for for the models. Yeah, that's what I [36:02] wanted to add. [36:03] >> Thank you. So um again in the interest [36:06] of time let's do a round robin on this [36:08] question. So Santosh, starting with [36:10] Santosh, what is the most overlooked [36:12] factor when trying to scale data science [36:15] uh from pilot a project from pilot to [36:18] enterprisewide adoption? [36:21] I [36:21] >> would say one of the most overlooked [36:23] aspect is I know we talked about order [36:25] of tools and framework but the human [36:26] element as well. Uh when it comes to [36:29] cross collaboration between teams, it's [36:31] important to have uh shared uh practices [36:34] and guidelines to build highquality data [36:37] pipelines. If you might you might have [36:39] the best tech stack but if there's [36:41] inconsistencies in the data prep [36:43] strategies across team members or teams [36:45] when you scale it you're going to [36:47] magnify those uh inconsistencies and [36:50] that would lead to a lot of rework. [36:51] That's my opinion. [36:53] >> Well said. [36:55] >> Agreed 100%. U I I will add to that [36:58] actually u silos not just between people [37:02] but silos between data. Uh I think when [37:05] it comes to building data science [37:06] models, what I've often seen is uh [37:09] there's there's not enough predictive [37:11] signals available uh at time of building [37:16] that could have helped model building be [37:18] a success and they're often realized [37:21] later on and thus have to be version two [37:24] version three iterations of the same [37:26] model after it goes to deployment. So I [37:28] would say the most overlooked factor uh [37:31] from a data science perspective is uh [37:34] using the right data and having the [37:36] right data available out of silos made [37:38] available to everyone across the [37:40] enterprise no matter how large your [37:42] enterprise is having that signal made [37:44] available for eventual success [37:48] >> uniformity and you're I guess you're [37:50] also relating to the semantic layers and [37:51] as you mentioned a logical layer there. [37:53] >> Yep. [37:54] >> Okay. And on top of that I will say the [37:56] certified data right so I believe [37:59] there's lot of data you know sometimes [38:00] they come they just do for testing and [38:02] validation but when it really goes from [38:04] the raw data all the way to the end [38:06] users how much of a data is really [38:08] certified and the certified data when I [38:10] say is basically it goes through [38:12] multiple different steps that itself is [38:14] a bigger topic like how exactly you put [38:16] the governance how exactly you put the [38:17] data engineering pipelines and also what [38:20] exactly uh do you call it call it from [38:22] the co perspective what elements can we [38:24] from the CD perspective. So all that [38:26] integrates and uh once you have a [38:28] certified data product I think that's [38:29] where the trust workers as a data prep [38:32] and all that really plays a big picture [38:34] on a whole. Thank you. [38:35] >> So through the semantic layer [38:37] implementation you are also creating a [38:40] trusted data layer off of your raw [38:43] layer. [38:44] >> Yes. So from not from the raw layer [38:46] perspective but from like you know raw [38:48] might be not a good one for the data [38:51] scientist team but because it's not yet [38:53] curated right but again it goes in the [38:55] transformation layer so finally building [38:57] the data product now when this data mesh [38:58] architecture which I was talking about [39:00] doesn't means that your data product the [39:02] final certified data product is has to [39:04] be in the final layer the the certified [39:07] data can be in the raw layer because [39:08] that's where exactly let's say all the [39:10] collector vehicle data that's coming [39:11] directly in the stream right that's [39:13] directly being leveraged by the team. So [39:15] how how do you call not that a certified [39:17] because that's a right source of truth. [39:19] You know who the SM is, you know the [39:20] domain practices. So all that elements [39:22] really add to it and that's where you [39:24] call it as a certified data product. [39:26] >> Awesome. Thank you very much for all the [39:28] responses. Yes, Madra, would you like to [39:30] add some insights there? [39:31] >> Yeah, like for blind spots, I think the [39:33] first thing that comes to my mind is [39:34] actually metrics, especially when you're [39:35] taking a product from PC to enterprise. [39:37] I I talked to a lot of small businesses [39:39] and it's it's it's [39:42] very interesting that nobody talks about [39:43] metrics, you know, like when you're [39:45] defining a PC and you want to take it to [39:47] enterprise, you should have like a suite [39:49] of hierarchy of metrics to say, okay, [39:51] you meet this and then you take this [39:53] product to to enterprise. So, it's like [39:56] defining those metrics at the model [39:58] level, but also at the production level [40:00] and and having a target and quantifying [40:01] how you're moving, I think it's kind of [40:03] overlooked, especially, you know, when [40:04] you go to like smaller businesses. Yeah. [40:07] Awesome. Thank you. And thank you uh [40:09] panel members. So now we will u solicit [40:13] questions from the audience. [40:16] >> My name is Lara Guridge and uh this is a [40:18] topic that is really of so much interest [40:20] to me and thank you for all of you and I [40:23] so much enjoy the insight the sharing. [40:26] But one critical thing that was missing [40:28] in this conversation that I didn't hear [40:30] anyone say well some implied but not [40:34] highlighted is stewardship [40:37] stewardship [40:39] this is critical the how the how to get [40:43] those 70% of time we spend in this data [40:47] prep to reduce that it has to start [40:51] really from the upstream [40:55] building those stewardship along those [40:57] data pipeline is critical and I was [40:59] waiting for you to say that. No, I I [41:01] think I I I I said the domain ownership, [41:03] right? So the domain ownership is [41:04] nothing but the stewardship what I was [41:06] reflecting to. It can it can be the data [41:08] engineers, it can be the data owners, it [41:10] can be the data domain owners, right? [41:11] Because they know what that product is [41:13] and how much is that a value for and is [41:16] it is that coming from the right trusted [41:17] sources as I said is it bringing the [41:20] right attributes like as I said not a [41:22] lot of null values and all that stuff. [41:23] >> Exactly. Otherwise we're going to [41:25] mechanize the all the way through and [41:27] that hasn't served us all the way for [41:29] years. So the only way we can of kind of [41:33] stop the wackamo game of keep redoing [41:35] that is really to bring the literacy [41:38] around litership stewardship around the [41:41] whole ecosystem of the data in every [41:43] organization. Thank you. [41:44] >> I would add to that I would just say [41:46] quickly uh it's not enough to have just [41:48] one data steward tied to uh a data [41:51] source uh or a data set. You would [41:54] ideally want to have uh one technical [41:56] person and one business person. both [41:58] together would probably make the most of [42:01] that data set rather than a single [42:03] person they might get overburdened with [42:05] the responsibilities. Yes. [42:07] >> Yeah. And it's a collective [42:09] responsibility. Yes.