Retail, FinTech AI Teams Overhaul Pipelines to Stem Model Drift

零售与金融科技AI团队重构流水线以遏制模型漂移 数据工程推动了80%的实际AI应用成功，需要严格的数据契约、模块化架构及领域专家协作，以防生产级流水线在企业级规模下发生故障。 业界与学术界专家探讨了Lambda、Kappa及现代湖仓等架构框架，以支持零售、金融科技和政府领域高要求的AI应用。专家组详细阐述了关键优化策略，包括存算分离、采用Parquet等列式格式，以及将繁重的模型训练转移至非高峰期批处理作业以大幅降低计算成本。针对复杂的多模态与传感器数据，演讲者提倡采用专用提取子流水线、晚期融合、模式注册表和自动化统计异常检测。专家们最后强调，AI约80%的成功取决于稳健的数据工程，这需要清晰的数据契约、端到端数据血缘以及与领域专家的直接协作。 拥有成熟数据流水线的机构在AI采纳率和整体投资回报率方面，跑赢行业同侪的概率约为后者的2.5倍。 (0:04) Lambda架构对企业AI依然至关重要，它通过维护双重流水线发挥作用：用于长期历史模型训练的分布式批处理，以及用于即时个性化推荐的实时流处理。 (2:21) 受监管领域与国防AI部署依赖混合架构，将基础敏感工作负载保留在本地专用GPU上，同时按需突发扩展至云端数据湖以控制运行算力成本。 (6:09) 实现存储与计算分离，并将繁重的模型训练转移至夜间非高峰期批处理作业，可削减高达30%的云计算开销。 (7:56) 相比扁平的CSV或JSON文件，将数据存储标准化为Apache Parquet或ORC等压缩列式格式，可防止查询扫描成本失控，并大幅降低流水线的存储占用空间。 (11:29)

According to McKinsey research cited by moderator Chetana Katpally, organizations with mature data pipelines are approximately 2.5 times more likely to outperform peers in AI adoption and return on investment. Panelists argue that sustaining this advantage requires focusing on upstream data engineering—which accounts for roughly 80% of operational effort—through modular pipeline architectures, strict schema governance, and close alignment with domain experts. Architecture Selection and Hybrid Infrastructure (2:21) Shiva Kumar Ramavath (Albertsons) explains that managing varied business workloads often requires a Lambda architecture to separate batch and real-time processing paths. In retail operations, historical sales records spanning four to six years are processed via Apache Spark for daily or monthly demand forecasting, while customer-facing interactions at mobile app login rely on real-time event streaming powered by Apache Kafka. Arjun Mantri (Varo Bank) adds that modern data platforms increasingly favor lakehouse designs and Change Data Capture (CDC) over raw object storage like AWS S3, enabling organizations to consume and normalize diverse data formats without rigid pipeline constraints. In government and defense environments, pure cloud solutions are often unfeasible due to regulatory mandates and compute costs. Hardik Gohel (University of Houston) describes a hybrid architecture utilized across federal projects: two dedicated on-premise servers equipped with NVIDIA Tesla GPUs handle local deep learning iterations, while burst workloads deploy to on-demand cloud instances and data lakes that are terminated immediately after execution to control expenses. Workload Decoupling, Storage Compression, and Cost Management (7:56) Balancing pipeline latency and throughput against operational costs requires strict workload segregation. In Department of Energy geophysical anomaly-detection initiatives, Gohel notes that isolating real-time cloud stream inference from heavy deep learning model training—and shifting training to nightly batch jobs—reduced compute costs by approximately 30%. Mantri and Ramavath recommend decoupling compute from storage to scale resources independently, while modularizing ETL pipelines across domains to prevent single-point failures and isolate query expenses. Storage and orchestration decisions also determine pipeline efficiency. Mantri cautions that unoptimized query engines like AWS Athena charge by the volume of scanned data, making poor joins and unpartitioned tables primary drivers of runaway costs. Transforming data directly in Apache Spark via Apache Airflow orchestration and standardizing on compressed columnar formats like Apache Parquet or ORC significantly lowers storage footprints and query execution times compared to raw CSV or JSON files. Bridging Prototype Gaps in Multimodal Systems (13:15) Transitioning AI models from experimental prototypes to multi-terabyte production systems frequently exposes structural data discrepancies. Gohel highlights a Department of Energy HVAC and IoT energy optimization deployment where a model trained successfully on a clean proof-of-concept dataset subsequently failed in production; multi-terabyte live streams introduced unexpected ingestion variations and unmodeled schema drift that triggered erratic predictions. Handling multimodal datasets—such as combining product text, images, and IoT sensor streams—requires dedicated sub-pipelines before consolidation. E-commerce workflows must extract semantic context through tokenizers and embeddings (such as Word2Vec or BERT) and isolate visual features via computer vision extractors. Downstream consistency is maintained by assigning synchronized timestamps, unique entity identifiers, and schema registries, culminating in late fusion techniques that merge disparate vector embeddings into unified representations for inference. Automated Data Quality, Drift Detection, and Lineage (22:33) Because client stakeholders frequently provide messy source data without investing in pre-ingestion cleaning, pipelines must embed automated statistical models and imputation workflows. These validation gates systematically purge duplicate records that could distort model weights and address null values before data reaches production applications. To guard against ongoing environmental shifts, teams configure automated statistical boundary checks—such as flagging inputs deviating beyond ±3σ from an expected normal distribution—and establish regular model retraining cadences. At enterprise scale, where single database tables reach terabytes or petabytes, traditional partitioning and indexing are insufficient for troubleshooting silent prediction failures. Granular end-to-end data lineage is critical to track transformations back to the source without protracted cross-team debugging. Panelists advocate centralizing pipeline logs in tools like Kibana and leveraging native Databricks lakehouse observability to isolate ingestion failures from machine learning algorithms. Data Contracts, Incremental Development, and Domain Context (34:05) Establishing formal data contracts across engineering, product, and governance teams defines explicit expectations for schemas, data volumes, and federated access controls, preventing unexpected upstream updates from corrupting downstream models. When engineering pipelines for enterprise data repositories, Gohel advises starting development on small, representative data samples to verify schemas and filtering logic before progressively implementing complex feature engineering and ingestion workers. To translate algorithmic outputs into operational value, models deployed in mission-critical settings require explainability frameworks like causal modeling and counterfactual analysis. Non-technical leadership evaluates deployments by realized business value rather than raw accuracy scores. Furthermore, because data engineers frequently transition across specialized domains like defense, energy, and healthcare, panelists emphasize pairing engineers directly with veteran domain experts who understand the physical and operational signals required by the models. While pre-trained foundation models have lowered the barrier to neural network development, production success remains constrained by upstream data architecture. The panel concludes that sustainable deployments rely on starting with small data samples under explicit data contracts, though bridging the persistent gap between uncurated source data and strict production schemas remains an ongoing operational challenge.

麦肯锡研究显示，具备成熟数据管道的企业在AI落地成效与投资回报率上领先同行约2.5倍；与会专家一致指出，现代企业级AI管道的核心挑战已由算法模型转向底层工程，必须依靠解耦架构、严苛的端到端质量治理以及深度的领域协同，方能化解复杂多模态数据与生产规模带来的双重压力。 架构范式与成本效益调优 (2:21) 在应对高并发与海量异构数据时，解耦存储与计算的现代数据湖仓以及结合批处理与流处理的Lambda架构已成为主流工程选择。例如，Albertsons采用Apache Spark定期分析4至6年的历史销售数据以驱动需求预测模型的批训练与推理，同时依托Apache Kafka对移动应用登录事件实施毫秒级实时流处理，以支持即时个性化服务。在金融领域，变更数据捕获机制与基于对象存储（如AWS S3）的异构湖仓设计，有效消除了传统刚性管道的数据转换壁垒。 针对受监管与高算力开销场景，休斯敦大学在联邦政府项目中采用了混合部署策略：日常计算依靠两台配备NVIDIA Tesla GPU的本地服务器处理并行深度学习，当算力超限时再临时突发至AWS云端按需调用并在执行后立即销毁。在能源部（DOE）的实时地球物理异常检测项目中，团队将非紧急的重度训练转移至夜间非峰值批处理，直接降低了约30%的算力成本；此外，团队通过全面推行Apache Parquet或ORC列式存储压缩，并重构低效的SQL关联与数据分区，有效遏制了因AWS Athena等引擎按扫描量计费而引发的成本失控。 破解原型到生产的鸿沟与多模态对齐 (13:15) AI模型从实验室原型向工业级和政务级部署过渡时，最致命的工程瓶颈在于多源非结构化输入的数据对齐与模式漂移。休斯敦大学在承担能源部针对政府建筑暖通空调（HVAC）与物联网节能优化项目时发现，概念验证（PoC）阶段基于干净小样本构建的模型表现优异，但接入数TB级的生产实时流后，管道摄取差异与未建模的模式漂移直接导致模型输出完全失真。这表明开发团队必须在立项之初预判数据多样性，并强制原型与生产管道遵循严格的模式一致性。 对于融合文本、图像与传感器流的企业级多模态数据，现代工程通常为每种模态设立专属特征抽取流水线，例如通过BERT或Word2Vec生成语义向量，利用专用视觉网络提取稠密图像特征，并对物联网传感器数据打上时间戳与实体标识。各分支管道依托模式注册表（Schema Registry）或读取时模式（Schema-on-Read）实施结构校验，再通过后期融合（Late Fusion）拼装为统一的特征向量供下游消费，严格遵循前置数据清洗以规避“垃圾进，垃圾出”的技术债务。 端到端质量治理与可观测性体系 (20:15) 在企业与联邦科研真实场景中，项目各方通常难以主动承担源头数据清洗，这要求数据工程师在数据写入特征库前自主内建去重机制、空值拦截以及统计插值模型，防止重复记录扭曲模型权重或空值破坏下游消费应用。由于对PB级分布式表的所有指标进行全量血缘追踪在计算与运维上并不现实，团队必须采取基于投资回报率（ROI）的关键路径监控策略，在Amazon CloudWatch等工具中针对各阶段设置告警阈值。 随着表体量扩展至TB甚至PB级别，常规的分区与索引已无法解决隐性预测故障的溯源问题。工程架构需内建统计边界监控，通过追踪特征分布偏离正态分布或标准差超出正负3倍标准差（±3σ）来预警输入异常，并据此触发自动化重训周期；配合Databricks原生追踪或Kibana集中日志面板，构建贯穿原始摄取、特征工程到最终推理的完整数据血缘与可观测性闭环，能够大幅缩减故障排查耗时与跨部门沟通摩擦。 数据契约、领域协同与渐进式工程 (30:00) 随着主流云厂商与开源社区的基础模型与神经网络架构高度通用化，构建实用型AI系统的工程工作量约有80%集中在数据工程阶段。为防止上游数据突变击穿下游系统，团队必须在全生命周期落地动态元数据框架与显式数据契约，对未通过模式校验的数据流实行程序化阻断，同时为医疗与金融等受监管领域构建联邦数据访问与合规防线。 在面对庞大且未经整理的复杂数据湖时，专家建议采取从小切片验证起步的渐进式策略：先通过代表性小样本检验数据形态与异常分布，再分步叠加清洗过滤、特征转换与调度逻辑。更为关键的是，由于工程师频繁跨越网络安全、核能或环境工程等高度专业化的领域，与一线业务及物理领域专家紧密协同能解决超过90%的开发卡点；借助GitHub Copilot、Cursor或Gemini等AI代码助手厘清老旧代码依赖，结合反事实分析与因果建模解释输出机理，才能真正将模型准确率转化为业务管理层认可的财务收益。 专家团队总结指出，扩展面向AI的大规模复杂数据管道绝非孤立的工具迭代，而是涉及计算存储解耦、严苛数据契约、统计漂移预警与跨学科领域对齐的综合系统工程；但在非结构化异构数据爆炸式增长与模型解释合规要求日益提高的背景下，如何在保障可解释性与因果推断价值的同时持续平抑全链路算力与治理成本，仍是企业规模化落地中亟待解决的核心课题。

## Background 本内容由 Ai4 于 2025 年 8 月 27 日发布在 YouTube 平台上。视频记录了一场由来自梅西百货、Albertsons、休斯敦大学及 Varo Bank 的四位资深专家参与的小组讨论，聚焦于如何构建可扩展的数据管道以应对 AI 模型面临的海量复杂多模态数据挑战，并探讨了系统性能、数据完整性与实时处理等核心议题。 ## TL;DR 圆桌讨论深入探讨了现代 AI 数据管道的架构选型与工程实践，对比了批流分离的 Lambda 架构、事件驱动流与现代 Lakehouse 湖仓一体模式的应用边界。专家们结合零售商超、金融科技与联邦政府科研（如能源部与国防部项目）的实战案例，分享了存算分离、多模态数据独立处理与晚期融合、统一 Parquet 格式以及通过夜间批处理降低云算力成本等关键策略。此外，嘉宾们强调了在生产环境中落地数据契约、端到端数据血缘与可观测性（如 Databricks/Kibana）的必要性，并指出 AI 工程中 80% 的工作在于数据工程，提倡从小规模样本渐进式开发并与跨领域专家紧密协作。 ## Atmosphere & style 整场讨论展现出浓厚的工业界实战与工程务实风格，节奏明快且信息密度极高。主持人的提问精准串联起各个议题，嘉宾们不仅毫无保留地分享了架构设计中的避坑经验（如小样本 POC 到生产环境的断层教训），还穿插着诙谐风趣的行业洞见与调侃。讨论在严谨的技术深度与轻松自然的交流氛围中推进，兼具高屋建瓴的架构视野与底层落地的工程指导意义。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=4s) 麦肯锡调研表明，拥有成熟数据管道的企业在 AI 应用普及度与投资回报率（ROI）方面高出同行约 2.5 倍。 - [02:21](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=141s) 面对海量多源数据，现代架构广泛结合了分流离线批处理与低延迟事件流的 Lambda 架构，并向云原生数据湖与 Lakehouse 模式演进。 - [06:09](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=369s) 联邦科研与政府项目因严格合规与成本限制更青睐混合 AI 架构，并在云端采用任务完成即下线的“用完即毁”资源管理机制以遏制账单。 - [07:56](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=476s) 控制系统成本与复杂度需要彻底实现存算分离、模块化 ETL 设计，并主动优化底层数据分区策略与 SQL 查询效率。 - [11:29](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=689s) 将数据摄取、处理与存储层解耦，结合 Apache Airflow 等编排工具及 Parquet 列式存储格式，能显著降低跨节点传输与 I/O 成本。 - [13:15](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=795s) 小规模 POC 的数据假设无法直接套用于全量生产流，必须在立项初期设计前瞻性压力测试，防止真实多模态数据摄入时流水线崩溃。 - [16:10](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=970s) 处理文本、图像及传感器等复杂多模态数据时，推荐构建独立子流水线抽取特征，并结合全局唯一 ID、时间戳及 Schema Registry 进行晚期融合。 - [20:15](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1215s) 由于全量打标成本过高，数据血缘追踪应以业务 KPI 与 ROI 为导向，在模块化独立流水线的核心节点进行分级监控与实时报警。 - [22:33](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1353s) 当面对未经清洗的混乱原始数据时，工程团队需自建去重、统计学异常校验及缺失值智能填补机制，防止脏数据误导下游模型。 - [25:35](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1535s) 通过监控输入特征标准差是否超出 ±3σ 检验数据漂移，并建立周期性的机器学习重训机制以应对生产环境中的模型漂移。 - [28:07](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1687s) 利用 Kibana 和 Databricks 搭建全链路可观测性技术栈，明确拆分数据工程、机器学习工程与数据科学团队的职责边界。 - [30:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1801s) AI 落地项目中约 80% 的工作量在于数据工程与 Schema 演进防御，同时需借助可解释性 AI（XAI）将预测结果转化为业务决策价值。 - [34:05](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2045s) 在高监管行业中引入数据契约（Data Contracts），可明确界定 Schema 模式与质量预期，有效落地安全合规的联邦数据访问机制。 - [36:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2161s) 复杂数据工程应遵循从小规模低复杂度样本切入的渐进式开发路径，并积极配备深耕行业的领域专家（Domain Expert）深度结盟协作。 - [38:30](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2310s) 数据工程师应深入既有代码库与现代 Lakehouse 架构，善用 GitHub Copilot、Cursor 等 AI 编程助手加速理解表依赖与特征管道。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=4s) 开场介绍：构建面向 AI 的高扩展性数据管道 讨论由来自梅西百货（Macy's）的机器学习工程高级经理 Chetana Katpally 主持。她开门见山地指出，在当前各组织全力加速业务价值转化的背景下，如何为 AI 构建能处理海量数据集与复杂数据类型的扩展性数据管道，已成为企业关注的核心议题。麦肯锡的一项调研更是为此提供了有力佐证：拥有成熟数据管道的组织，在 AI 应用普及度及投资回报率（ROI）方面，跑赢同侪的概率要高出约 2.5 倍。 在正式切入技术探讨前，三位背景涵盖零售商超、政府科研与金融科技的专家分别进行了简要自我介绍： - **Shiva Kumar Ramavath**：Albertsons 的首席 AI/机器学习工程师，目前正结合 Agentic AI 框架推进需求预测（Demand Forecasting）系统的研发。 - **Hardik Gohel**：休斯敦大学（University of Houston）应用人工智能实验室主任兼研究主管，主导面向美国能源部（DOE）、国防部（DoD）以及国土安全部（DHS）的政府 AI 解决方案，管理超过 5 年、每年 200 万美元的联邦科研经费，同时经营着自己的 AI 引擎咨询公司。 - **Arjun Mantri**：Varo Bank 主任数据工程师（此前曾负责 TikTok 的实验平台搭建），目前专注于为银行构建核心数据平台，并为数据科学团队设计特征工程管线。 --- ### [02:21](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=141s) 核心架构选型：Lambda 架构、Lakehouse 与云原生模式 进入第一个议题，主持人 Chetana 询问面对海量多源数据时，业内最有效的架构模式与技术选型是什么。Shiva Kumar 指出，虽然市面上的架构模式数不胜数，但最根本的两大基石仍是 Lambda 架构与 Kappa 架构。在 Albertsons 的零售业务实践中，Lambda 架构应用广泛，因为它能同时分流批处理与流式处理： ```mermaid flowchart TD accTitle: Lambda Architecture Overview subgraph Ingestion["数据摄入层"] SRC["多源业务与用户数据"] end subgraph BatchLayer["批处理层 (历史深度)"] SPK["Apache Spark 分布式网络"] HIST["读取 4-6 年历史数据"] TRAIN["需求预测模型训练 / 推理"] end subgraph SpeedLayer["实时流处理层 (低延迟)"] KFK["Apache Kafka 消息队列"] STREAM["实时事件驱动数据"] REC["实时个性化推荐系统"] end SRC --> SPK SRC --> KFK SPK --> HIST --> TRAIN KFK --> STREAM --> REC ``` 在 Albertsons 的具体业务场景下，不同场景对处理链路的要求截然不同： - **需求预测模型**：侧重离线批量数据，通常需要调度系统拉取 4 至 6 年的历史大周期数据进行模型训练与离线推理，核心调度 cadence 可以是每日或按月度执行，计算依托 Apache Spark 这类分布式计算框架。 - **实时个性化推荐**：当用户打开应用时，系统需要毫秒级捕捉实时上下文，此时必须依赖像 Apache Kafka 这样的事件驱动流平台来拉取实时事件流。 针对流与批的边界，Arjun 进行了补充。他指出如今不仅有 CDC（变更数据捕获）驱动的事件流架构，越来越多的现代技术栈正在转向数据湖（Data Lake）与湖仓一体（Lakehouse/Warehouse）模式。企业倾向于直接基于 AWS S3 或 Google Cloud 原生存储底座构建统一数据湖，以此绕过传统冗长的数据流转链路，在事件驱动和批流混合之上赋予下游最大程度的消费灵活性。 --- ### [06:09](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=369s) 联邦科研与国防场景的混合云实践与格式治理 当主持人询问学术界与政府科研项目如何落地时，Hardik Gohel 透露，联邦项目的技术选型受严格的保密协议与成本控制约束，几乎从不采用“100% 纯公有云”方案，而是坚定的**混合 AI 架构（Hybrid AI Architecture）**。实验室本地配备了 Tesla GPU 并行计算服务器；当处理超大规模复杂数据集、本地算力不足时，才会向外扩展至支持的微软云生态或 AWS 基础设施。 在云端资源的使用策略上，团队采用了“用完即毁”的精细化机制：在 AWS 上搭建 Data Lake 和 GenAI 实例，一旦特定计算或数据清洗任务完成，实例会立刻停机下线，以最大限度遏制云端账单失控。 随后 Arjun 特别强调了一个底层前置条件：无论选择何种架构模式，在数据流入管道之前，必须针对复杂多变的数据格式做好清洗与标准化（Normalization）。数据格式的异构性若未在初始阶段被规范化，后续模型管道必将承载严重的工程债务。 --- ### [07:56](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=476s) 权衡三维三角：性能、成本与系统复杂度的平衡之道 在讨论如何在高吞吐、低延迟、可维护性与成本之间寻找平衡时，Hardik 分享了一个具体的政府科研案例：在为美国能源部（DOE）研发地球物理项目时，系统需要执行实时异常检测（Real-time Anomaly Detection），这部分链路必须全天候依赖低延迟的云上流处理；然而，针对大型重度深度学习模型的重训任务，他们则全部迁移到夜间批处理作业（Nightly Batch Jobs）执行。仅仅通过这一策略微调，模型重训的算力支出就直接削减了 30%。 针对大规模系统的成本与复杂度控制，Shiva 与 Arjun 提出了系统化的工程解法： - **存算分离（Decouple Storage and Compute）**：Shiva 指出，衡量平台 ROI 的首要原则是将存储层与计算层彻底解耦。明确存储边界与计算引擎的开销配比，结合统一数据平台，可以有效压降 AI 训练和推理链路的整体复杂度。 - **模块化架构设计（Modular Design）**：针对市场（Marketing）、财务（Finance）等不同业务域划分独立的 ETL 模块，不仅隔离了业务复杂性，还能针对性地实施算力优化。 - **下钻优化现有管道**：Arjun 强调，很多成本失控并不源于宏观架构，而是源于底层工程细节。定期回溯旧有管道、检查数据分区（Data Partitioning）策略、排查效率低下的 SQL Join 逻辑至关重要。 - **谨慎的技术栈开销选型**：例如在 AWS 上使用 Athena 这类交互式查询引擎时，计费直接挂钩于扫描的数据量（同时消耗计算与存储读取成本），若未做好列式存储与有效分区，无节制的全表扫描会导致账单呈指数级膨胀。 ### [11:29](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=689s) 解耦架构、任务编排与统一文件格式优化 在构建大规模数据流水线时，将计算优化与架构解耦相结合是降低成本与提升效率的核心。Arjun Mantri 强调，在 Spark 内部直接完成数据清洗与转换再推送到特征存储（Feature Store），能有效减少不必要的开销，但这需要工程师深刻理解底层技术细节，做好查询优化并把控影响性能的关键参数。主持人 Chetana Katpally 总结道，将数据摄取（Ingestion）、数据处理（Processing）与数据存储（Storage）拆分为独立的实体与进程，是节省时间和成本的关键架构策略。 Shiva Kumar Ramavath 证实了这一思路，并指出利用开源工具能极大提升编排能力。例如，使用 Apache Airflow 作为工作流编排器，可以灵活管理 Spark 作业：从底层存储拉取数据、执行批处理或流式计算，并严格按照下游 AI 模型所需的时间窗口和格式进行任务调度。此外，无服务器（Serverless）计算方案也是极具弹性的可选路径。 Arjun Mantri 补充了容易被忽视的文件格式与压缩策略。他指出，许多内部团队因习惯仍在使用低效的 CSV 或原始 JSON 格式，这在数据规模扩大时会带来沉重的存储与 I/O 负担。工程师应当强力推动团队统一转向 Parquet 等高效的列式存储与压缩格式，从底层直接压降跨节点传输和存储的成本。 ```mermaid flowchart LR accTitle: 数据流水线解耦架构 subgraph S1["存储层 Storage"] Raw["原始数据池"] end subgraph S2["调度与处理层 Processing & Orchestration"] AF["Apache Airflow 编排"] --> Spark["Apache Spark 计算引擎"] end subgraph S3["特征与模型层 Feature & Model Store"] Parquet["统一 Parquet 存储"] --> FS["Feature Store / AI 模型"] end Raw --> Spark Spark --> Parquet ``` --- ### [13:15](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=795s) 多模态复杂数据整合与 POC 到生产环境的断层挑战 随着现代 AI 应用越来越多地依赖非结构化文本、图像、视频、IoT 传感器流和系统日志等复杂数据，如何将高度异构的数据转化为模型可用的统一格式成为巨大瓶颈。Hardik Gohel 分享了他在政府及科研前沿项目（如美国能源部资助的大型政府建筑暖通空调 HVAC 系统及 IoT 能源优化）中的真实经历。他指出，处理多太字节（Multi-terabyte）级别的数据时，最大的难题在于将音频、视频、图像、文本和物理传感器信号映射到统一的 Schema 空间中，以供模型训练与部署。 Hardik 坦言了一个在业界极为普遍但极具警示意义的教训：在为能源部（Department of Energy）撰写项目提案时，团队基于极小规模的数据集快速构建了概念验证（POC），实验效果非常出色并顺利拿到了项目资助；然而，当项目正式落地、面对海量真实数据流入时，原有的数据流水线瞬间崩溃。更糟糕的是，进入真实模型的数据分布完全脱节，导致模型产生了一大堆在训练阶段从未见过的异常预测值。"故事的寓意在于：在训练模型之前，必须提前预测并规划从 POC 到大规模可扩展 AI 解决方案中所有潜在的数据多样性与摄取差异。" - **Schema 标准化**：必须在输入端将多模态非结构化数据映射到统一特征维度与标识空间。 - **环境断层预警**：小样本 POC 的数据假设无法直接套用于全量实时生产流，需在立项初期设计前瞻性压力测试。 --- ### [16:10](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=970s) 多模态独立处理流水线与晚期融合策略 针对复杂多模态数据的处理架构，Shiva Kumar Ramavath 介绍了 Albertsons 等零售场景下的实践经验。在电商与零售业务中，单一商品往往同时包含文本描述、商品图片、宣传视频以及 IoT 货架传感器信号。最佳实践是为每种数据类型构建独立的子流水线： - **文本数据**：进行分词（Tokenization）并通过 BERT 或 Word2Vec 等模型提取语义向量（Embeddings）； - **图像/视频数据**：利用计算机视觉主干网络抽取视觉特征并生成图像 Embedding； - **标准化汇聚**：将各独立流水线输出的高阶特征在下游进行归一化合并，使 AI 模型能够全面理解商品上下文并输出精准的推理结果。 Arjun Mantri 则从数据工程角度提出了另一种互补视角：如果不希望维护过多分散的独立流水线，可以通过在数据流中强制注入统一的时间戳（Timestamps）和全局唯一 ID 作为元数据，构建高精度的时序关联数据集。此外，利用数据晚期融合（Late Fusion）、Schema Registry（模式注册表）以及读时模式（Schema on Read）等机制，能够有效应对异构数据在不同刷新频率下的合并难题。对于时间戳的妙用，Hardik 幽默地回应："时间戳这招我肯定得回去试试！" Chetana Katpally 和 Arjun 进一步强调，不同模态流水线的刷新频率大相径庭，引入 Schema Registry 能够精准跟踪上游演进，确保数据在合并前必须经过严格清洗，杜绝 "Garbage in, garbage out"（垃圾进，垃圾出）的恶性循环。 --- ### [20:15](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1215s) 大规模数据质量监控、全链路血缘与报警机制 当流水线达到海量规模时，微小的数据质量偏差都会在下游模型端被数倍放大。Arjun Mantri 指出，所有数据工程师都经历过这样的痛苦场景：流水线好不容易搭建完毕交付给机器学习（ML）或数据科学团队，对方很快找上门来抱怨 "数据里全是 Null 空值，模型跑崩了"。随着 AI Agent 和各类自动化组件在多系统间广泛调用，全链路数据血缘（Data Lineage）与可追溯性（Traceability）变得比以往任何时候都更加关键。 由于对所有数据字段无差别打上全量血缘标签在成本上并不可行，Arjun 建议采取 ROI 驱动的务实策略： 1. **对齐业务 KPI**：识别出对核心业务指标和投资回报率（ROI）最具决定性影响的关键特征与数据流。 2. **模块化分级监控**：在由 Shiva 提到的模块化独立流水线关键节点上，嵌入细粒度的血缘追踪。 3. **分级预警机制**：结合 AWS CloudWatch 等监控工具，在核心阶段设置针对空值率、分布漂移和延迟的实时报警。 Shiva Kumar Ramavath 补充强调，数据质量保障必须贯彻端到端理念，从最源头的数据摄取阶段开始设置数据校验关卡（Validation Checks），并沿着整个血缘链路持续验证，直至数据被在线推理服务完全消费。 - **阶段性断言**：在 Source 端、Transform 端、Feature Store 端分别设立自动化数据契约与校验规则。 - **精准警报**：优先保障高价值业务流的监控密度，避免无效警报疲劳。 ### [22:33](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1353s) 数据质量校验与应对混乱原始数据的统计学方法 在机器学习生命周期中，数据质量校验必须贯穿于数据摄取与模型训练的每一个环节。Shiva Kumar Ramavath 指出，上游微小的数据缺陷都会对下游推理产生连锁反应。例如，训练集中混入重复记录会直接误导模型的学习方向；而若未妥善处理空值（Null），模型生成的错误推理最终会污染终端消费者应用程序。因此，在模型构建前部署轻量级的数据质量检查与校验机制，是保障模型高精度推理的基石。 针对现实业务中的数据痛点，Hardik Gohel 分享了其在主持联邦政府及科研项目时的真实经历。他坦言，客户往往直接将未经清洗的原始数据“一股脑扔过来”，他们对重复率、缺失值填补算法或是 $R^2$ 拟合优度等统计指标根本不感兴趣，唯一的诉求就是尽快交付一个立即可用的 AI 模型。当团队向客户抱怨数据质量差并要求源头修复时，客户通常不愿介入。为了解决这一被动局面，Hardik 所在的实验室开发了一套专门的统计学模型与定制化填补技术，在数据输入 AI 模型前自动执行深度清洗与有效特征筛选。 ```mermaid flowchart TD accTitle: 数据质量与校验管道流程图 accDescr: 展示从原始数据输入、统计学质量校验、Schema 拦截到模型训练与推理的端到端数据流。 subgraph Ingestion["数据摄取与清洗"] A["原始/异构数据输入"] --> B["数据去重与空值校验"] B --> C["定制统计模型与智能填补"] end subgraph Validation["Schema 与漂移监控"] C --> D{"元数据 Schema 校验"} D -- "未通过" --> E["丢弃 / 拦截报警"] D -- "通过" --> F["统计漂移检测 (±3σ)"] end subgraph Modeling["模型生命周期"] F --> G["AI 模型训练 / 重新训练"] G --> H["模型在线推理"] H --> I["可观测性与血缘追踪 (Databricks/Kibana)"] end ``` - 重复数据会导致模型在训练过程中产生严重的表征偏见。 - 客户与业务方通常缺乏清洗源头数据的意愿，工程团队必须自建高质量的数据预处理流水线。 - 针对复杂多源的数据集，通过自研统计模型进行特征评估，能够显著提升进入训练阶段的数据有效性。 --- ### [25:35](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1535s) 数据漂移与模型漂移检测，以及海量数据下的血缘追踪 在生产环境中维持模型稳定性，需要对统计学层面的数据漂移与模型漂移建立全天候监控。Shiva 解释道，以客户销售数据管道为例，工程团队可以通过监控标准差是否超出 $\pm 3\sigma$（3 个标准差范围）或偏离正态分布，来快速识别输入特征是否发生数据漂移（Data Drift）。一旦模型上线部署，其实际推理表现可能会偏离最初的设计预期，即发生模型漂移（Model Drift），这就要求系统必须具备定期的机器学习重训机制（ML Retraining Cadence），以保障业务输出的持续准确。 ``` 数据漂移监控 (Data Drift) 模型漂移监控 (Model Drift) ┌───────────────────────────────┐ ┌───────────────────────────────┐ │ • 监控输入特征分布 │ │ • 监控线上推理准确率 │ │ • 标准差阈值 (超出 ±3σ 报警) │ ──> │ • 评估推理偏离业务预期的程度 │ │ • 正态分布偏离检验 │ │ • 触发自动化定期重训 (Cadence)│ └───────────────────────────────┘ └───────────────────────────────┘ ``` 谈及大规模数据管道的运维痛点，Arjun Mantri 结合自身在大型社交媒体公司的工程经验强调了数据血缘（Data Lineage）的不可替代性。他指出，在面对单张数据表规模就高达 TB 甚至 PB 级的场景时，即便系统已经完成了精细的索引构建、数据分区（Partitioning）与并行计算处理，一旦下游报表或模型出现异常，回溯问题根因仍然如同大海捞针。如果没有清晰的数据血缘，工程师与业务团队之间将陷入漫长而痛苦的拉锯沟通中。引入端到端的数据血缘追踪，能够大幅降低定位问题所需的时间与计算资源成本。 - 统计学检测规则：将特征标准差波动超出 $\pm 3\sigma$ 作为数据漂移的硬性预警指标。 - 应对模型漂移的标准解法是建立周期性的自动化重训流水线。 - 单表 PB 级数据量下，索引与分区仅能解决查询吞吐问题，数据血缘才是解决故障排查与合规审计的核心工具。 --- ### [28:07](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1687s) 可观测性技术栈构建与数据团队职责划分 针对数据科学家在数据探索阶段耗费过多精力的现状，团队之间应当进行合理的职责划分与自动化交接。Shiva 提出，可以将底层的数据提取与流水线编排工作剥离并交由数据工程（DE）或机器学习工程（MLE）团队负责，使数据科学家能够专注于核心模型的算法创新与调优。 为了实现全流程的可追溯性，现代数据架构依赖于成熟的可观测性（Observability）技术栈： - **Kibana**：可用于搭建直观的可视化仪表板，追踪数据摄取管道的健康状况、监控模型在线推理的实时延迟与性能指标。 - **Databricks**：提供开箱即用的统一可观测性与血缘追踪能力，能够无缝追踪从原始数据源、特征加工到最终模型推理输出的每一个计算步骤。 针对这种清晰的分工模式，现场嘉宾风趣地调侃道：“这么说来，数据工程师现在可算是有活干了，这挺好。” --- ### [30:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1801s) Schema 演进、跨职能协作与业务导向的可解释性（XAI） 在复杂数据类型不断涌入的大模型时代，管道对 Schema 演进（Schema Evolution）的弹性防御至关重要。Shiva 说明道，系统应当维护一套独立的元数据 Schema（Metadata Schema）。每当新数据流入并准备输入模型时，系统都会先对照该元数据标准进行严格验证；凡是无法通过 Schema 校验的脏数据或异常结构，都会被直接拦截，严禁流入模型训练与推理环节。 Hardik 补充指出，当今 AI 落地任务中有 **80% 的工作量其实都落在数据工程上**。虽然各大科技巨头（如 Microsoft、AWS、Meta、OpenAI）已经提供了高度成熟且鲁棒的基础模型与算法库，部分工程师可能仍执着于自研神经网络，但现实中自建网络往往难以全面超越现有的成熟框架。因此，真正的核心竞争力在于数据处理与多方协同。要实现业务目标，必须促成五方紧密联动： | 协作角色 | 核心职责与关注点 | |:--- |:--- | | **客户 / 业务方 (Client)** | 明确商业目标与实际落地价值（Dollar Value） | | **数据工程师 (Data Engineer)** | 负责 80% 的管道构建、Schema 校验与血缘追踪 | | **数据科学家 (Data Scientist)** | 负责模型架构选型、算法实验与精细化调优 | | **AI 治理团队 (AI Governance)** | 制定合规准则、监控数据使用权限与生命周期 | | **AI 伦理团队 (AI Ethical Team)** | 审核模型偏见、保障算法透明度与输出公正性 | Shiva 进一步强调了模型可解释性（Explainability）在商业决策中的核心地位。他直言，业务人员与管理层根本不在乎模型在测试集上的准确率是 90% 还是 95%，他们真正关心的是这个模型究竟能为企业创造多少“真金白银的商业价值”（Dollar Value）。企业目前正广泛引入反事实推理（Counterfactuals）与因果模型（Causal Models）等 XAI 技术，将晦涩的神经网络预测转化为业务人员能够听懂、敢于采纳的决策逻辑。 ### [34:05](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2045s) 数据契约与合规性要求 在医疗健康等高监管行业中，模型与管道的“可解释性与可理解性”（Explainability and Interpretability）是团队在推进任何方案前必须首要考虑的基石。为了跨团队协同保障数据质量与合规标准，引入**数据契约**（Data Contracts）极为关键。尽管制定数据契约需要投入大量时间和跨团队沟通成本，但它明确界定了数据模式（Schema）、数据体量（Data Size）以及交付预期。通过在组织内部推行数据契约，团队能够有效落地面向安全与合规的联邦数据访问（Federated Data Access）机制，从而在源头上规避数据漂移与权限漏洞。 ```mermaid flowchart TD accTitle: 复杂数据工程管道构建与工程实践流程 subgraph S1["1. 需求与契约治理"] A["明确业务与合规需求<br/>(可解释性 / 安全规范)"] --> B["制定数据契约 (Data Contracts)<br/>(Schema 模式 / 体量与质量预期)"] end subgraph S2["2. 渐进式验证与管道构建"] B --> C["小规模样本数据探索<br/>(理解数据特征与结构)"] C --> D["端到端构建核心流水线<br/>(摄取 / 过滤 / 特征工程 / 模式校验)"] end subgraph S3["3. 扩展与效能优化"] D --> E["性能调优与架构演进<br/>(分区 / 并行化 / 压缩格式 / Lakehouse)"] E --> F["AI 辅助工具赋能<br/>(Copilot / Cursor / Gemini 加速代码理解与开发)"] end ``` --- ### [34:55](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2095s) 构建大规模复杂数据管道的核心建议与技术演进 面对大规模且多源异构的复杂数据，数据工程师在构建流水线初期必须把扩展性（Scalability）、数据质量（Data Quality）与数据血缘（Data Lineage）置于核心位置。在数据规模快速膨胀的过程中，必须主动引入深度优化技术，包括数据分区（Data Partitioning）、并行计算（Parallelism）以及针对性的文件存储格式与压缩算法。 工程师不能固守既有技术栈。如果现有方案无法满足时效性或资源瓶颈，团队需要具备追根溯源的技术魄力，敢于重构技术选型（Pivot Technology），持续迭代并缩短特征交付周期。此外，现代工程实践应当充分利用各类 AI 辅助工具，将 AI 原生能力融入到数据管道的设计与优化中，大幅提升流水线的构建效率与运行稳健性。 - **核心架构考量**：在设计之初就将扩展性、质量检测探针与全链路数据血缘追踪内置于管道中。 - **计算与存储优化**：结合业务查询特征实施精细化数据分区、并行处理流水线，并评估高效列式存储格式与压缩策略。 - **敏捷技术迭代**：定期审视交付延迟，勇于对过时组件进行技术栈迁移与持续重构。 --- ### [36:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2161s) 渐进式工程开发：从小规模样本切入与跨领域专家协作 在企业团队结构中，通常既有资深的架构师与管理者，也有新入行的初级数据工程师。为了确保团队高效产出，切忌在项目初期盲目接入海量、非结构化且高度异构的完整数据集。 ``` “不要指望刚毕业的新人一进团队就能施展魔法创造奇迹，工程落地从来不是那样运作的。” ``` Hardik Gohel 分享了他在西雅图参与美国能源部（Department of Energy, DOE）项目的经历：当时合作方提供了海量数据，但他明确要求团队先抽取小规模、复杂度较低的样本数据进行探索，摸清数据分布与基础特征后，再逐步增加复杂度，循序渐进地补齐质量校验、特征工程、过滤清洗、数据摄取与 Schema 固化等环节。 此外，针对数据工程师必须具备特定领域知识的误区，必须认识到工程人员不可能在三周内快速掌握地球物理学或核能等高度精密的专业理论。当工程师从网络安全等领域跨界到核能或环境修复项目时，必须为其配备专属的**领域专家（Domain Expert）**进行结对协作。 --- ### [38:30](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2310s) 深入数据本质、现代湖仓架构与 AI 编程工具赋能 数据工程师职业成长的核心基础在于深入理解数据背后的业务逻辑与业务域（Domain）。初级工程师可以通过扎实掌握基础 SQL 以及积极与团队成员沟通交流来快速建立领域认知。在架构认知方面，掌握现代**湖仓一体架构**（Lakehouse Architecture）至关重要——该架构涵盖了从底层源应用数据摄取、加工处理到统一数据仓库与分析建模的全流程。 在实际开发中，工程师还需深入现存的代码资产： - **深入既有代码库**：现实业务中旧有代码库不可能被全盘推翻，工程师必须沉下心亲自完成端到端的管道搭建，摸清复杂管道的上下游脉络。 - **善用 AI 编程助手**：熟练利用 GitHub Copilot、Cursor 或 Gemini 等工具，编写精准的 Prompt 来梳理代码库中跨数十处调用的表依赖关系、特征工程逻辑与模型调用链，从而快速吃透老旧代码库。 - **与领域专家深度结盟**：在网络安全、AI 靶向给药与远程医疗、新能源、气候变化以及基于 IoT 传感器的可持续基础设施等多个跨界 AI 项目中，与领域专家紧密配合能够解决超过 90% 的业务与建模难题。正如 Hardik Gohel 幽默指出的那样：“工程师不要只想着跟经理或 CTO 套近乎，因为拿不出成果他们一样会解雇你；真正能帮你通关的是那些深耕行业 5 到 10 年、深谙模型最终指标意图的领域专家。” ## Notable quotes > [00:04](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=4s) 拥有成熟数据管道的组织，在 AI 应用普及度及投资回报率（ROI）方面，跑赢同侪的概率要高出约 2.5 倍。 — Chetana Katpally > [06:09](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=369s) 在云端资源的使用策略上，团队采用了“用完即毁”的精细化机制：在 AWS 上搭建 Data Lake 和 GenAI 实例，一旦特定计算或数据清洗任务完成，实例会立刻停机下线，以最大限度遏制云端账单失控。 — Hardik Gohel > [13:15](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=795s) 故事的寓意在于：在训练模型之前，必须提前预测并规划从 POC 到大规模可扩展 AI 解决方案中所有潜在的数据多样性与摄取差异。 — Hardik Gohel > [16:10](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=970s) 时间戳这招我肯定得回去试试！ — Hardik Gohel > [20:15](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1215s) 数据里全是 Null 空值，模型跑崩了 — Arjun Mantri > [28:07](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1687s) 这么说来，数据工程师现在可算是有活干了，这挺好。 — Hardik Gohel > [30:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1801s) 80% 的工作量其实都落在数据工程上 — Hardik Gohel > [30:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=1801s) 业务人员与管理层根本不在乎模型在测试集上的准确率是 90% 还是 95%，他们真正关心的是这个模型究竟能为企业创造多少“真金白银的商业价值”（Dollar Value）。 — Shiva Kumar Ramavath > [36:01](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2161s) 不要指望刚毕业的新人一进团队就能施展魔法创造奇迹，工程落地从来不是那样运作的。 — Hardik Gohel > [38:30](https://www.youtube.com/watch?v=a1TdTAgNI0E&t=2310s) 工程师不要只想着跟经理或 CTO 套近乎，因为拿不出成果他们一样会解雇你；真正能帮你通关的是那些深耕行业 5 到 10 年、深谙模型最终指标意图的领域专家。 — Hardik Gohel ## Mentioned Arjun Mantri, Chetana Katpally, Shiva Kumar Ramavath, Hardik Gohel, Varo Bank, Macy's, Albertsons, University of Houston, TikTok, Apache Spark, Apache Kafka, Apache Airflow, AWS S3, AWS Athena, AWS CloudWatch, Google Cloud, Microsoft Azure, OpenAI, Meta, Databricks, Kibana, BERT, Word2Vec, Parquet, GitHub Copilot, Cursor, Gemini, DOE, DoD, DHS ## Video description Arjun Mantri, Staff Data Engineer, Varo Bank Chetana Katpally, Senior Manager - Machine Learning Engineering, Macy's Shiva Kumar Ramavath, Lead AI Machine Learning Engineer, Albertsons Hardik Gohel, Director of Applied AI Research Laboratory, University of Houston Abstracts As AI models become more complex, so do the demands on data pipelines. This session will explore strategies for building scalable data pipelines that can handle large datasets and diverse data types. Panelists will share insights into managing high-volume data flows, ensuring data integrity, and optimizing pipelines for real-time data processing. Learn how to balance performance with scalability and discover the best practices for preparing complex data—such as text, images, and sensor data—for AI models, ensuring that your data infrastructure supports advanced AI applications. [Source](https://www.youtube.com/watch?v=a1TdTAgNI0E)

[0:04] Chetana Katpally: 拥有成熟数据管道的组织，在 AI 应用普及度及投资回报率（ROI）方面，跑赢同侪的概率要高出约 2.5 倍。 [6:09] Hardik Gohel: 在云端资源的使用策略上，团队采用了“用完即毁”的精细化机制：在 AWS 上搭建 Data Lake 和 GenAI 实例，一旦特定计算或数据清洗任务完成，实例会立刻停机下线，以最大限度遏制云端账单失控。 [13:15] Hardik Gohel: 故事的寓意在于：在训练模型之前，必须提前预测并规划从 POC 到大规模可扩展 AI 解决方案中所有潜在的数据多样性与摄取差异。 [16:10] Hardik Gohel: 时间戳这招我肯定得回去试试！ [20:15] Arjun Mantri: 数据里全是 Null 空值，模型跑崩了 [28:07] Hardik Gohel: 这么说来，数据工程师现在可算是有活干了，这挺好。 [30:01] Hardik Gohel: 80% 的工作量其实都落在数据工程上 [30:01] Shiva Kumar Ramavath: 业务人员与管理层根本不在乎模型在测试集上的准确率是 90% 还是 95%，他们真正关心的是这个模型究竟能为企业创造多少“真金白银的商业价值”（Dollar Value）。 [36:01] Hardik Gohel: 不要指望刚毕业的新人一进团队就能施展魔法创造奇迹，工程落地从来不是那样运作的。 [38:30] Hardik Gohel: 工程师不要只想着跟经理或 CTO 套近乎，因为拿不出成果他们一样会解雇你；真正能帮你通关的是那些深耕行业 5 到 10 年、深谙模型最终指标意图的领域专家。

[0:04] Chetana Katpally: organizations with mature data pipelines are approximately 2.5 times more likely to outperform their industry peers in AI adoption and return on investment [2:21] Shiva Kumar Ramavath: Lambda incorporates both batch and real-time stream processing layers in parallel, whereas Kappa focuses strictly on single-path stream processing [6:09] Hardik Gohel: federal projects cannot rely exclusively on 100% cloud-native solutions due to stringent security mandates and high operational compute expenses [7:56] Hardik Gohel: shifting heavy training to off-peak nightly batches alone reduced their compute costs by approximately 30% [7:56] Arjun Mantri: unoptimized query engines charge directly across scanned compute and storage, making sloppy queries a primary driver of runaway costs as data scales [13:15] Hardik Gohel: ingestion differences and unmodeled schema drift caused the production model to output erratic predictions never observed during training [16:10] garbage in, garbage out [30:00] roughly 80% of the operational effort resides in data engineering rather than model creation [32:36] Non-technical business leaders are rarely motivated by abstract metrics like 90% or 95% model accuracy; their core metric is the tangible 'dollar value' generated by the deployment [38:30] while pleasing management is common practice, a data engineer's most critical operational ally is always the domain expert

[0:04] Good morning everybody. Thank you for [0:05] joining us on this discussion for about [0:07] scaling data pipelines for AI handling [0:10] large data sets and complex data types. [0:13] Um a topic on top of organizations mind [0:17] who are trying to accelerate value. Now, [0:19] according to McKenzie, organizations [0:22] with mature data pipelines are about two [0:24] and a half times more likely to [0:27] outperform their peers in AI adoption [0:29] and ROI. Further highlighting the [0:32] importance of handling large data sets, [0:34] comp large and complex data sets. Um, I [0:37] am Chetna Kpali, senior manager machine [0:39] learning engineering at Macy's and I am [0:41] joined by a fantastic panel. So, before [0:44] we dive in, let's go around have some [0:47] introductions. So panel members please [0:49] share your name, role and in one [0:51] sentence what are you working on [0:52] currently? [0:53] >> Sure. Uh good morning everyone. My name [0:56] is Shivakumar. I'm a lead AI engineer [0:59] for Albertson's. I'm currently working [1:01] on a demand forecasting problem uh to [1:04] implement along with the agentic AI [1:06] framework. Uh nice to meet you all. [1:09] [Music] [1:10] >> Hello good morning everyone. I'm Hardik [1:12] Gohel. I'm a director of research and [1:15] director of applied AI. uh lab at [1:18] University of Houston. Uh I have been [1:21] involved in uh developing AI solutions [1:23] for government especially focusing on US [1:26] Department of Energy, defense and [1:28] homeland security. Uh I'm handling uh [1:31] federal government projects more than 5 [1:33] years uh 2 million funding every year [1:37] and also having uh my consulting company [1:41] and working on AI engine solution [1:43] development. [1:46] Hello everyone. Uh I'm Arjun. Uh I work [1:49] in building like data platform and uh [1:52] data engineering. Uh I was with uh [1:55] working on like previously on the Tik [1:57] Tok uh experimentation platform and [1:59] currently working with the bank to build [2:02] their uh data platform and feature [2:04] engineering for data science team. [2:07] >> Thank you. So we have a diverse panel. [2:10] So we have retail CPG industry uh [2:13] education and run government and funded [2:16] uh federal projects and we have finance [2:19] and banking banking sector. Neat. So [2:21] let's dive in. So my first question [2:23] Shiva and Arjun so designing scalable [2:26] data pipelines involves countless [2:29] architectural choices. So from your [2:31] experience, what architectural patterns, [2:34] technologies or frameworks have you [2:36] found most effective for building [2:39] pipelines that can handle large volumes [2:42] and diverse data types? [2:44] >> Yeah, absolutely. Uh so yeah, designing [2:47] um data pipelines for handling the large [2:50] data data sets is critical for any large [2:53] organization or small or medium scale [2:55] organizations. So uh to do that there [2:58] are two major there are countless of [3:00] architectures available in the market [3:03] but on top of that there is two major [3:06] architectures which is one lambda and [3:08] another one is kappa. So lambda is the [3:11] architecture where it involves both [3:13] batch processing and the streaming [3:15] processing and coming to the kappa it [3:18] it's only the batch processing you can [3:20] say it as a one way now in lambda as we [3:23] have the both parallelly can be done so [3:26] if if I wanted to do any build any AI [3:29] model which which uh needs to have a [3:32] huge amount of data if you ask me about [3:34] in a retail company I need to have a [3:36] very good architecture so in order to do [3:38] that we are like u using lambda [3:42] architecture. Let's say if I'm using [3:44] demand forecasting model where it needs [3:46] only the batch processing of the data. [3:49] So in that case we will be reading the [3:52] historical uh uh data like five years, [3:55] four years, 6 years to have a better [3:57] model to have a better inference. And [3:59] similarly if I wanted to do a like [4:02] personalization on the items whenever a [4:03] user logs logs into the app and wanted [4:06] to see the uh personalized item we need [4:09] to have a real-time capability. So for [4:11] that we need to use along with that the [4:14] real time B real-time processing. So to [4:17] handle this batch processing and the [4:19] realtime processing it will be a [4:20] separate thing based on business use [4:22] case based on the business use case this [4:25] can be done. So if you um if you ask me [4:28] on batch processing side how we do it, [4:30] we need to have a orchestration where it [4:33] can be like where where data can be [4:35] pulled maybe regularly in terms of uh [4:39] every day or maybe for the inference or [4:41] when we try when we do the model [4:43] training we can do it monthly basis on [4:46] the uh model training cadence and [4:48] similarly for the batch processing where [4:51] sorry similarly for the realtime [4:53] processing where we need the realtime uh [4:57] uh event based data for that we can use [4:59] any real-time streaming platforms like a [5:02] uh Apache Cafka and for the batch batch [5:06] processing like for uh Spark where it [5:08] uses the distributed uh distributed [5:11] network uh framework [5:13] yeah just to add on what Shiva said so I [5:17] think that's like event driven and [5:18] streaming analysis or streaming [5:20] architecture [5:22] uh which does have like CDC data or [5:24] change capture data. So you need to make [5:26] sure uh your in that case you can use [5:30] the heterogeneous approach like you want [5:32] to do batch or realtime streaming but [5:34] there is also uh growing things that [5:38] people are using like data lake and data [5:41] uh lakehouse and warehouses approach [5:43] where they don't want to go through the [5:45] traditional routes and things and have [5:47] like S3 storage or on the Google cloud [5:49] itself so as to make sense or use of [5:53] data from there itself. [5:54] and use however they want to use it. But [5:58] uh there are like data lake data lake [6:01] approaches and things what you can use [6:03] on top of event driven or uh batch [6:06] processing or real-time streaming [6:07] approaches. [6:09] >> So hardik how does this relate to your [6:11] domain uh on federal projects? [6:13] >> Yes. Uh so for federal projects uh we [6:16] are a little different uh because we are [6:18] not using 100% cloud-based resources and [6:21] data pipeline. uh we do have a hybrid uh [6:24] AI architect. How does it mean by [6:27] hybrid? Uh because u in our lab uh we do [6:32] have GPU servers. We have Tesla GPU [6:35] server uh two server for do like [6:38] parallel processing and uh when we need [6:41] to handle like large amount of data that [6:44] uh GPU cannot handle, we use uh [6:46] Microsoft ecosystem. That is what our uh [6:50] system is uh supporting. But we started [6:52] exploring AWS infrastructure where we [6:55] use this uh AWS uh data lake. Uh we also [6:59] uh have uh geni uh AI instances. So what [7:04] we do like you know when we need to [7:06] process those data we use this uh AWS [7:09] instances and try to do a processing and [7:13] then we immediately shut down because [7:16] cost is something uh really you know big [7:18] major impact for us. We're going to [7:20] discuss it uh in my later discussion. [7:23] But for us like uh nothing like 100% on [7:27] the cloud or commercial solution, it's [7:29] always hybrid. Just just adding in uh [7:33] one point here like while building or [7:36] your pipelines or uh structure and thing [7:39] you need to keep in mind different [7:41] diverse formats that you are going to [7:43] use which we are going to touch upon. So [7:46] while making sure which architecture to [7:48] choose or not make sure you have a [7:50] correct format and things and normalize [7:52] that as well like before using it but we [7:54] are going to touch point. Yeah. [7:56] >> Thank you. So scaling data pipelines [7:59] often introduces trade-offs between [8:02] performance cost and complexity. So how [8:05] do you approach balancing high [8:06] throughput and low latency while [8:09] ensuring your pipelines remain [8:11] maintainable and cost effective? [8:14] >> Yeah. uh I'll go first real quick [8:15] because I just mentioned about the [8:17] hybrid architect. So in our lab and when [8:20] we are deploying in the government [8:22] organization what we do like we use [8:24] again you know hybrid approach the [8:27] reason is a it cost effective too. So [8:30] for example let's say we need to do some [8:32] set of like you know for example one of [8:34] the geoysics project that we have done [8:37] for department of energy one of the task [8:39] was like considering like realtime [8:41] anomaly detection to do that realtime [8:44] anomaly detection we need to do that set [8:48] of you know processing and training real [8:50] time that's where we are using a real [8:53] cloud services okay but let's say if we [8:56] need to train AI model like heavy [8:59] deep learning models we do like nightly [9:02] batch job and that is reducing our cost [9:05] to 30%. So we always use hybrid approach [9:09] uh just because uh there is a data [9:11] security involved and for cost cutting [9:14] as well. So yeah uh I would like to add [9:16] on on that. So before that uh when we [9:21] like this is like a three-dimensional [9:23] approach like where performance cost and [9:25] complexity goes hand in hand for that [9:28] first we need to yeah uh for that first [9:31] we need to decouple our storage and then [9:34] against the compute and then we can [9:37] estimate our cost. So whenever we are [9:40] whenever we uh deal in terms of ROI in [9:43] terms of how our platform that are been [9:46] adopted by the organization is been [9:48] effective or not. So in order to do it [9:50] first we need to decouple it and [9:52] identify what storage we are using it [9:55] and then to to measure the performance [9:59] of that uh storage or the data what we [10:02] have in the platform we need to have a [10:04] better compute. So these are all things [10:06] goes hand hand in hand and along with [10:08] that to do it we need to have a uni [10:11] unified platform. If we have a unifi [10:13] unified platform it will definitely give [10:16] an advantage of overall the complexity [10:20] uh in terms of uh running the data [10:24] pipeline model that is required data [10:26] pipeline that is required for the AI [10:27] model to train at the same time to do a [10:30] inference as well. Adding on to that I [10:33] think we can have a modular design [10:34] approach as well. Uh let's say you are [10:37] using ETL for specific use cases or [10:40] things like marketing, finance and [10:41] things. So that modular approach will [10:44] also help you to reduce the cost. But [10:47] going one level deeper uh you can [10:50] revisit your old pipeline look into your [10:52] data partitioning uh how you can [10:54] optimize the existing queries and [10:56] things. uh many of times it's as simple [10:59] as like your joints and things like that [11:01] right I know uh some of the old pipeline [11:04] or things that can have issues or [11:06] anything but that's what I think when [11:08] you scale up or things this will bug you [11:10] a lot so having a modular design as you [11:12] go ahead is important choosing the right [11:15] technology uh is also important for [11:19] example if you do something in Athena uh [11:21] let's say for AWS or anything and use [11:24] that it cost you like compute versus [11:26] storage both thing right whereas if you [11:29] move it your like feature stores or [11:31] anything and transformation in spark [11:33] itself you're reducing that cost and uh [11:36] but yeah you need to understand the [11:38] technicality of it and make sure you are [11:41] doing your query optimization and having [11:43] the right factors included in that to [11:46] make sure you reduce your cost and help [11:48] that in that [11:50] >> right so so what I'm hearing is you're [11:52] suggesting that we separate the [11:54] injection data ingestion processing and [11:57] the storage into separate entities and [12:00] processes that would save save a lot of [12:03] uh time and also cost. [12:05] >> Yes, that's absolutely true. On top of [12:08] it u so uh as I said like we do have an [12:12] open-source uh open source libraries [12:15] also if I if I tell you like airflow is [12:18] an orchestration where it will help us [12:20] to manage the spark spark jobs. it will [12:24] gets the uh data from the storage and do [12:27] the processing and then again do the [12:29] orchestration also as per the needs of [12:31] the model that is required. So yeah [12:33] that's you're absolutely right [12:35] >> and perhaps a serverless option is also [12:37] one of the um choices. Yeah. Right. [12:39] Thank you. [12:39] >> I would add like using the correct uh [12:42] file compression techniques also like [12:44] parc things. Uh I know or Jason people [12:49] who are using it. Make sure you see what [12:52] is appropriate and things which suits [12:54] your needs and things. Decide that file [12:57] format and make sure it's unified. I [12:59] know some people uh or some internal [13:02] teams or their teams they just use CSV [13:04] or something like that. So make sure you [13:06] push them to use the correct file format [13:08] or things and it's going to help you to [13:10] reduce your cost and uh things going [13:12] forward when you scale. Yeah, [13:15] >> good um good answer there. So that segus [13:18] into our next question uh which AI [13:21] applications increasingly rely on [13:22] complex data like text uh images um IoT [13:26] data and also logs. So what challenges [13:29] have you encountered when integrating [13:31] and preparing diverse data types for AI [13:34] pipelines and how have you solved them? [13:37] Would you like to go first? [13:39] >> Yeah. Uh so uh basically uh again you [13:42] know um our projects are little you know [13:44] different from industry scale. Uh but [13:47] definitely uh we are building AI [13:49] solution which is deployable to the [13:51] government uh space. uh we prepare proof [13:54] of concepts or we prepare some set of [13:57] instances that directly go for the [13:59] deployment in terms of data complexity [14:03] that what we are handling is really [14:05] huge. Sometimes the reason behind like [14:08] it's a diversified data. [14:11] The biggest challenge that what we have [14:13] seen in the data type is like let's say [14:16] we have unstructured data audio video [14:19] images text sound anything we need to [14:22] bring them into the equal schema. So we [14:25] can feed them into the AI model for [14:28] training and testing and deployment and [14:30] rest of the process is there. Now here [14:32] is a thing that we handle multi-terabyte [14:36] of the data. uh one of the project uh so [14:39] one of the task that I would like to uh [14:41] you know brief here uh one project was [14:45] related to like uh government premises [14:48] Hvex system and there was another [14:51] project with IoT energy uh you know [14:53] optimization where we were handling a [14:56] very large data set. The first one was [14:58] like you know we had I mean I need I [15:00] needed to work with company because they [15:02] wanted to prepare a proof of concept [15:05] write a proposal and submit to [15:06] department of energy. So what happened [15:09] the data size was very small and then we [15:12] build a proof of concept real quick. We [15:15] prepared the proposal and we submit. [15:17] When the project funded the same [15:20] pipeline did not work because ingestion [15:22] was different and the real data that [15:25] cames to real model it was just throwing [15:28] bunch of prediction which we have never [15:31] seen in training. So the thing is like [15:34] you know when we need to train uh data [15:38] we need to make sure that we have a set [15:42] of schema and data in equal space that [15:46] interpreting in the same level of [15:48] identification and then work on the AI [15:51] training model. So moral of the story is [15:54] the same for us like before we train the [15:56] model we make sure that from proof of [15:59] concept to the scalable AI solution what [16:02] will be the situation that we need to [16:04] forecast in terms of the data diversity [16:08] before we train AI model. [16:10] >> Yeah. Uh I would like to add on top of [16:12] that as well as Hardik rightly said we [16:16] need to like uh in my experience uh we [16:19] uh I dealt with all these different uh [16:21] complex data types like videos, images, [16:24] um textual data, unstructured data. So [16:27] what we have done is we created uh each [16:30] separate uh you can say a pipeline where [16:32] it will generate its own um uh data in [16:36] the sense if I talk about um text data [16:39] where we created the tokenization and [16:41] then embeddings to capture the uh [16:43] context and similarly for a same thing [16:46] let's say if I if I'm talking about a [16:48] product uh in a retail company I I [16:51] wanted to build a model for that I'll [16:53] get the desript description which is in [16:54] the text format and similarly Okay, when [16:56] I get when I take the image, I can read [16:59] the features uh by extracting the [17:01] features out of the image and then again [17:02] create the embeddings on top on top of [17:05] it. Similarly for the video data and the [17:07] IoT data also. So when we have this [17:10] diverse different complex data types, we [17:13] need to have a each separate pipeline [17:15] you can say it and then combine together [17:18] together a normalized way where the AI [17:21] model can understand properly about that [17:24] product. uh for example and then it has [17:26] to give a proper inference. So for doing [17:30] that having each each separate thing is [17:33] essential to tackle these kind of [17:35] complex uh complex data types and along [17:37] with that as Hardik pointed out having [17:40] the schema or metadata metadata driven [17:43] things on the data it helps to evaluate [17:46] when a model being uh trained or when a [17:49] model being used in the live [17:51] environments. So [17:53] >> I think while decoupling I agree Shiva [17:55] it works or anything but there is [17:57] another approach like let's say you need [17:59] to have like one system or one pipeline [18:01] or anything. So in that case like you [18:05] need to have some kind of time stamp [18:07] inclusion or adding unique ids or things [18:10] to your different data that is flowing [18:12] through as a metadata or thing to make [18:14] sure you build that time series data [18:17] type or things. uh you can include uh [18:20] and there are other techniques also [18:21] things like late fusion of the data or [18:23] things which you can implement uh there [18:26] is like apply schema registry or schema [18:28] on read or things like that which would [18:30] help you for your data pipelines or [18:32] pipelines that you're building for [18:34] feature stores data science or whatever [18:36] it is [18:37] >> the time stamp I'm going to try for sure [18:39] you know [18:40] >> right so um Shiva based on what you were [18:42] saying processing uh each data type [18:46] individually and clearly there will be a [18:48] different model that is suitable for a [18:50] specific use case and the data type. So [18:53] you're suggesting that we the outputs of [18:56] one model can be input to another and we [18:59] just refine the entire workflow until we [19:03] have that expected predicted outcome. [19:06] >> Yes. Uh I can give one example. Let's [19:09] say we have embeddings right where we [19:11] can use different types of embedding [19:13] models like word to work or simple and [19:16] then bird model where it generates a [19:18] different uh embedded where it captures [19:20] the cont context properly right so we [19:23] need to define those so in order to do [19:25] it we need to have for each separate [19:29] complex data to have its own separate [19:32] model uh as in to give a proper [19:35] contextual [19:37] information to the model to train. So as [19:39] I said that is true [19:41] >> and the data the injection uh pipelines [19:44] and the data refreshes at a different [19:46] pace across different data types. So and [19:49] as Arjun had suggested I think model [19:51] schema registry would definitely help in [19:53] this case so that we are keeping track [19:55] of those changes and know exactly what [19:57] to leverage at what point. [19:59] >> Yes. [20:00] >> Yeah. I was saying like yeah make sure [20:03] before like merging or doing that your [20:05] data is clean or anything. So that is [20:07] also an important step to keep in mind [20:09] when you're merging your tree because [20:11] yeah garbage in garbage out as simple as [20:13] that right. [20:15] >> So at scale even small data quality [20:18] issues can have huge downstream impacts [20:21] on AI models. What practices or tools [20:24] have you found effective for maintaining [20:26] data quality, integrity and traceability [20:29] in large scale pipelines? [20:32] I think [20:33] one of the most important thing and I'm [20:35] sure everybody in the room has faced [20:37] this issue. Okay, building pipeline is [20:39] okay. You build it, you have it ready uh [20:42] you have it ready for your ML team, data [20:43] science team, anything they come back, [20:45] oh we have this issue, what should we [20:47] do? I have null values so I have no [20:49] values or anything. So one thing to keep [20:51] in mind when you're building your [20:53] pipelines or that infrastructure make [20:56] sure you have your data lineage like at [20:58] each stage or anything because uh with [21:02] now I think uh everything agent or [21:05] anything whatever you're using or things [21:08] building multiple tools and multiple [21:10] locations right I think lineage and [21:12] tracibility [21:14] would be more and more important to make [21:16] sure you get like proper alerting proper [21:18] monitoring okay You cannot do data [21:20] lineage on all of the things or [21:22] anything. Make sure you have like uh ROI [21:25] generated KPI or metrics that are [21:29] essential for your company for your [21:30] business. Ask that questions relevant [21:32] questions and make sure you put that [21:34] alerts and monitoring and uh data [21:36] traceability and lineage into your [21:38] pipeline at least the importance one. [21:40] And as Shiva said, if you use the [21:41] modular approach, segregating those [21:44] pipelines, what you can do is at least [21:47] make sure your important pipelines or [21:50] the relevance one add that data lineage [21:52] at every stage uh or anything. Maybe it [21:54] might be a cloud watch alerts or uh any [21:57] other alert that you can think of you [21:58] can put on KPIs which are essential. [22:01] Yeah, data lineage I would say is one of [22:03] the essential aspect. [22:06] >> Yeah. So uh along with that as uh Arjun [22:10] said um doing this validation checks at [22:14] each point of uh I mean at each point of [22:18] uh data uh in the sense when we get the [22:21] source data and then make it as Ajun [22:24] said rightly pointed out to data lineage [22:26] and from the source data till the model [22:28] use used it and consumed by the model uh [22:31] inference by application application [22:33] side as well. So to have the better um [22:37] to have the better inferences on the [22:39] model side having these data quality [22:42] checks at each every stage of um of our [22:45] model training or or or the data [22:47] injection process it definitely going to [22:49] help at the same time the data [22:51] validation checks like simple data [22:53] validation check whether are we having a [22:55] duplicate records duplicate records in [22:57] the sense if if when a model being [22:59] trained on the same records uh it will [23:02] be misleaded again right So similarly if [23:05] if I'm having like a arun set if I'm [23:07] having a null null data then again the [23:10] model will might not give a proper [23:12] inference which can lead to a in [23:15] incorrect [23:16] consumer application data usage right so [23:20] in order to uh overcome that challenges [23:23] having this small data quality checks at [23:26] the same time data validations before [23:28] the model training model building will [23:32] help and get the uh a better model to [23:35] have a better inference as well. [23:38] >> Yes. Uh I would uh add a little bit uh [23:40] on that uh because this is really good [23:43] question when we are handling like [23:45] federal projects the data quality uh [23:48] they just throw you know data row data [23:52] and uh when we say like oh you know this [23:55] is what uh you know redundancy this is [23:58] what imputation is required your R [24:01] square you know um value is this and [24:04] that they really don't care about it [24:06] they want to see whether we can you know [24:08] build AI model which they can use. So [24:12] now here is the thing like uh client [24:14] they really don't want to spend the data [24:17] and let's say if we want let's say we [24:19] complain like okay you know this data is [24:21] not useful we need to fix this if we [24:23] need to fix that mostly they don't want [24:26] to do that on your behalf uh before you [24:28] train AI model or do the data ingestions [24:31] so in that case what we do at our lab [24:35] and also like you know uh we spend most [24:37] of the time in data quality so uh other [24:41] than this feature engineering and data [24:44] imputation. We build our own statistical [24:47] model that we use to make sure that we [24:51] improve the quality of data before we [24:53] feed into uh data model. So here is the [24:57] thing like uh uh redundancy is one thing [25:00] but identifying a data which is useful [25:04] for the AI model is a critical when [25:07] there is a diversified of the data but I [25:10] would I would say like you know um yeah [25:12] Arjun and Shiva already mentioned the [25:14] techniques that you can definitely use [25:16] but at our lab what we do to maintain [25:19] the quality data in a pipeline we have [25:23] our own custom welldesigned [25:26] very calculated uh imputation technique [25:29] and data quality improvement techniques [25:31] and I would like to discuss more about [25:33] it but I can take it offline because [25:34] it's too much time. [25:35] >> Yeah and also uh it's a good point which [25:37] you raised on the statistical side as [25:39] well. Uh that that's absolutely true. uh [25:42] we we also use the similar approach to [25:46] identify any data drifts any data drifts [25:48] in the sense let's say if if I have a a [25:52] data related to a customer uh sales and [25:55] when I built a uh model data pipeline if [25:59] a data is drifting which means let's say [26:01] if I if I talk in the statistical terms [26:04] if the standard deviation is crossing [26:06] more than three uh plus or minus three [26:08] which means the data having some issues [26:11] right or or data might not following the [26:14] normal distribution way. So those kind [26:17] of checks will definitely uh helps to [26:20] identify those data drifts or data uh uh [26:22] data validation thing. And along with [26:24] that once the model has been trained and [26:26] once the model is in production and uh [26:29] it's been working in production there [26:31] might be a cases it's a it's a common [26:34] case where the model drift will happen. [26:36] model drift in the sense the model [26:38] whatever it is inferencing it is going [26:40] off from the actual what it is intended [26:43] so in that case we need to have a [26:45] regular ML retraining cadence so that [26:48] the model generates appropriate um uh [26:52] results yeah adding on to what Shiva [26:55] said in addition to that so I've I've [26:58] I've dealt with like TBS of data and [27:00] pabytes of data for a single table as [27:02] well I'm not talking like your entire [27:04] warehouse or things but single table. [27:07] though uh with my previous like social [27:09] media companies and things so which I [27:12] cannot speak much about it but [27:15] make the steps that we are talking [27:17] through I think this is one of the [27:19] important I cannot emphasis more like [27:21] data lineage or things because let's say [27:24] you have one TB of data or one pabyte of [27:26] data you have indexed it you have [27:29] partitioned it you have done parallelism [27:31] with it and it's there but if you find [27:33] an issue or anything with that to trace [27:36] sit back, think about like quering that [27:39] things, trying to find out that issue. [27:42] It takes time, efforts, resources and [27:45] two and fro between business folks and [27:48] you guys. I'm talking like actual [27:50] day-to-day work that would happen to [27:52] you, right? So [27:54] introducing like data lineage or [27:56] anything would reduce your time like [27:59] significantly as all of us are paneling [28:02] are emphasizing. I think this is an [28:03] important step building towards the end [28:06] I would say. [28:07] >> So having that proper traceability or [28:10] even to build those build those feedback [28:12] loops we we need to have uh proper [28:15] observability stack implemented and [28:18] while data processing the initial [28:20] exploration takes a lot of time from the [28:23] data scientist from the data science [28:25] teams. Do you suggest um transferring [28:30] some of this responsibility over to the [28:32] data engineering teams where they can uh [28:35] automate this process eventually that [28:37] feeds into your uh traceability uh [28:40] pipeline? [28:41] >> Yes, that's true. uh I can give one [28:43] example like uh uh sharing or [28:47] segregating the work on the retrieving [28:49] the data uh to a data engineering team [28:51] or ML engineering team you can say it uh [28:54] and model building or creating the I [28:57] models done by data science team there [28:59] is a distinction but along with that as [29:02] you asked about the traceability we can [29:04] still use I can give one example like [29:06] kibana kibbana where it can help us to [29:10] identify the traces on how the model is [29:13] working or how our data injection [29:15] process is working and how on how our [29:18] model is also doing a better job or not [29:21] where those checks can also be added to [29:23] that uh Kibbana dashboards along with [29:27] that we we have different um uh tools as [29:30] well like data bricks as in one uh tool [29:32] where it will have its own uh obser [29:35] obser observability and also the [29:37] traceability where we can check and [29:40] where we and see how each each step of [29:43] the from the uh source data till the [29:47] model been in inference each step how it [29:50] has been working and how it is [29:52] generating the uh results as well. So [29:54] there we can use the traceability as in. [29:57] >> So engineering does have a job now. [29:59] Okay, that's good. [30:00] >> Yeah, [30:01] >> right. So how do you ensure your [30:02] pipelines remain resilient to schema [30:05] evolution and the introduction of new [30:07] data types especially when AI models are [30:09] involved? Shiva, do you want to Okay. [30:12] Yeah, sure. So it's it's about the uh [30:15] schema evolution which is uh pretty much [30:19] required as we just before talked about [30:21] the data quality and data validation [30:24] things. So in order to have those [30:26] properly, to have those properly, we [30:29] need to have this schema uh evolution or [30:32] schema validation thing at each stage of [30:35] uh of our entire AI model life cycle [30:38] because it helps to identify where the [30:41] issue is because each each let as we [30:45] talk like different complex data types [30:47] which which we have discussed where we [30:50] can maintain a different schema for [30:53] metadata schema you can Okay or metadata [30:55] information you can say where it will uh [30:58] have those information whenever a new [31:00] data is been coming and whenever it is [31:03] been used for the model it is been [31:05] evaluated against the schema. If it is [31:08] not being evaluated then it will be a uh [31:10] it that data won't will not be used for [31:13] the model inference or model training. [31:15] So it's a essential part that needs to [31:18] be added for every AI model to be [31:20] trained or machine learning model that [31:22] needs to be created. [31:24] Uh yeah I mean I'll be real quick just [31:26] like in two lines. uh yeah uh the task [31:29] is 80% of uh data engineering and rest [31:32] of the things are you know pretty much [31:34] standard nowadays like Microsoft AWS [31:39] even though meta uh and open AAI they [31:42] are coming with their AI models which [31:44] are pretty much robust libraries I still [31:48] believe that you know my build custom [31:50] neural network is uh you know uh more um [31:55] quality than the Microsoft library but [31:59] no that's not the true case just because [32:01] you know I can build my own neural [32:03] network that doesn't mean that my uh you [32:05] know libraries are better than what [32:07] available you know existing but here is [32:10] the thing like when we talk about the [32:12] data we really need to spend the time so [32:16] to you know answer to that question I [32:18] would say like client [32:21] data engineer [32:23] data scientist [32:25] AI governance AI ethical team they all [32:28] need to work together to make sure that [32:30] we get uh appropriate output from the [32:33] model. [32:34] >> Oh no I I vouch for that. Yeah 100% [32:36] >> and also on top of it as uh Hardik [32:40] pointed out explanability also comes uh [32:43] a lot more uh important thing right now [32:46] as we are like I know most of our we are [32:50] using the AI models but are those models [32:53] are good enough to explain why that [32:56] inference or why that prediction is [32:58] generated and how that going to use it [33:01] is that prediction valid is that [33:04] prediction correct So in that sense [33:07] explanability plays a lot of important [33:10] things. So for that reason explanability [33:13] adopting the explanability is also at [33:16] most a important thing. Now most of the [33:19] major companies are trying to um use [33:22] different explanability approach maybe [33:24] counterfactuals casual models that can [33:27] help us to identify what the model [33:30] inference is what the model prediction [33:32] is to understand for a normal business [33:35] person because business person doesn't [33:37] know what the model is generating. [33:39] business person not even care about [33:41] what's the accuracy is. If the model is [33:43] giving 90 95 accuracy it doesn't you [33:46] don't want you don't want those accuracy [33:48] you just need what's the dollar value [33:50] that is being generated through that [33:52] model or through that through through [33:55] that model right so in order to do that [33:57] in order to understand that model [33:58] inference the explanability part plays a [34:01] pivotal role here [34:03] >> exactly and you know I say like in um [34:05] for the industry like healthcare yeah [34:08] explanability and interpretability is [34:10] the first thing that you need to [34:11] consider in your team before we move on. [34:14] >> So I think something I've done in my [34:16] previous jobs or anything is like data [34:18] contracts that helps a lot. Uh so I know [34:21] it takes time efforts and thing [34:23] involving multiple teams setting up the [34:25] expectations uh setting up the schemas [34:28] data size or anything. So having that [34:31] data contracts in place I think is is is [34:35] helpful as well in my opinion like yeah [34:38] if if you try to add on and give it a [34:40] shot like in your team or anything if [34:42] you have data contracts written or [34:43] anything that might help you guys. [34:46] >> In other words you're suggesting [34:47] federated data access for security [34:51] and compliance reasons. [34:53] >> Yes. [34:55] >> Right. And uh what advice would you give [34:57] to data engineers who are starting to [35:00] build pipelines designed to handle large [35:02] scale and complex data? [35:06] >> Yeah, I can I can so yeah uh we touched [35:09] upon this and we went like [35:10] chronologically. So yeah uh plan for [35:13] scalability, data quality and uh lineage [35:16] are uh the important factor as I said uh [35:20] and as you grow or anything make sure [35:22] you include like optimization and uh [35:25] techniques uh like data partitioning [35:27] parallelism [35:29] uh and go to the root of the things if [35:32] you have to pivot the technology as well [35:35] you can educate your team and reiterate [35:37] and see if you are able to reduce the [35:40] time what you're delivering. delivering [35:41] your features or anything and reiterate [35:43] and change the technologies and things [35:46] file formats uh compression techniques [35:48] and things and nowadays we have like AI [35:51] assisted things so make most of it and [35:54] see if you can use it to enhance or make [35:57] more efficient pipelines created. [36:00] [Music] [36:01] Uh yeah so uh uh in like so let's say in [36:04] in this room um you know let's say we [36:08] have like two uh different group of [36:10] people the one is like manager and [36:12] seniors and then one another is like a [36:14] junior junior who is like coming [36:16] upcoming data engineer for both of them [36:19] you know work together and to get best [36:22] out of it I would suggest that first you [36:26] know start with the small amount of data [36:29] which is less complex [36:31] If you are working as a data engineer [36:33] and then you know so uh recently I did [36:35] one project I was in Seattle this summer [36:38] and I did one you know a department of [36:39] energy project and they said oh we have [36:42] a huge data for you guys to play with I [36:45] said that's absolutely amazing but can [36:48] we have little bit small amount of data [36:50] to take a look at what is the data looks [36:53] like before we jump into the large [36:55] amount of data unstructured data [36:58] diversified data [37:00] Because let's say you know if a data [37:02] engineer especially recently graduate [37:05] comes to your team okay and you are a [37:08] CTO or you are a director of AI your [37:10] expectation is like okay they need to do [37:12] some miracle and get something magic out [37:14] of it you know that doesn't work that [37:16] way so I would suggest like you know um [37:19] in short when we start to deep diving [37:23] into the data engineering and building [37:25] scalable pipeline we start with the [37:27] small amount of data and then add [37:29] step-by-step complexity and do all the [37:32] tasks that we discuss in this panel. [37:34] Okay, quality check and then feature [37:38] engineering, filtering, uh ingestion, uh [37:43] finalizing after we have a appropriate [37:45] schema on time. So everything we need to [37:48] do that. Uh another thing that I would [37:50] say like when we are having data [37:53] engineering in our team they come with [37:56] the background that okay you know they [37:59] should have some set of field domain [38:02] expertise. It's not 100% true. So let's [38:05] say the person previously work in the [38:08] cyber security project. Now all of a [38:09] sudden that person needs to work on the [38:12] environmental remediation or let's say [38:14] nuclear energy. Okay. I mean they cannot [38:17] read about geoysics and nuclear energy [38:20] for 3 weeks and get expertise in that [38:22] domain. So definitely we need one domain [38:25] expert to work with a data engineering [38:27] that is I highly recommend. Yeah. Uh I [38:30] will add on top of it. So for any junior [38:33] data engineer who wants to pursue their [38:36] field u I mean who want to pursue as a [38:40] data engineer I the main thing I suggest [38:43] is try to understand the data. [38:46] the try to understand the data in the [38:47] sense to have a domain knowledge on the [38:50] data is very much at most important [38:54] thing to do any sort of what we [38:58] discussed so far. So in order to gain [39:00] that knowledge you need to have that [39:02] expertise or you need to have the [39:03] understanding on the uh domain. So once [39:06] you have the domain that we will get it [39:08] get it by using ba basic SQL knowledge [39:11] or maybe connecting with the peers to [39:14] understand that and along with that [39:16] having the knowledge on architectural [39:18] design architectural design in the sense [39:21] now the most of the major companies use [39:24] um uh lakehouse architecture there where [39:27] they uh manage end toend data where they [39:31] manage end toend data from the [39:33] applications from the source application [39:35] s to the warehousing. So if you have [39:38] that understanding then definitely it [39:40] will going to help them to have excel [39:43] help them to excel in in this field [39:46] just adding few points. So I think yeah [39:49] knowing the data is the first step. The [39:51] second step I would say is yeah go to [39:53] your co-pilot or anything in the [39:56] codebase see like which are the [39:57] pipelines running what is the existing [39:59] code base or anything try and understand [40:01] the complex pipelines and things in the [40:04] codebase or anything uh get your hands [40:06] dirty building one and do it like end to [40:10] end uh and use now more of this tool [40:13] like copilot or jamina whatever your [40:16] company as cursor or anything so make [40:19] sure you put the correct promp prompt [40:21] and things which tables it's referring [40:23] to what are the features being used [40:24] which is the model which is being used [40:26] in the existing codebase at 10 different [40:28] places. So you're getting like uh [40:31] smarter more efficient and resulting in [40:33] like uh quickly adapting and growing [40:37] through the code base which is existing [40:38] there because let's let's be honest like [40:41] one thing is for sure to stay like your [40:43] existing code base that's not going to [40:46] be replaced or anything. So you need to [40:48] get hands-on or anything existing [40:50] codebased something on top of knowing [40:52] your data sets or things. [40:54] >> Absolutely. And you know like a little [40:55] bit I want to add uh you know more 30 [40:58] secondond in last five years uh I have [41:00] been involved in different type of [41:02] projects uh cyber security physical [41:05] security using AI uh drug delivery and [41:08] remote healthcare monitoring using AI. [41:10] Uh renewable energy climate change and [41:13] environmental remediation using AI. [41:15] sustainable infrastructure building [41:17] using AI where IoT and sensors are [41:20] involved. One thing that I would say [41:23] that data engineer needs to be a good [41:27] friend of the domain expert. Okay. Yeah. [41:30] Don't try to you know uh be a good [41:32] friend with your manager and CTO because [41:35] you know if you will not get result you [41:37] know they will fire you immediately. But [41:39] the domain expert really helps. uh Shiva [41:42] mentioned about like AI based you know [41:44] lack uh house and other you know um [41:47] solutions available um we tried that but [41:51] in our case you know domain expert [41:54] working with domain expert always help [41:56] and solve over 90% of the problem [41:58] because they have spent multi- like you [42:01] know more than like five 10 years in the [42:03] field and they understand like what they [42:05] want to see out of the model so that's [42:07] helpful