Fidelity Engineer Pairs MLflow and W&B to Fix ML Data Versioning Gaps

Building Reproducible ML Pipelines with Mlflow & Weights & Biases With Fidelity Investments — Ai4

Ai4

Building production machine learning pipelines requires coupling MLflow's execution orchestration with Weights & Biases to overcome MLflow's lack of native dataset versioning and ensure complete experiment lineage.

AI-assisted summary of Building Reproducible ML Pipelines with Mlflow & Weights & Biases With Fidelity Investments — original source. Check the original for full context.

Source published: 2025-08-24

Detailed notes

## Background Published by Ai4 on YouTube on August 24, 2025, this presentation features Priyanka Asnani, a Senior ML Engineer at Fidelity Investments. The session demonstrates how to construct an end-to-end, reproducible customer churn prediction pipeline using MLflow Projects for orchestration and Weights & Biases for experiment tracking, artifact lineage, and data versioning.

## TL;DR Priyanka Asnani walks through building an end-to-end customer churn prediction pipeline using the Telco benchmark dataset and an XGBoost classifier. The system architecture uses MLflow Projects to modularize and orchestrate discrete data ingestion, cleaning, feature engineering, and model training stages into reproducible components. Weights & Biases manages immutable data and artifact versioning, hyperparameter sweep tracking, and visual lineage mapping across runs. Top-performing candidate models are standardized with explicit MLflow signatures and registered into a central model registry for containerized downstream deployment.

## Atmosphere & style The presentation carries a pragmatic, tutorial-like tone characteristic of an experienced practitioner delivering an enterprise tech talk. Priyanka speaks with structured clarity, methodically stepping through each pipeline component from raw data ingestion to deployment considerations. The pacing is efficient and focused on technical reproducibility, concluding with an open and candid Q&A session where she transparently discusses architectural trade-offs between MLflow and Weights & Biases.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=4s) Predicting customer churn enables businesses to take proactive retention measures, which is significantly more cost-effective than acquiring new customers. - [01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) Exploratory analysis on the Telco dataset identifies month-to-month contracts, fiber optic internet, higher monthly charges, and shorter tenures as key churn drivers. - [03:29](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=209s) The end-to-end ML lifecycle cleanly separates training (validation, preprocessing, training, registry) from inference (artifact packaging, Docker containerization, serving). - [04:36](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=276s) MLflow Projects provides reproducible, language-agnostic pipeline orchestration by encapsulating code, pinned dependencies, and execution parameters across environments. - [04:36](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=276s) Orchestrating sub-components via chained mlflow.run calls creates a modular directed acyclic graph (DAG) where outputs serve as inputs for subsequent steps. - [08:10](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=490s) Packaging transformations and the XGBoost classifier into a scikit-learn Pipeline prevents data leakage between training and validation splits. - [08:10](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=490s) Model metric selection must align with business goals: prioritizing recall minimizes missed churners, while prioritizing precision avoids wasted retention incentives. - [10:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=626s) Attaching model signatures and input examples via mlflow.sklearn.save_model enforces input/output schema contracts to prevent production serving failures. - [11:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=694s) Hydra sweeps automate hyperparameter grid searches across parameters like max_depth and n_estimators for systematic model optimization. - [11:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=705s) Weights & Biases provides run comparison dashboards and end-to-end artifact DAG visualizations linking model binaries directly to exact data versions and hyperparameters. - [14:01](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=841s) A centralized Model Registry acts as the single source of truth, organizing release candidates with staging/production tags for deployment and automated rollbacks. - [15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) The hybrid tooling approach pairs MLflow Projects for modular execution with Weights & Biases to overcome MLflow's lack of native dataset versioning. - [15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) Building a baseline reproducible tracking pipeline and experiment dashboard takes approximately two weeks using standardized template code.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=4s) Overview and the Business Case for Churn Prediction

Priyanka introduces the session on building reproducible machine learning training pipelines using MLflow and Weights & Biases (W&B). Her roadmap traces an end-to-end journey starting with the problem statement and dataset exploration, moving through system architecture and pipeline orchestration using MLflow Projects, and culminating in experiment tracking, hyperparameter optimization, and the model registry. The overarching focus centers on delivering reproducible, production-ready ML workflows.

The project frames customer churn as a supervised binary classification problem: predicting whether a given customer is likely to discontinue their subscription based on past behavior, account attributes, and service usage patterns. From a business perspective, retaining existing customers is "significantly more cost effective than acquiring new ones." By identifying high-risk accounts early, organizations can take proactive retention measures such as personalized outreach, targeted discounts, or plan modifications before a customer defects.

[01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) Exploratory Data Analysis of the Telco Dataset

The demonstration uses the Telco Customer Churn dataset, a publicly available benchmark from IBM containing records for over 7,000 customers. The data captures a mix of demographic data, account information, and service usage attributes, including contract type, internet service provider, and monthly charges, with a binary target label indicating whether the customer churned.

```mermaid flowchart LR accTitle: Key Churn Drivers in Telco EDA subgraph Drivers["Identified Churn Risk Factors"] C["Month-to-Month Contract (Shortest commitment)"] I["Fiber Optic Internet (Service dissatisfaction/pricing)"] M["Higher Monthly Charges (Price sensitivity)"] T["Shorter Tenure (Fragile onboarding period)"] end Drivers --> R["Elevated Churn Risk"] ```

Initial exploratory analysis reveals class imbalance, with significantly more retained customers than churned ones—a factor requiring explicit handling during modeling. Cross-referencing churn against specific features exposes clear behavioral patterns: - **Contract Type:** Month-to-month contracts exhibit the highest churn rate, confirming that customers without long-term commitments leave far more readily. - **Internet Service:** Subscribers with fiber optic service show a noticeably higher churn rate compared to those using DSL or lacking internet service, signaling potential dissatisfaction with pricing or connection reliability. - **Monthly Charges:** Customers paying higher monthly fees demonstrate elevated churn rates, pointing to direct price sensitivity. - **Tenure:** Those who churn typically have shorter tenures, indicating that early customer onboarding and initial experiences are "very critical for their retention."

[03:29](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=209s) High-Level ML Lifecycle: Training and Inference

Priyanka outlines the end-to-end machine learning lifecycle by splitting it into two distinct tracks: the training pipeline and the inference pipeline. The training lifecycle begins with data validation to ensure schema consistency and data quality, moves into preprocessing to transform raw inputs into model-ready features, proceeds through model training and evaluation, and ends by saving and registering vetted models in a model registry.

```mermaid flowchart TD accTitle: High-Level End-to-End ML Architecture subgraph TrainingPipeline["Training Track"] DV["Data Validation & Quality Checks"] --> DP["Data Preprocessing"] DP --> MT["Model Training & Evaluation"] MT --> MR["Model Registry"] end subgraph InferencePipeline["Inference Track"] MR --> PK["Package Artifacts into Docker"] PK --> DP_SERV["Deploy Model Service"] DP_SERV --> UI["Streamlit User Interface"] end ```

The inference pipeline pulls trained model artifacts from the registry, packages them into containerized Docker images, and deploys them as reachable services. While the hands-on project focuses strictly on training and tracking rather than running Docker or Streamlit directly, Priyanka incorporates these downstream stages into the architecture to illustrate how production systems package and expose models for real-time user inputs.

[04:36](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=276s) Orchestrating Pipelines with MLflow Projects

To avoid environmental drift and ad-hoc scripts, the pipeline leverages MLflow Projects. An MLflow Project provides an opinionated specification that encapsulates code, software environments, and execution parameters. Because dependencies can be declared via Conda or Docker, pipelines execute identically across local workstations and cloud compute. Furthermore, the format is CLI-based and language-agnostic, supporting Python alongside R or Julia across diverse engineering teams.

```mermaid flowchart LR accTitle: MLflow Project Pipeline DAG Execution M["main.py (Orchestrator)"] --> S1["get_data Component"] S1 -- "Raw Artifacts" --> S2["basic_cleaning Component"] S2 -- "Clean Artifacts" --> S3["training Component"] S3 -- "Model Artifacts" --> S4["evaluation Component"] ```

Orchestration is handled by a root-level `main.py` script that invokes discrete sub-components through chained `mlflow.run` calls. The execution graph operates as a directed acyclic graph (DAG), where each step consumes the logged output artifacts of the preceding step as its inputs. Structurally, the repository separates concerns cleanly: - **`MLproject` (Root):** Defines the pipeline entry point, execution environment, and configurable workflow parameters. - **Environment Configuration (`conda.yaml`):** Pins exact Python runtime versions and third-party library dependencies. - **`main.py`:** Orchestrates component execution order and manages artifact passing between stages. - **Component Subdirectories:** Isolated modules (`get_data`, `basic_cleaning`, `training`, `evaluation`), each equipped with its own dedicated `MLproject`, environment manifest, and execution script (`run.py`).

[07:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=446s) Data Ingestion and Basic Cleaning

The pipeline begins with a modular data acquisition and cleaning step. Raw data is downloaded directly from Weights & Biases artifact storage. The component immediately performs baseline data hygiene: dropping non-predictive identifiers, casting the `TotalCharges` feature to a numeric type, and discarding invalid or corrupted rows.

Once sanitized, the resulting clean dataset is uploaded back to Weights & Biases as a versioned artifact. This step ensures that every downstream training experiment is linked to an immutable, traceable data snapshot. Priyanka notes that while this cleaning takes place locally within the component script, enterprise production architectures would typically handle these aggregations within dedicated feature engineering pipelines integrated with a centralized feature store.

[08:10](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=490s) Feature Engineering and Model Training with XGBoost

Feature engineering groups incoming attributes into three explicit pipelines: binary, categorical, and numerical. For binary columns (such as gender and partner status), an `OrdinalEncoder` is applied. These transformations are consolidated using a scikit-learn `ColumnTransformer`, which is then packaged alongside an XGBoost classifier into a unified `Pipeline` object to prevent data leakage.

```mermaid flowchart LR accTitle: Feature Engineering and Training Pipeline subgraph DataPrep["Preprocessing (ColumnTransformer)"] B["Binary Features"] --> OE["OrdinalEncoder"] C["Categorical Features"] --> OHE["One-Hot / Imputer"] N["Numerical Features"] --> SC["Scaler / Imputer"] end DataPrep --> Pipe["scikit-learn Pipeline"] XGB["XGBoost Classifier"] --> Pipe Pipe --> Fit["Stratified Train / Val Fit"] Fit --> Log["Metric Logging (W&B)"] ```

Target values (`Churn`) are mapped from string values (`Yes`/`No`) to binary integers (`1`/`0`). Data splitting uses stratified sampling to preserve the target class distribution across training and validation sets. Once fitted, the pipeline is evaluated across key classification metrics: - **Metric Prioritization:** Accuracy, Precision, Recall, and F1-score are tracked, with the selection of the primary metric dictated by business goals: - If the organization aims to minimize false negatives (unnoticed churners), **recall** is prioritized. - If avoiding unnecessary retention incentives on non-churners (false positives) is paramount, **precision** takes precedence. - When balancing both errors equally, the **F1-score** acts as the primary evaluation target. - **Logging:** All validation scores are automatically recorded in Weights & Biases for experiment tracking.

[10:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=626s) Standardized Model Packaging and Artifact Tracking

To package the trained artifact, the pipeline utilizes `mlflow.sklearn.save_model`. This utility serializes the composite scikit-learn pipeline—encapsulating preprocessing, scaling, encoding, and the trained XGBoost estimator—into a standardized MLflow model directory format.

Critically, the export attaches a model signature defining expected input and output schemas, alongside an input example that serves as a schema test vector for inference requests. This contract prevents downstream failures:

> "This helps us prevent any schema mismatches in production environment."

After saving the model directory locally, the pipeline uploads the entire directory to Weights & Biases as an artifact. This dual-logging strategy links the versioned model binaries directly with run metadata, hyperparameter logs, and evaluation metrics, establishing full lineage.

[11:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=694s) Hyperparameter Tuning via Hydra Sweeps

To squeeze maximum performance from the model, the training component integrates Hydra sweeps. This configuration driver initiates an automated grid search over core XGBoost hyperparameters, running systematic sweeps across the parameter space to find the optimal combination.

[11:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=705s) Hyperparameter Sweeps and Weights & Biases Experiment Tracking

Running multiple training jobs with varying hyperparameter combinations—such as `max_depth`, `n_estimators`, and `subsample`—generates a large volume of runs that must be logged and systematically compared to identify the top performer. Weights & Biases provides an organized, transparent dashboard for experiment tracking and reproducibility, making workflows production-ready.

```mermaid flowchart LR accTitle: Experiment Tracking and Lineage Architecture accDescr: Diagram showing how training runs log metrics and hyperparameters into a W&B run dashboard and DAG lineage view. subgraph Sweep["Hyperparameter Runs"] R1["Run 1: max_depth, n_estimators"] R2["Run 2: max_depth, n_estimators"] R3["Run N: max_depth, n_estimators"] end subgraph WnB["Weights & Biases Tracking"] Dash["Run Dashboard • Accuracy, Precision, Recall • Training Time, Hyperparameters"] DAG["Artifact & DAG View • Input Datasets • Trained Model Artifacts • End-to-End Lineage"] end R1 --> Dash R2 --> Dash R3 --> Dash Dash --> DAG ```

In the W&B run dashboard, each row corresponds to a single model training run. Priyanka logs all core evaluation metrics (accuracy, precision, recall, and training duration) alongside the specific hyperparameter configurations used for each model. Input datasets and trained model binaries are logged and versioned as output artifacts. This artifact management is crucial for debugging: "If I go to my model a month later, which happens very often when we are building these machine learning pipelines, I know precisely which version of the dataset and what hyperparameters and configurations were used at the time of training." Additionally, the W&B Directed Acyclic Graph (DAG) view maps out the complete experiment lineage, showing how raw inputs flow through each component and what artifacts are produced at each stage.

- Key metrics tracked across runs: accuracy, precision, recall, and training execution time. - Key hyperparameters logged: `max_depth`, `n_estimators`, and `subsample`. - Full lineage visibility: Input data versions, code, configs, and output models linked through DAG visualization.

---

[14:01](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=841s) Model Versioning, Model Registry, and Deployment Flow

Once training and evaluation finish, each model is versioned alongside its corresponding code, configuration files, and artifacts to enable historical performance comparison. When a model is selected as the top performer, it is tagged with an environment label (such as `staging` or `prod`) and registered into the Model Registry.

```mermaid flowchart TD accTitle: Model Promotion and Deployment Pipeline accDescr: Flowchart illustrating the promotion of a top model to the registry, containerization with Docker, and deployment. subgraph Selection["Model Selection & Registry"] Best["Top Performing Model"] --> Tag["Tag: 'staging' / 'prod'"] Tag --> Reg["Model Registry (Single Source of Truth)"] end subgraph Deploy["Downstream Deployment"] Reg --> Pull["Pull Model Artifacts & Test on Prod Data"] Pull --> Dock["Containerize via Docker"] Dock --> Svc["Deploy as Microservice"] end ```

The Model Registry serves as the central single source of truth across the engineering team, holding approved release candidates ready for deployment, continuous monitoring, or rollbacks. In the downstream release flow, engineers pull the registered model artifact, validate it against production data, containerize the serving application with Docker, and deploy it as a live service.

- Tagging scheme: Environment markers like `staging` and `prod` categorize candidate models. - Central registry functions: Acts as a single source of truth for deployment, monitoring, and automated rollbacks. - Containerization: Packaging the verified model artifact with Docker to serve inference requests.

---

[15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) Q&A: Production Monitoring, MLflow vs. W&B, and Setup Timelines

Following the presentation, Priyanka opens the floor to audience questions regarding production validation, tooling trade-offs, and implementation timelines.

```mermaid flowchart LR accTitle: Tooling Comparison MLflow vs Weights & Biases accDescr: Visual mapping of features supported by MLflow versus Weights & Biases. subgraph MLflow["MLflow Core Strengths"] P1["MLflow Projects (Pipeline Packaging)"] P2["Experiment Tracking & Registry"] end subgraph WnB["Weights & Biases Core Strengths"] W1["Native Data Versioning"] W2["Experiment & Metric Tracking"] W3["Lineage Artifact DAGs"] end ```

When asked whether the model is currently active in production and how it has performed, Priyanka clarifies that this was built as a personal project. While the pipeline was deployed, dedicated post-deployment monitoring was not built out for this presentation. She highlights that the critical next operational step would be implementing automated monitoring for data drift and model drift to prevent prediction degradation over time.

An audience member asks why Weights & Biases was selected alongside MLflow when MLflow already includes native experiment tracking and artifact logging. Priyanka explains that MLflow's major limitation is its lack of native dataset versioning: "MLflow doesn't support data versioning, and I wanted one tool that supports everything from data versioning to experiment tracking to having a model registry." While MLflow can log datasets, it does not version them out of the box. Consequently, she opted for a hybrid architecture: MLflow Projects packages and orchestrates the pipeline steps, while Weights & Biases handles data versioning, artifact lineage, and experiment tracking.

Finally, an attendee asks about the time required to adopt and implement this stack from scratch. Priyanka notes that while initial architecture design and automation took experimentation, adopting the approach using template code (which she will make available on GitHub) should take an engineer roughly two weeks to set up a baseline pipeline and experiment tracking dashboard. She emphasizes the necessity of this tracking infrastructure because machine learning is inherently iterative, and teams frequently need to backtrack to earlier high-performing configurations without losing the exact data and parameter lineage that generated them.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=4s) significantly more cost effective than acquiring new ones. — Priyanka Asnani

> [01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) very critical for their retention. — Priyanka Asnani

> [10:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=626s) This helps us prevent any schema mismatches in production environment. — Priyanka Asnani

> [11:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=705s) If I go to my model a month later, which happens very often when we are building these machine learning pipelines, I know precisely which version of the dataset and what hyperparameters and configurations were used at the time of training. — Priyanka Asnani

> [15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) MLflow doesn't support data versioning, and I wanted one tool that supports everything from data versioning to experiment tracking to having a model registry. — Priyanka Asnani

## Mentioned Priyanka Asnani, Fidelity Investments, Ai4, MLflow, Weights & Biases, Hydra, XGBoost, IBM, Telco Customer Churn, scikit-learn, Docker, Streamlit, Conda, Python, R, Julia, GitHub

## Video description Priyanka Asnani, Senior ML Engineer, Fidelity Investments

Abstracts

In this session, Priyanka Asnani will walk you through building an end-to-end, reproducible customer churn prediction pipeline powered by MLflow Projects and Weights & Biases.

We’ll start with the problem statement, dataset overview, and exploratory data analysis before diving into the high-level system design. You’ll learn how to structure ML pipelines using MLflow Projects, chain components effectively, and leverage Hydra sweeps for hyperparameter optimization.

Key components—data cleaning, feature engineering, XGBoost training, experiment tracking, and model registry—will be demonstrated with practical examples. Along the way, we’ll show how to log and track artifacts, metrics, and models using both MLflow and Weights & Biases, and how to register the best model for deployment.

Whether you’re new to MLops or looking to sharpen your pipeline skills, this session offers actionable insights for building scalable, collaborative, and production-ready machine learning workflows.

[Source](https://www.youtube.com/watch?v=8vneD7DPtWQ)

中文简报

富达工程师结合 MLflow 与 W&B 弥补机器学习数据版本管理缺陷构建生产级机器学习流水线需要将 MLflow 的执行编排能力与 Weights & Biases 相结合，以克服 MLflow 缺乏原生数据集版本控制的短板，并确保完整的实验血缘追溯。 Priyanka Asnani 详细介绍了如何利用 Telco 基准数据集和 XGBoost 分类器构建端到端客户流失预测流水线。该系统架构使用 MLflow Projects 将独立的数据摄取、清洗、特征工程和模型训练等阶段模块化并编排为可复现的组件。Weights & Biases 负责管理不可变数据与工件版本控制、超参数寻优跟踪以及跨运行的可视化血缘映射。表现最佳的候选模型会通过显式 MLflow 签名进行标准化，并注册到中央模型注册表中，用于容器化的下游部署。预测客户流失使企业能够采取主动的挽留措施，这远比获取新客户更具成本效益。 (0:04) 对 Telco 数据集的探索性分析表明，按月合同、光纤网络、较高的月费以及较短的在网时长是导致客户流失的关键驱动因素。 (1:34) 端到端机器学习生命周期将训练阶段（验证、预处理、训练、注册）与推理阶段（工件打包、Docker 容器化、服务部署）清晰地分离开来。 (3:29) MLflow Projects 通过跨环境封装代码、固定依赖项和执行参数，提供了可复现且与语言无关的流水线编排能力。 (4:36) 通过链式调用 mlflow.run 编排各子组件，可构建出模块化的有向无环图（DAG），其中前序步骤的输出将作为后续步骤的输入。 (4:36)

Reading summary

Priyanka outlines an end-to-end framework for reproducible machine learning by combining MLflow's execution orchestration with Weights & Biases' native data versioning and experiment tracking. Using customer churn prediction as a benchmark, she shows how modular directed acyclic graphs and schema contracts eliminate environmental drift and maintain traceable provenance. Churn Risk Indicators and Metric Prioritization (0:04) Using IBM's Telco Customer Churn dataset of over 7,000 customer records, the project frames retention as a supervised binary classification task. Exploratory data analysis highlights significant class imbalance alongside clear behavioral drivers of churn: month-to-month contracts, fiber optic internet subscriptions, higher monthly charges, and shorter tenures all correlate with elevated attrition. Because retaining existing customers is significantly more cost-effective than customer acquisition, early detection allows businesses to intervene with targeted outreach, discounts, or plan changes. Priyanka emphasizes that evaluation metrics must directly reflect organizational priorities. When the business goal is to minimize unnoticed churners (false negatives), recall is prioritized. If the organization wants to avoid wasting retention offers on customers who were not planning to leave (false positives), precision takes precedence. When balancing both types of error equally, the F1-score serves as the primary optimization target. Modular Pipeline Orchestration and Schema Validation (4:36) To prevent environmental drift across local workstations and cloud environments, the architecture organizes execution using MLflow Projects. A root-level main.py orchestrates a directed acyclic graph (DAG) across isolated subdirectories—get_data, basic_cleaning, training, and evaluation—each containing its own MLproject file, execution script, and pinned Conda environment. Chained mlflow.run invocations ensure that each module executes sequentially, consuming output artifacts logged by the preceding stage. Within the training module, feature transformations (such as ordinal encoding for binary variables) are assembled alongside an XGBoost classifier into a unified scikit-learn Pipeline to prevent data leakage. The model is exported using mlflow.sklearn.save_model with an explicit signature and input example. Priyanka notes that this standardized schema contract defines expected inputs and outputs, preventing downstream inference failures caused by schema mismatches in production environments. Systematic Hyperparameter Sweeps and Lineage Tracking (11:45) Hyperparameter optimization is managed using Hydra sweeps to execute grid searches across XGBoost parameters, including max_depth, n_estimators, and subsample. Rather than relying on manual tracking, each training job logs evaluation metrics—accuracy, precision, recall, and training runtime—alongside hyperparameter configurations directly into a Weights & Biases (W&B) dashboard for side-by-side comparison. W&B simultaneously versions input datasets and serialized model binaries as artifacts. Its DAG view visually maps the lineage connecting raw datasets, intermediate data snapshots, execution code, and final models. Priyanka highlights that this artifact management enables engineers to inspect a model months after deployment and identify the exact dataset version and parameter set used during training. Hybrid Tooling Strategy and Model Promotion (14:01) Addressing why both tools are necessary, Priyanka explains that while MLflow excels at environment packaging and pipeline orchestration, it lacks native, out-of-the-box dataset versioning. Weights & Biases was selected to fill this gap, providing native dataset versioning, artifact lineage, and metric dashboards while MLflow Projects governs execution modularity. Once evaluation identifies a top-performing model, it is labeled with environment tags like 'staging' or 'prod' and cataloged in a Model Registry. The registry acts as the single source of truth across engineering teams. Downstream deployment workflows pull vetted models from the registry, test them against production data, and package them into Docker containers to serve live inference requests. Priyanka notes that the demonstrated pipeline was created as a personal demonstration project without live post-deployment monitoring; she stresses that full operationalization requires implementing automated checks for data drift and model drift. She estimates that adopting this template-based architecture from scratch takes an engineer roughly two weeks to establish baseline orchestration and tracking.

精炼阅读摘要

Priyanka 阐述了如何结合 MLflow Projects 的模块化编排与 Weights & Biases 的版本控制及追踪能力，构建完全可复现且满足生产标准的客户流失预测流水线。业务目标定义与客户流失特征分析 (0:04) 项目将客户流失界定为监督式二分类问题，基于用户行为、账户属性与服务使用历史预测退订风险。演讲者指出，留存既有客户的成本显著低于获取新客，早期识别高风险账户便于企业实施精准干预。针对 IBM 提供的 7,000 余条 Telco 开源数据集分析表明，样本存在明显的类别不平衡。特征分析显示，采用按月订购合同、使用光纤宽带、月费金额偏高以及在网时间较短的客户流失率显著更高。为兼顾不同业务需求，评估指标需进行动态权衡：若以减少漏判流失客户为目标应优先看重召回率（Recall），若为避免向非流失客户发放多余挽留优惠则侧重精确率（Precision），而在两者权重相当时则以 F1 分数作为核心评价标准。模块化流水线解耦与计算环境隔离 (4:36) 针对机器学习研发中常见的环境漂移与脚本散乱问题，系统借助 MLflow Projects 实现流程的标准化封装。架构将整体生命周期解耦为独立的训练链路与推理链路，并通过 Conda 依赖文件（conda.yaml）精确锁定 Python 运行时与第三方库版本，确保流水线在本地机器与云端计算节点之间具备完全一致的执行表现。流水线的执行由位于根目录的 main.py 统一调度，采用链式调用形成有向无环图（DAG）。项目将数据获取、基础清洗、模型训练与评估拆分为相互隔离的子组件目录，各组件通过输入输出工件形成依赖，实现了各阶段职责的明确划分与可维护性。特征管道封装与生产契约管理 (8:10) 数据清洗组件在去除无关标识符并处理异常字段后，将清洗成果作为版本化数据工件重新上传记录。为防止特征转换阶段的数据泄露，流水线利用 scikit-learn 的 ColumnTransformer 对二元特征进行序数编码，并组合数值与类别处理分支，将预处理逻辑与 XGBoost 分类器打包在同一个 Pipeline 对象内，配合分层抽样保持训练与验证集的类别分布一致。在模型保存环节，系统通过 mlflow.sklearn.save_model 导出标准目录，并显式附加模型签名（Model Signature）与输入测试样例。演讲者指出，这种契约化定义明确规范了模型对输入输出数据结构的约束，能够有效杜绝生产环境中因特征模式不匹配（Schema Mismatch）导致的服务故障。混合架构下的超参数优化与血缘追溯 (11:45) 系统通过集成 Hydra sweeps 对 XGBoost 的核心超参数（包括最大深度、估算器数量和采样比例）实施自动化网格搜索。针对在 MLflow 已具备指标记录功能下引入 Weights & Biases 的原因，Priyanka 解释称 MLflow 的主要短板在于缺乏开箱即用的原生数据集版本控制，而研发团队需要统一管理数据版本、实验参数与模型产物。因此系统采用混合架构：由 MLflow Projects 承担管道组织与执行封装，Weights & Biases 负责全面记录训练指标、运行耗时、输入数据快照及生成模型。借助 W&B 的 DAG 拓扑视图，工程师在数月之后排查问题时，依然能够精准复现当时的原始数据版本与超参数配置，确保端到端实验血缘的完整透明。演讲者总结指出，经过评估的最优模型将被打上 staging 或 prod 标签归档至模型注册中心，作为微服务部署的唯一定型产物。该系统目前属于个人概念验证项目，尚未建立针对生产环境数据漂移与模型漂移的自动化持续监控机制；基于现有模板代码落地该套架构，工程人员预计需要约两周时间完成基线搭建与仪表板集成。

Detailed notes — zh

## Background 该视频由频道 Ai4 于 2025 年 8 月 24 日在 YouTube 平台发布。视频由来自富达投资（Fidelity Investments）的高级机器学习工程师 Priyanka Asnani 主讲，主要介绍了如何结合使用 MLflow Projects 与 Weights & Biases 构建端到端、可复现的客户流失预测机器学习流水线，涵盖从数据分析、组件编排到超参数优化与模型注册的全流程。 ## TL;DR 分享以电信客户流失预测为实战案例，详细展示了如何利用 MLflow Projects 将数据清洗、特征工程和 XGBoost 建模拆分为标准化、解耦的流水线步骤。系统通过 Hydra Sweeps 开展自动化超参数网格搜索，并依托 Weights & Biases 实现完整的数据集版本控制、指标追踪和依赖血缘可视化。最终，经过评估的最优模型将被推入模型注册表作为单一真实信任源，为后续 Docker 容器化与服务化部署提供支撑；问答环节还深入对比了 MLflow 与 W&B 的互补职责并探讨了生产监控策略。 ## Atmosphere & style 整场演讲呈现出清晰、严谨且高度实战化的工程技术风格。主讲人节奏沉稳、条理分明，通过详实的架构逻辑与流程图解，将复杂的 MLOps 理论拆解为具体可落地的代码与配置结构。问答环节坦诚务实，直面个人原型的局限性与工具选型取舍，为观众营造了专业且极具启发性的技术交流氛围。 ## Key takeaways - [00:50](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=50s) 企业留存现有客户的成本显著低于获取新客户，通过将客户流失建模为二分类问题并提前识别高风险群体，业务团队可及时采取针对性干预措施降低收入损失。 - [01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) 探索性数据分析（EDA）表明客户流失存在明显的类别不平衡，且按月签约、使用光纤宽带、账单金额偏高及在网时间短的客户具有显著更高的流失倾向。 - [03:29](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=209s) 端到端机器学习架构将训练流水线与生产推理流水线清晰解耦，模型经训练评估合格后注册入库，随后通过 Docker 容器化打包对外提供实时预测服务。 - [04:36](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=276s) 采用 MLflow Projects 标准对各子任务进行模块化封装，通过 conda.yaml 锁定依赖并在 main.py 中编排 DAG，能保证跨机器与跨团队调度的强确定性与高可复现性。 - [07:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=446s) 使用 ColumnTransformer 针对二值、多分类及数值特征执行差分预处理，并将其与 XGBoost 统一封装为 Scikit-Learn Pipeline，可从根本上杜绝数据泄露并保证特征转换一致性。 - [08:47](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=527s) 针对类别不平衡需启用分层抽样（Stratified Split），并在评估阶段根据业务倾向权衡 Precision（降低误报折扣成本）与 Recall（减少客户流失漏报）。 - [10:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=626s) 在持久化模型时显式定义模型签名（Model Signature）与输入样例，并在训练阶段结合 Hydra Sweeps 声明式网格搜索，可有效防止生产推理输入异常并最大化模型性能。 - [11:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=705s) Weights & Biases 仪表盘记录了每次运行的超参数、评估指标与工件，配合 DAG 血缘依赖视图，确保工程师在长期迭代中能精准回溯任一历史模型的完整上下文。 - [14:01](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=841s) 模型注册表（Model Registry）作为团队协作的单一真实信任源，通过 staging/prod 标签管理候选模型晋级，极大简化了下游容器化部署、监控与安全回滚流程。 - [15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) 在架构选型上，MLflow Projects 专精于任务模块化编排与模型打包，而 Weights & Biases 补足了端到端数据版本控制（Data Versioning）、实验对比与血缘追踪的短板。 - [18:54](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=1134s) 机器学习研发具有高频迭代特性，在两周内跑通标准化基线流水线的同时，必须配套严格的数据与超参数版本控制，防止新版本退化时无法定位最优配置。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=4s) 演讲背景与议程概述来自富达投资（Fidelity Investments）的高级机器学习工程师 Priyanka Asnani 带来了关于利用 MLflow 和 Weights & Biases（W&B）构建可复现机器学习训练流水线的分享。在快速发展的生产环境中，机器学习项目常常面临环境不一致、实验难以追踪以及难以平滑部署等挑战，而一套标准化、模块化的流水线架构是解决这些痛点的核心。本次分享涵盖了端到端机器学习生命周期的关键环节：首先明确客户流失预测的业务背景与数据集情况，随后展示探索性数据分析（EDA）与高层次系统架构；重点剖析如何使用 MLflow Projects 构建与编排训练流水线的各个独立组件——涵盖数据预处理、特征工程、XGBoost 模型训练及超参数调优，最后阐述实验追踪（Experiment Tracking）与模型注册表（Model Registry）的落地实践，全方位展示生产级 ML 工作流的构建过程。 ### [00:50](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=50s) 业务问题定义与客户留存价值本次项目聚焦的核心问题是：“我们能否根据客户的历史行为、账户信息以及服务使用模式，准确预测其是否可能流失？”该任务被建模为标准的二分类（Binary Classification）问题，目标变量明确标注客户是否流失（Churn / Not Churn）。从商业价值角度来看，解决这一问题对企业至关重要。维护和留存现有客户的成本显著低于获取新客户的获客成本（CAC）。通过在早期准确识别出高流失风险的客户群体，业务团队可以及时采取针对性的干预措施，例如开展个性化回访、提供专项折扣或调整服务方案，从而有效提升客户留存率并减少收入流失。 ### [01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) 电信客户流失数据集与探索性数据分析（EDA）该项目选用了经典的公开数据集——由 IBM 提供的 Telco Customer Churn 数据集。该数据集包含超过 7,000 条客户样本记录，融合了用户人口统计学特征、账户属性以及各项服务的使用模式（如合约类型、月度费用、网络服务类型等）。在深入建模之前，团队通过探索性数据分析（EDA）揭示了数个关键业务洞察： - **类别不平衡（Class Imbalance）**：绝大多数客户选择留存，流失客户占比较小。这一分布特性在后续的模型设计、分层抽样与评估指标选择中得到了重点处理。 - **合约期限与流失率**：按月签约（Month-to-month）的客户流失率显著最高，直观反映出短期合约客户粘性较低、随时可能离开。 - **网络服务类型差异**：使用光纤宽带（Fiber Optic）的客户流失率明显高于使用 DSL 或无网络服务的客户，可能暗示了用户对光纤网络的价格敏感性或对服务稳定性的不满。 - **月度账单金额（Monthly Charges）**：账单金额较高的客户流失倾向更高，表明价格敏感度是引发流失的重要驱动因素。 - **在网时长（Tenure）分布**：流失客户的在网时长普遍较短，印证了客户生命周期初期的入职与使用体验对于长期留存具有决定性作用。 ### [03:29](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=209s) 端到端 ML 系统架构与生命周期高层次的机器学习系统架构被划分为两条核心流水线：训练流水线（Training Pipeline）与推理流水线（Inference Pipeline）。训练端从数据校验和模式一致性检查开始，随后进入数据预处理生成模型就绪特征，经由模型训练与评估后，达到上线性能标准的模型将被打包并推送到模型注册表。而在生产推理端，服务会拉取注册的模型工件，封装进 Docker 容器并部署为 API 服务，最终对接 Streamlit 等前端交互界面为最终用户提供实时流失预测。虽然本次演讲的代码重点聚焦于核心训练流水线，但整体架构完整映射了生产环境的标准落地形态。 ```mermaid flowchart TD accTitle: End-to-End ML Pipeline Architecture accDescr: High-level overview of Training and Inference pipelines using MLflow and Weights & Biases. subgraph TrainingPipeline ["训练流水线 (Training Pipeline)"] A["原始数据 / W&B Artifacts"] --> B["数据清洗与校验 (Basic Cleaning)"] B --> C["特征工程与转换 (Pre-processing)"] C --> D["XGBoost 训练与超参优化 (Hydra Sweeps)"] D --> E["模型评估与指标记录 (W&B Metrics)"] E --> F["MLflow 模型打包与注册 (Model Registry)"] end subgraph InferencePipeline ["推理流水线 (Inference Pipeline)"] F -.-> G["加载已注册模型工件"] G --> H["Docker 容器化服务封装"] H --> I["Streamlit 前端 / 实时预测 API"] end ``` ### [04:36](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=276s) 基于 MLflow Projects 的模块化流水线设计为了实现高复现性与跨团队协作，项目全面采用 MLflow Projects 标准对各阶段任务进行封装。MLflow Projects 能够严格固化代码、运行环境依赖以及输入参数，保证无论在本地调试还是在云端集群执行，都能获得确定性的运行结果。该工具具备高度灵活性与语言无关性，支持 Conda 与 Docker 等运行环境定义，不仅适配 Python，还可轻松调用 R、Julia 等任何支持命令行界面的工具。整个流水线的执行流在 `main.py` 入口文件中通过 `mlflow.run()` 进行编排，构建出有向无环图（DAG）结构。上游步骤生成的输出工件（Artifacts）会作为下游步骤的输入参数传入，确保了组件间明确的依赖关系与数据流转。项目目录采用了清晰解耦的模块化结构： - **根目录 `MLproject`**：作为整个流水线的全局入口，定义项目名称、全局环境配置以及各子阶段的调用参数与入口点。 - **环境描述文件 `conda.yaml`**：锁定 Python 版本及所有依赖包的具体版本，确保异构机器与不同团队成员间环境的一致复现。 - **调度主程序 `main.py`**：调用 `mlflow.run()` 串联各独立步骤，负责在 DAG 节点间传递工件 URI 与配置。 - **独立组件子目录**：每个阶段（如 `get_data`、`basic_cleaning`、`train`、`evaluate`）均独立成文件夹，内部各自包含专属的 `MLproject`、环境配置文件以及承载具体业务逻辑的 `run.py`。这种模块化设计让每个步骤都能被独立隔离、单独调试与重复运行。 ### [07:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=446s) 数据清洗与 Scikit-Learn 特征工程管道流水线的第一步是数据获取与基础清洗阶段。系统从 Weights & Biases 中拉取原始数据集，执行一系列规范化操作：剔除客户 ID 等无预测意义的标识字段、将 `TotalCharges`（总费用）从字符串强制转换为数值型，并过滤无效或缺失行，最终将清洗后的数据集作为新版本 Artifact 重新上传至 W&B，保障下游训练的可溯源性。在实际生产场景中，这些聚合处理通常会归入特征工程流水线并存入特征存储库（Feature Store）。在训练组件内部，特征被系统性地划分为三类：二值特征（Binary）、多分类特征（Categorical）与数值特征（Numerical）。 - 对于性别（Gender）、是否有伴侣（Partner）等二值列，直接采用 `OrdinalEncoder` 转换。 - 利用 `ColumnTransformer` 将各特征子集的预处理管道进行统一组装，实现特征编码、缺失值插补与标准化。 - 将组合后的 `ColumnTransformer` 与 XGBoost 分类模型进一步封装成一个统一的 Scikit-Learn Pipeline（`Pipeline` 对象），避免数据泄露并保证特征变换逻辑与模型推理逻辑的强绑定。 ### [08:47](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=527s) 分层训练、模型评估与 W&B 追踪记录在训练执行阶段，程序首先提取目标变量 `Churn`，将二元字符标签（Yes/No）显式映射为数值（1/0）。为了应对数据集中存在的类别不平衡问题，数据集划分时启用了分层抽样（Stratified Split），确保训练集与验证集具有一致的流失标签比例。随后，完整的推理流水线在训练集上进行拟合，并在验证集上运行预测以计算核心评估指标： - **准确率（Accuracy）**、**精确率（Precision）**、**召回率（Recall）** 以及 **F1 分数（F1-score）**。 - 指标权衡考量：如果业务目标侧重于减少假阴性（漏报流失客户），则优先优化 Recall；如果目标是降低假阳性（避免给未流失客户误发高额折扣），则关注 Precision；若需要兼顾两者，则以 F1-score 为核心优化指标。 - 所有计算出的度量指标均实时记录至 Weights & Biases 平台，实现指标轨迹的完整可追溯。 ### [10:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=626s) MLflow 模型持久化与 Hydra 超参搜索完成模型训练与评估后，系统使用 `mlflow.sklearn.save_model` 方法将整个 Scikit-Learn Pipeline（包含预处理逻辑与 XGBoost 模型主体）保存为标准的 MLflow 目录格式。该保存机制提供了两项关键生产保障： 1. **模型签名（Model Signature）**：严格定义预期输入与输出的 Schema，在生产推理阶段防止特征缺失或数据类型不匹配引发的异常。 2. **输入示例（Input Example）**：提供基准输入样例，作为后续推理请求与测试的格式参考。 ``` mlflow.sklearn.save_model( sk_model=pipeline, path="model_dir", signature=signature, input_example=input_example ) ``` 本地保存的模型目录随后作为 W&B Artifact 上传，与超参数、训练日志及评估指标一同完成多维度版本控制。为了进一步压榨模型性能，系统引入了 Hydra Sweeps，通过声明式配置对 XGBoost 的超参数空间执行系统化网格搜索（Grid Search），自动触发多组超参数组合的高效迭代。 ### [11:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=705s) 实验追踪与超参数调优记录在执行多次训练运行（Runs）时，每次运行都会尝试不同的超参数组合，例如树的最大深度（`max_depth`）、估计器数量（`n_estimators`）以及子采样比例（`subsample`）。通过记录所有运行的参数与指标，团队能够直观对比所有模型的性能，并高效筛选出表现最优的模型。 Weights & Biases 提供了强大且结构化的方式来追踪与复现实验，使整个机器学习工作流更加透明并达到生产就绪状态。在 Weights & Biases 的运行仪表盘（Run Dashboard）中，每一行对应一次独立的模型训练运行，详细记录了准确率（Accuracy）、精确率（Precision）、召回率（Recall）以及训练耗时（Training Time）等核心指标。这种视图让工程师可以清晰对比随时间推移训练出的各类模型，并依据关注的指标快速锁定最佳候选。 ```mermaid flowchart LR accTitle: 实验追踪与血缘图 subgraph Tracking["Weights & Biases 运行记录"] R1["Run 1: max_depth / n_estimators"] R2["Run 2: max_depth / n_estimators"] M["核心指标: Accuracy / Precision / Recall"] R1 --> M R2 --> M end subgraph Lineage["DAG 血缘视图"] D["输入数据集版本"] --> T["训练步骤 Run"] T --> A["输出模型 Artifact"] end Tracking --> Lineage ``` - **超参数追踪**：完整记录每次运行的训练参数（如 `max_depth`、`n_estimators` 等）。 - **版本控制与工件归档**：输入数据集与训练好的模型均作为工件（Artifacts）进行版本化记录。 - **可复现性保障**：即使一个月后重新审视模型，也能准确获知当时使用的数据集版本、超参数和环境配置，这对于故障排查和完全复现至关重要。 - **DAG 依赖血缘视图**：通过 DAG 可视化展示实验的全流程血缘关系，清晰呈现每一步运行对应的数据输入与产出的工件输出。 --- ### [14:01](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=841s) 模型注册表与服务化部署完成模型训练与评估后，系统会将模型推送到仓库中。在 Weights & Biases 中，每个训练好的模型都与其关联的代码、配置文件和数据工件进行严格的版本绑定，从而支持对模型历史性能进行长期跟踪与深度对比分析。 ```mermaid flowchart LR accTitle: 模型晋级与部署流程 subgraph Staging["模型版本控制与标记"] V["训练完成的模型 Artifact"] --> T["打标签 (staging / prod)"] end subgraph Registry["模型注册表 (单一信任源)"] T --> MR["Model Registry"] end subgraph Deployment["生产服务化"] MR --> P["拉取模型工件 & 生产测试"] P --> D["Docker 容器化"] D --> S["部署为在线服务"] end ``` 当确定最佳模型后，开发人员会为其打上对应环境的标签（例如 `staging` 或 `prod`），并通过界面将标记的模型关联到模型注册表（Model Registry）中。该步骤标志着该模型正式成为生产候选。模型注册表作为全团队唯一的“单一真实信任源”（Single Source of Truth），集中存放经过最终审核批准、准备用于下游任务（如部署、监控和回滚）的模型，避免团队协作中产生版本混乱。 - **模型打标与晋级**：通过打上 `staging` 或 `prod` 标签，将优质模型链接至 Model Registry。 - **单一信任源**：统一团队协作基准，便于后续执行部署、实时监控与安全回滚（Rollback）。 - **容器化部署流程**：从注册表中拉取模型工件并在生产数据上验证，随后利用 Docker 进行容器化打包，最终作为独立服务发布上线。 --- ### [15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) 问答环节：生产监控与 MLflow vs. W&B 工具选型在演讲结束后的问答环节中，现场观众就模型生产表现与技术栈选型提出了关键问题。针对现场关于“模型在生产环境表现如何”的提问，Priyanka 说明该工作是一个个人验证项目。虽然完成了部署，但尚未建立后续的监控系统。她指出，接下来的关键步骤正是构建监控流水线，以检测数据漂移（Data Drift）与模型漂移（Model Drift），确保模型在真实生产场景下的预测能力不会随时间发生退化。针对“MLflow 本身已具备实验追踪和工件管理功能，为何还要引入 Weights & Biases”的疑问，Priyanka 进行了详细的架构选型解释： ```mermaid flowchart TD accTitle: MLflow 与 Weights & Biases 职责划分 subgraph MLflow["MLflow 职责"] P1["MLflow Projects: 编排流水线独立任务"] P2["模型打包与标准化"] end subgraph WandB["Weights & Biases 职责"] W1["数据集版本控制 (Data Versioning)"] W2["实验追踪与指标对比"] W3["DAG 血缘可视化与 Model Registry"] end ``` - **MLflow 的局限性**：MLflow 虽然支持实验追踪、工件记录与模型注册表，但目前不支持真正意义上的数据版本控制（Data Versioning，尽管支持记录数据集元数据，但无法直接版本化数据）。 - **W&B 的核心优势**：Weights & Biases 能够原生提供端到端的数据版本化、运行对比与血缘追踪能力。 - **混合架构策略**：采用 MLflow Projects 来封装、调度流水线中的各个任务模块并打包模型，同时利用 Weights & Biases 承担数据版本控制、实验追踪与模型注册体系。 --- ### [18:54](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=1134s) 问答环节：落地周期与迭代追踪建议有观众询问在企业内部从零引入 MLflow 到最终建立模型需要多长的落地周期与学习成本。Priyanka 回应称，在初期探索自动化训练流水线时确实需要花费一些时间摸索，但如果参考现成的项目模版，搭建基础流水线并跑通首个基线模型通常“不会超过两周”。她表示后续会将这套实现代码开源至 GitHub 供大家参考。 Priyanka 强调，机器学习本质上是一个高度迭代的过程。工程师需要反复尝试不同的算法模型与超参数配置，并且经常会遇到“当前迭代的模型反而不如之前某一版本”的情况。如果无法清晰回溯是哪一份特定数据和哪组配置生成了那个更优的模型，迭代就会陷入混乱。因此，建立完善的实验追踪与数据版本控制，是构建工业级机器学习流水线中最基础且不可或缺的核心能力。 ## Notable quotes > [00:50](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=50s) 我们能否根据客户的历史行为、账户信息以及服务使用模式，准确预测其是否可能流失？ — Priyanka Asnani > [00:50](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=50s) 维护和留存现有客户的成本显著低于获取新客户的获客成本（CAC）。 — Priyanka Asnani > [01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) 按月签约（Month-to-month）的客户流失率显著最高，直观反映出短期合约客户粘性较低、随时可能离开。 — Priyanka Asnani > [01:34](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=94s) 使用光纤宽带（Fiber Optic）的客户流失率明显高于使用 DSL 或无网络服务的客户 — Priyanka Asnani > [04:36](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=276s) MLflow Projects 能够严格固化代码、运行环境依赖以及输入参数，保证无论在本地调试还是在云端集群执行，都能获得确定性的运行结果。 — Priyanka Asnani > [08:47](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=527s) 如果业务目标侧重于减少假阴性（漏报流失客户），则优先优化 Recall；如果目标是降低假阳性（避免给未流失客户误发高额折扣），则关注 Precision；若需要兼顾两者，则以 F1-score 为核心优化指标。 — Priyanka Asnani > [10:26](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=626s) 模型签名（Model Signature）：严格定义预期输入与输出的 Schema，在生产推理阶段防止特征缺失或数据类型不匹配引发的异常。 — Priyanka Asnani > [11:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=705s) 即使一个月后重新审视模型，也能准确获知当时使用的数据集版本、超参数和环境配置，这对于故障排查和完全复现至关重要。 — Priyanka Asnani > [14:01](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=841s) 模型注册表作为全团队唯一的“单一真实信任源”（Single Source of Truth），集中存放经过最终审核批准、准备用于下游任务（如部署、监控和回滚）的模型，避免团队协作中产生版本混乱。 — Priyanka Asnani > [15:45](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=945s) MLflow 虽然支持实验追踪、工件记录与模型注册表，但目前不支持真正意义上的数据版本控制（Data Versioning，尽管支持记录数据集元数据，但无法直接版本化数据）。 — Priyanka Asnani > [18:54](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=1134s) 机器学习本质上是一个高度迭代的过程。工程师需要反复尝试不同的算法模型与超参数配置，并且经常会遇到“当前迭代的模型反而不如之前某一版本”的情况。 — Priyanka Asnani > [18:54](https://www.youtube.com/watch?v=8vneD7DPtWQ&t=1134s) 建立完善的实验追踪与数据版本控制，是构建工业级机器学习流水线中最基础且不可或缺的核心能力。 — Priyanka Asnani ## Mentioned Priyanka Asnani, Fidelity Investments (富达投资), Ai4, MLflow, MLflow Projects, Weights & Biases (W&B), Telco Customer Churn Dataset, IBM, XGBoost, Scikit-Learn, Hydra, Hydra Sweeps, Docker, Streamlit, Conda, GitHub ## Video description Priyanka Asnani, Senior ML Engineer, Fidelity Investments Abstracts In this session, Priyanka Asnani will walk you through building an end-to-end, reproducible customer churn prediction pipeline powered by MLflow Projects and Weights & Biases. We’ll start with the problem statement, dataset overview, and exploratory data analysis before diving into the high-level system design. You’ll learn how to structure ML pipelines using MLflow Projects, chain components effectively, and leverage Hydra sweeps for hyperparameter optimization. Key components—data cleaning, feature engineering, XGBoost training, experiment tracking, and model registry—will be demonstrated with practical examples. Along the way, we’ll show how to log and track artifacts, metrics, and models using both MLflow and Weights & Biases, and how to register the best model for deployment. Whether you’re new to MLops or looking to sharpen your pipeline skills, this session offers actionable insights for building scalable, collaborative, and production-ready machine learning workflows. [Source](https://www.youtube.com/watch?v=8vneD7DPtWQ)

Source quotes

[0:50] Priyanka Asnani: 我们能否根据客户的历史行为、账户信息以及服务使用模式，准确预测其是否可能流失？ [0:50] Priyanka Asnani: 维护和留存现有客户的成本显著低于获取新客户的获客成本（CAC）。 [1:34] Priyanka Asnani: 按月签约（Month-to-month）的客户流失率显著最高，直观反映出短期合约客户粘性较低、随时可能离开。 [1:34] Priyanka Asnani: 使用光纤宽带（Fiber Optic）的客户流失率明显高于使用 DSL 或无网络服务的客户 [4:36] Priyanka Asnani: MLflow Projects 能够严格固化代码、运行环境依赖以及输入参数，保证无论在本地调试还是在云端集群执行，都能获得确定性的运行结果。 [8:47] Priyanka Asnani: 如果业务目标侧重于减少假阴性（漏报流失客户），则优先优化 Recall；如果目标是降低假阳性（避免给未流失客户误发高额折扣），则关注 Precision；若需要兼顾两者，则以 F1-score 为核心优化指标。 [10:26] Priyanka Asnani: 模型签名（Model Signature）：严格定义预期输入与输出的 Schema，在生产推理阶段防止特征缺失或数据类型不匹配引发的异常。 [11:45] Priyanka Asnani: 即使一个月后重新审视模型，也能准确获知当时使用的数据集版本、超参数和环境配置，这对于故障排查和完全复现至关重要。 [14:01] Priyanka Asnani: 模型注册表作为全团队唯一的“单一真实信任源”（Single Source of Truth），集中存放经过最终审核批准、准备用于下游任务（如部署、监控和回滚）的模型，避免团队协作中产生版本混乱。 [15:45] Priyanka Asnani: MLflow 虽然支持实验追踪、工件记录与模型注册表，但目前不支持真正意义上的数据版本控制（Data Versioning，尽管支持记录数据集元数据，但无法直接版本化数据）。 [18:54] Priyanka Asnani: 机器学习本质上是一个高度迭代的过程。工程师需要反复尝试不同的算法模型与超参数配置，并且经常会遇到“当前迭代的模型反而不如之前某一版本”的情况。 [18:54] Priyanka Asnani: 建立完善的实验追踪与数据版本控制，是构建工业级机器学习流水线中最基础且不可或缺的核心能力。

Source quotes

[0:04] Priyanka Asnani: significantly more cost effective than acquiring new ones. [1:34] Priyanka Asnani: very critical for their retention. [10:26] Priyanka Asnani: This helps us prevent any schema mismatches in production environment. [11:45] Priyanka Asnani: If I go to my model a month later, which happens very often when we are building these machine learning pipelines, I know precisely which version of the dataset and what hyperparameters and configurations were used at the time of training. [15:45] Priyanka Asnani: MLflow doesn't support data versioning, and I wanted one tool that supports everything from data versioning to experiment tracking to having a model registry.

Original transcript

[0:04] Good afternoon everyone. Uh my name is [0:06] Priyanka and today I will be presenting [0:09] on building reproducible ML training [0:12] pipelines with ML flow and weights and [0:14] biases. [0:17] Here is a quick overview of what I'll be [0:20] covering today. So we'll start with the [0:21] problem statement um the data set then [0:24] walk through the exploratory analysis [0:27] followed by the system architecture. I [0:29] will highlight how I used MLflow [0:31] projects to build and orchestrate the [0:33] training pipeline covering each com key [0:36] component from data preparation to model [0:38] training and optimization and wrap up [0:41] with experiment tracking and model [0:43] registry. The focus will be on building [0:46] reproducible and production ready ML [0:48] workflow. [0:50] The problem that we are solving here is [0:53] can we accurately predict whether a [0:56] customer is likely to churn or not based [0:59] on their past behavior, account [1:01] information and service usage patterns. [1:03] This is framed as a binary [1:05] classification task where the target [1:07] variable is whether the customer has [1:09] churned or not. Solving this problem is [1:12] important because re retaining existing [1:15] customers is significantly more cost [1:18] effective than acquiring new ones. By [1:21] the identifying high-risk customers [1:23] early on in the process, the business [1:25] can take proactive steps such as [1:28] personalized outreach discounts or [1:30] service changes to improve the [1:31] retention. [1:34] In this project, we use telco customer [1:36] churn data set which is a publicly [1:38] available data set provided by IBM and [1:41] widely used for churn modeling. It [1:44] contains records for over 7,000 [1:46] customers with a mix of demographic [1:48] information, account level attributes [1:50] and service usage patterns such as [1:53] contract type, monthly charges and [1:55] internet service. The target variable is [1:58] churn indicating whether the customer [2:01] has discontinued the service and we [2:03] frame it as binary classification [2:05] problem. [2:07] Now to better understand the behavior [2:10] behind the customer churn, we began our [2:12] analysis with a look at the overall [2:15] churn distribution. Here we have the [2:18] imbalance in the data set with a larger [2:20] proportion of customers uh staying than [2:23] leaving. So this class imbalance work [2:25] was was considered during the model [2:27] development. Diving deeper, we analyzed [2:30] churn rates by contract type. The [2:32] month-to-month contract had the highest [2:34] churn rate, which intuitively aligns [2:37] with the idea that customers with [2:40] shorter term contracts are more likely [2:42] to leave. We then examined internet [2:44] service types and noticed that customers [2:47] using fiber optic u had a higher churn [2:50] rate compared to those with DSL or no [2:53] internet service. So this could indicate [2:55] dissatisfaction with the performance or [2:57] the pricing. [2:59] Look at the uh we we then look at the [3:01] monthly charges versus churn. Uh we saw [3:04] that customers with higher monthly [3:06] charges were more more likely to uh [3:09] churn which signals possible pricing [3:11] sensitivity. And lastly we analyze [3:14] tenure distribution by churn status. And [3:17] we see customers who churn generally had [3:19] shorter tenure period suggesting that [3:22] early periods of customer experience are [3:24] very critical for their retention. [3:29] This slide outlines a typical machine [3:32] learning model life cycle on the high [3:34] level. So we divided this into two core [3:37] pipelines training and inference. Uh on [3:40] the training side we began with the data [3:42] validation to ensure that the data um [3:45] quality and schema consistency followed [3:47] by data prep-processing to transform raw [3:50] inputs into production ready features [3:52] model ready features. The clean data is [3:54] then used for model training and [3:56] evaluation. And once the model meets the [3:59] performance criteria, it is saved and [4:01] registered in the model registry. On the [4:04] inference side, what we do is when when [4:06] we pull our uh train modeled artifacts, [4:09] package the model into a docker image [4:11] and then deploy it as a service. The [4:14] deployed model is then exposed through a [4:16] streamlit interface um where users can [4:19] input the new data and get realtime [4:23] predictions. While my project doesn't [4:25] use Docker or Streamlit, they are [4:27] included here to represent how models [4:29] are typically typically deployed and [4:31] consumed in a real production [4:34] environment. [4:36] MLFlow projects uh provide a very [4:38] structured way to package our machine [4:41] learning code for reproducibility and [4:43] collaboration. So each project captures [4:46] the code, the environment and [4:48] parameters. So whether I run my project [4:51] locally or on the cloud, I get [4:53] consistent results. It is also very [4:56] flexible. So I can specify my envir [4:58] dependencies using either cond or [5:01] docker. It is language agnostic. Um so [5:04] I'm not tied to python specifically. I [5:06] can use any language that supports [5:08] command line interface like R, Julia. Um [5:11] which makes it ideal for diverse teams [5:14] and workflows. [5:16] In our pipeline, the execution flow is [5:18] orchestrated using mlflow.run [5:22] calls inside main.py uh main.python file [5:25] where each component is treated as a [5:28] standalone step. So these steps are [5:30] changed chained in a direct u directed [5:33] cyclic graph or DAC structure where the [5:36] output artifact of one step becomes the [5:39] input to the next step. This ensures a [5:42] clear dependency aware flow across [5:44] stages like data loading, cleaning, [5:46] pre-processing and model training. [5:50] At the root of the folder structure, we [5:53] have ML project file. So this acts as [5:56] the entry point. It defines the project [5:58] name environment and how the project [6:01] should be executed through the entry [6:03] points and the parameters. [6:06] Next is [6:08] file. This specifies the Python version [6:11] and all the required dependencies to run [6:13] your project. By using this, we ensure [6:16] that the environment is consistent and [6:19] reproducible across all different uh [6:22] machines and team members. [6:25] Then we have the main.py [6:27] main.python file which orchestrates the [6:30] overall pipeline. So it chains together [6:32] the individual components of the [6:34] pipeline using mlflow.run run passing [6:38] artifacts from one step to the next. [6:40] Each component lives inside its own [6:43] folder. So as we see here on the p uh on [6:46] the image uh to the left that we have [6:49] each component inside its own folder [6:51] like we have folders for get data basic [6:55] cleaning training and evaluation and h [6:58] each folder has its own ML project file. [7:03] file and run.py file. So the run.p py [7:06] file in each step contains the actual [7:09] logic for that part of the workflow such [7:12] as data transformation, model training [7:15] and evaluation. This modular design [7:18] allows us to isolate and reuse and rerun [7:21] any step in the pipeline making it very [7:23] easy to scale and maintain. [7:26] So the first step um that we have is [7:30] download step where we download the raw [7:33] data set from the weights and biases and [7:36] perform basic cleaning like dropping [7:38] ids, converting total charges column to [7:41] a numeric column and removing some [7:43] invalid rows and then saving it as the [7:46] clean data. We log this as a new [7:49] artifact in weights and biases to ensure [7:51] versioning and reproducibility for [7:53] downstream training. Now we have this [7:56] step here but in real world this all [7:58] these aggregations that we perform will [8:00] be a part of the feature engineering [8:03] pipeline and your uh features will get [8:06] will get logged into a feature store. [8:10] Then um the next step is training [8:12] pipeline component. Uh we be here we [8:15] begin by grouping our features into [8:17] three types binary categorical [8:20] numerical. So if I for binary columns [8:23] like gender and partner status, we use [8:25] the ordinal encoder for the [8:27] transformation and then what we do is we [8:29] combine all of these pre-processing [8:32] individual pre-processing pipelines into [8:34] a single column transformer to apply [8:36] these transformations on the data set. [8:39] We wrap both column transformer and the [8:41] XG boost model into a scikitlearn [8:44] pipeline. [8:47] Next is um next we begin by extracting [8:50] the target variable which is churn from [8:53] the data set. So since this is a binary [8:55] column and contains values yes and no, [8:58] we map it to um we map it to one and [9:01] zero respectively. Next we split the [9:04] data set into train test um split and [9:07] here uh we use this function to split [9:10] our data set into training and [9:11] validation sets. Since uh we are dealing [9:14] with imbalance in uh we are dealing with [9:17] imbalanced classes in our data set. What [9:19] we use is a stratification technique to [9:22] ensure that the class distribution is [9:24] preserved across both um training and [9:26] validation sets. We then call a helper [9:29] function to get our inference pipeline. [9:33] This pipeline includes all the [9:35] pre-processing steps like encoding, [9:37] imputation, scaling, etc. and an XG [9:40] boost classifier. We then fit the entire [9:43] pipeline on the training data. After [9:46] training, we run the predictions on the [9:48] validation data set and compute key [9:50] metrics like accuracy, precision, [9:53] recall, F1 score. Now, these um uh [9:57] metrics will vary depending on what we [9:59] are trying to optimize for. If we are [10:01] trying to minimize false negatives, we [10:04] are going to be focusing on recall. If [10:05] you're going to be minimizing um false [10:08] positives, then we are going to be [10:09] focusing on precision. And if we want to [10:12] optimize for both precision and recall, [10:14] then we are going to be considering F1 [10:16] score. We then compute these metrics and [10:19] log all the metrics um into weights and [10:23] biases for full traceability. [10:26] In this step, um now the next step is [10:29] where we are going to save our trained [10:31] model uh with MLflow and weights and [10:34] biases artifacts. So in this step, we [10:36] save our trained skyitlearn pipeline [10:39] using MLflow.s sklearn.save model [10:42] function. So this exports the entire [10:45] pipeline including pre-processing and [10:47] the XG boot model into a standardized [10:50] MLflow directory format. Now what the [10:54] advantage of it is that along with the [10:56] model we also include a signature which [10:59] defines the expected input and output [11:01] schema and an input example which serves [11:05] as a reference input for our future um [11:08] inference request. So this deta this [11:12] helps us prevent any schema mismatches [11:14] in production environment. After saving [11:17] the model locally, we log the model [11:20] directory as the weights and biases [11:22] artifact to ensure that the model is [11:24] versioned and tracked alongside training [11:28] metadata such as hyperparameters and [11:30] evaluation metrics [11:34] to optimize uh model performance. uh we [11:37] used hydra sweeps uh with to perform [11:40] grid search over all the xg boost [11:42] hyperparameters. This command triggers [11:45] multiple runs each with different [11:47] combination of hyperparameters like [11:50] magstep n estimators and subsample. We [11:53] then log all the runs making it easy to [11:56] compare the performance of all the model [11:59] and select the best performing model. [12:04] The next is experiment tracking. Um so [12:06] the weights and biases gives us a gives [12:09] us a very powerful and organized way to [12:11] track and reproduce our experiments [12:14] which makes our workflow more [12:16] transparent and production ready. So [12:18] here we can see a snapshot of weights [12:21] and biases run dashboard. Each row [12:24] represents a different model training [12:26] run. I have logged all the key metrics [12:30] the accuracy, precision, recall and [12:32] training times. And this view allows me [12:34] to quickly compare different models over [12:37] time and identify which one is the best [12:40] performing model depending on the [12:42] metrics that we are tracking. [12:47] For each run I am also lo I've also [12:50] logged all the hyperparameters that I [12:52] used to train the model. This includes [12:55] settings like max depth and number of [12:58] estimators. I versioned and logged all [13:02] the input data sets the train model as [13:04] an output artifact. If I go to my model [13:08] a month later which happens very often [13:11] uh when when we are building these [13:13] machine learning pipelines I know [13:15] precisely which version of the data set [13:18] and what hyperparameters and [13:20] configurations were used at the time of [13:22] training and this is really critical for [13:24] debugging and reproducibility purposes. [13:31] The final screenshot um shows weights [13:34] and biases DAG view which visualizes the [13:37] entire lineage of experiments that we [13:40] performed. We can see that the [13:42] individual runs and the and the and the [13:44] corresponding inputs and outputs at each [13:47] step. It gives a clear trace of how our [13:50] input is flowing through each run and [13:52] the output artifacts that are that are [13:55] being produced at each step. [14:01] Now once we have [14:04] trained and evaluated our models and [14:07] pushed our models uh we make sure that [14:10] every trained model is versioned but [14:12] also with the associated code [14:15] configurations and artifacts. So in [14:17] weights and biases this versioning [14:19] allows us to track and compare and [14:22] analyze model performance over time. So [14:25] when we are ready to identify the best [14:27] performing model, we we tag it typically [14:30] with labels like staging or prod [14:33] depending on what environment we want [14:35] want to deploy our model on. For from [14:38] there we link the tagged model into a [14:41] model registry. Um as you can see on on [14:44] the right side of the image from uh we [14:47] link the tagged model to a model [14:49] registry using the button on the right. [14:51] The steps marks as our official [14:54] production candidate. The model registry [14:57] acts as a central source of truth [14:59] holding the final approved models that [15:02] are ready for downstream task like [15:04] deployment and monitoring and roll [15:07] backs. This ensures that everyone on the [15:10] team is working on only one single [15:13] source of truth. From here what we do is [15:16] we pull the model artifacts of our [15:19] model. um we we test it on the [15:22] production data. We dockerize uh we we [15:26] um containerize the uh model using [15:29] Docker and then we deploy it as a [15:32] service. Um yeah, so thank you so much. [15:38] [Applause] [15:45] Thank you so much Priyanka. Let's give [15:47] her one more round of applause. [15:50] [Applause] [15:52] Did you want to do a question and answer [15:55] session? You do have time. [15:56] >> How much time? [15:57] >> Five minutes. [15:57] >> Yeah, we can take questions. Yeah. Any [16:00] questions? Raise your hand nice and big. [16:11] Thank you for the talk. Priyanka, could [16:12] you talk about how the model did the is [16:14] it in production? How's it been [16:15] performing in production? [16:17] >> Yeah, so um this is um what I did as a [16:21] personal project. Um so um you know we I [16:24] I didn't I I deployed it um but due to [16:27] time constraints I am um I I don't have [16:30] those slides in here but um [16:34] to to your question right so I deployed [16:36] it but I didn't go a further step to [16:39] monitor my models or um test it out u [16:43] but yeah that would be the next step to [16:45] actually monitor the performance of the [16:47] model for data drifts model drift to see [16:49] how my model is doing um and and then um [16:53] you know make sure that the model [16:55] predictions don't degrade over time. [16:57] Yeah, that that would be the next step. [16:59] Yeah. [17:01] Um does that answers your question? [17:03] Okay, [17:05] >> great presentation Brian. I just have [17:06] one question. So uh I according to my [17:09] understanding there is uh experiment [17:11] tracking and uh artifacts uh feature in [17:15] MLflow itself. Have you considered using [17:18] that instead of weights and biases? I'm [17:20] just curious because [17:21] >> Yeah. Can you please repeat? I I don't [17:23] think I understood your question. [17:25] >> You just have to talk about it. Yeah. [17:27] >> Okay. Uh so my question is uh I think [17:30] MLflow itself has experiment tracking [17:32] and logging functionality. I was just [17:34] curious why you chose weights and biases [17:36] instead. [17:37] >> Yeah, that's that's a very good [17:38] question. Right. So for MLflow doesn't [17:40] supports data versioning and I wanted a [17:43] tool that I I wanted one tool that [17:45] supports everything from data versioning [17:48] to experiment tracking to having a model [17:50] registry. Um so sure MLflow does support [17:53] experiment tracking and it does have uh [17:55] model registry but unfortunately doesn't [17:58] support data versioning which I believe [18:00] is a very important part of uh building [18:03] a reproducible ML pipeline and that's [18:05] why I chose weights and biases but I am [18:08] uh packaging my model using MLflow and [18:11] I'm using MLflow projects to build my [18:14] pipeline as you know h having a separate [18:16] task um in in a pipeline. So I'm using [18:19] both but MLflow I could have done this [18:22] solely using MLflow but the fact that it [18:24] doesn't supports data versioning. Um you [18:27] can log your data sets but you cannot [18:29] version your data set. So um it it took [18:32] me some experimentation and kind of to [18:35] to figure out okay we we need to combine [18:37] these two tools and we cannot just go [18:39] with MLflow. I I hope that that answers [18:41] your question. Yeah data versioning is a [18:44] very big feature that MLflow doesn't [18:46] supports at the moment. Yeah. [18:48] >> Thank you. Yeah. Yeah. [18:54] >> This will be our last question. [18:59] >> Hi, thank you. So, I was kind of curious [19:01] about because I I might want to [19:03] implement this in my own company. Um, [19:06] what was like the time to I I know you [19:08] didn't deploy it, but the time to market [19:10] from deploying MLFlow itself to actually [19:13] building the model. Um was there like a [19:16] ballpark of how long it personally took [19:18] you in learning the tool sets and um [19:21] going through all the documentation and [19:23] stuff like that? [19:23] >> Yes. So it took me some time um to [19:26] figure out because we were um uh I I was [19:30] figuring out how we can you know [19:32] automate this entire process of training [19:35] uh pipelines. But for you um um you know [19:38] I'll be um you know maybe you know we [19:41] can get in touch. I'll be pushing this [19:42] code to GitHub and then I don't think it [19:45] will take you more than two weeks to [19:47] kind of get everything set up uh and and [19:50] just have that initial baseline of how [19:52] your model is doing and just being able [19:55] to track all your experiments while uh [19:57] you are playing with these different [19:59] models um and and hyperparameters which [20:02] is a very important um step because in [20:07] machine learning for people who are [20:09] already working in the field it's a [20:10] highly iterative process. process. So [20:12] you you keep uh you keep trying [20:14] different models, their configurations [20:16] and every time you have to come back to [20:18] to the model that was performing way [20:20] better for the than the model that [20:22] you're working on right now and you [20:23] don't know which specific data sets or [20:26] configurations that produce that model, [20:28] right? So you need to be able to track [20:29] those experiments um as you're uh [20:32] experimenting and playing with these [20:34] configurations. [20:36] Uh does that answer your question? [20:38] >> Yeah, thank you so much. [20:38] >> Yeah, no problem.