General Mills Uses Synthetic Outages to Train Self-Healing Pipeline AI

Synthetic by Design: Privacy-First Data Engineering for Scalable, High-Fidelity AI Training — Ai4

Ai4

Building a canonical case library of simulated infrastructure failures enables AI agents to diagnose and resolve production pipeline crashes in real time without exposing sensitive data.

AI-assisted summary of Synthetic by Design: Privacy-First Data Engineering for Scalable, High-Fidelity AI Training — original source. Check the original for full context.

Source published: 2026-08-18

Detailed notes

## Background Published by Ai4 on YouTube on August 18, 2026, this presentation features Sanchit Srivastava, Sr. Manager of Data Analytics & Architecture and Principal Architect at General Mills. The talk explores how synthetic data generation has transitioned from a niche testing workaround into a production-grade, privacy-first engineering discipline designed to overcome data scarcity and regulatory bottlenecks in AI pipelines.

## TL;DR Sanchit Srivastava redefines synthetic data from a build-time testing artifact into a multifaceted runtime asset spanning six programmatic categories, from statistical generation to simulation-based proxies. He explains how synthetic virtual layers protect sensitive enterprise tables from LLM prompt leakage and enable safe agentic reasoning. Highlighting a real-world implementation at General Mills, he details an autonomous self-healing data pipeline agent trained on a vectorized Case Library of synthetically simulated failure modes across Pub/Sub, Dataflow, dbt, and BigQuery. The talk concludes with key architectural lessons on consolidating vector infrastructure within native cloud hyperscalers, transitioning from static cold stores to living case libraries, and enforcing rigorous statistical governance to prevent drift.

## Atmosphere & style The presentation carries a pragmatic, authoritative, and engineering-first tone, anchored by real-world enterprise realities rather than academic hype. Srivastava speaks with the measured clarity of an enterprise architect, using vivid, relatable analogies—such as comparing meticulous pipeline engineering to Da Vinci and demanding business users to toddlers drawing on walls—to cut through complex technical concepts. The delivery feels brisk, structured, and focused, leaving the audience with an inspiring yet grounded blueprint for operationalizing agentic AI in mission-critical data platforms.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) Synthetic data has evolved from simple QA mock testing into an advanced engineering discipline spanning partial synthetic generation and simulation-based systems. - [01:28](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=88s) Programmatic synthetic data divides into zero-real-data approaches (statistical, rule-based, anomaly injection) and data-informed generative methods like CTGAN and simulation. - [04:45](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=285s) Synthetic data now functions as a runtime reasoning proxy, shielding sensitive production databases from direct LLM interaction and preventing memory or prompt leakage. - [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) General Mills developed a self-healing pipeline agent by synthetically generating hundreds of failure modes across tools like Pub/Sub, Dataflow, dbt, and BigQuery. - [12:31](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=751s) Structuring failure logs, remediation actions, and outcomes into a canonical JSON format prevents context bloat and eliminates hallucinations in RAG-driven triage. - [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) The self-healing runtime agent uses multi-shot reasoning across historical case outcomes while keeping a human-in-the-loop for approval before executing fixes. - [17:00](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1020s) Consolidating vector stores, logs, and compute natively within the same cloud provider reduced retrieval latency and architectural complexity compared to third-party vector databases. - [19:21](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1161s) Next-generation autonomous data platforms are transitioning from static cold stores and bounded taxonomies to living hot case libraries that absorb novel agent-proposed solutions. - [20:13](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1213s) Rigorous statistical governance and continuous drift monitoring are necessary to prevent synthetic data skewness from degrading downstream AI model performance.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) The Evolution and Spectrum of Synthetic Data

The traditional perception of synthetic data has long been pigeonholed into basic QA environments and mock unit tests. Sanchit Srivastava notes that when someone asked him earlier in the morning, "synthetic data are we still talking about testing?", it underscored the need to clarify how drastically the discipline has evolved. While data remains the root cause of AI friction, synthetic data is no longer merely dummy records generated for build-time test runs. Modern data engineering now spans a spectrum from fully synthetic datasets to partial synthetic generation and simulation-based systems.

```mermaid flowchart TD accTitle: Evolution of Synthetic Data Paradigms subgraph SyntheticSpectrum["Synthetic Data Paradigms"] direction TB A["Fully Synthetic (Legacy testing & QA)"] B["Partial Synthetic (Fills ML feature & class gaps)"] C["Simulation-Based (Guardrails, failures & agent reasoning)"] end A --> B --> C ```

Partial synthetic data has become vital for advanced machine learning and AI pipelines where production data exhibits severe sparse-class issues or statistical blind spots. In these architectures, the synthetic data is not real, but it meticulously preserves the mathematical characteristics, distributions, and structural relationships of production datasets. Simulation-based synthetic data takes this further by purposefully generating synthetic failure modes, edge cases, and behavioral guardrails that developers explicitly do not want production systems to execute unchecked. This simulated data serves as the foundation for autonomous reasoning agents capable of maintaining data platform health.

[01:28](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=88s) Six Programmatic Categories and Core Tooling

Programmatic synthetic data generation divides cleanly into six modern categories, split between techniques that require zero exposure to real data and generative methods that learn from real-world samples:

```mermaid flowchart TD accTitle: Six Programmatic Categories of Synthetic Data Generation subgraph ZeroRealData["Zero Real Data Required"] direction TB C1["Statistical-Driven (scikit-learn make_classification)"] C2["Rule-Based / Templated (Faker-style schemas, PII mocks)"] C3["Anomaly & Outlier Injection (PyOD contamination tuning)"] end subgraph DataInformed["Data-Informed / Generative"] direction TB C4["Generative Deep Learning (CTGAN red/blue team modeling)"] C5["Data Augmentation (Transformation & perturbation rules)"] C6["Simulation-Based (Simulated system states & edge cases)"] end ```

The first three methods eliminate data leakage risks because they rely entirely on parameterized definitions rather than production inputs: - **Statistical-Driven Generation**: Generates programmatic distributions, feature vectors, and classification boundaries using libraries such as `scikit-learn`'s `make_classification` and `make_regression`. - **Rule-Based and Templated Generation**: Creates structural entities like synthetic addresses, names, and email schemas where the values are artificial but conform strictly to validation rules and data contracts. - **Anomaly and Outlier Injection**: Rather than generating "clean" data, frameworks like `PyOD` (Python Outlier Detection) deliberately self-inflict statistical contamination and outlier vectors at configurable percentages to stress-test downstream models against anomalies.

The remaining three categories derive synthetic patterns from real data representations: - **Generative Deep Learning (GANs / CTGAN)**: Uses architectures like Conditional Table GANs (CTGAN) to learn multi-dimensional distributions from seed data samples, generating synthetic records for both positive and negative scenarios via adversarial red-team/blue-team mechanics. - **Data Augmentation**: Applies rule-based transformations, perturbations, and semantic permutations to expand limited training sets. - **Simulation-Based Generation**: Actively simulates missing environments, upstream failures, and rare behavioral events that cannot be captured from standard steady-state logs.

While generative deep learning tools like CTGAN require caution because they ingest real data samples, they excel at generating high-utility synthetic data for fraud detection, firewall algorithms, and adversarial evaluation.

[04:45](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=285s) Repositioning Synthetic Data: From Build-Time Testing to Runtime Reasoning Proxies

The operational location of synthetic data in the data engineering lifecycle has shifted fundamentally. In legacy architectures, synthetic data lived purely in build-time test suites. Today, it serves as a dual-phase asset that spans both build-time contract validation and runtime reasoning proxies.

```mermaid flowchart LR accTitle: Synthetic Data as a Runtime Reasoning Proxy subgraph DevPhase["Build-Time Validation"] direction TB D1["Non-Prod Sandboxes"] D2["Contract Verification"] D3["GDPR / CCPA Guardrails"] end subgraph RuntimePhase["Runtime AI Execution"] direction TB R1["AI Reasoning Loop"] R2["Synthetic Virtual Layer"] R3["Production Data Execution"] end DevPhase --> RuntimePhase R1 <-->|"Safe Reasoning"| R2 R1 -->|"Action Dispatch"| R3 ```

Traditional engineering pressures like GDPR and CCPA mandate that raw customer PII and sensitive production tables never touch non-production environments where security guardrails may be relaxed. Furthermore, the rise of third-party AI vendors and external execution sandboxes introduces perimeter risks where security guarantees might not match enterprise compliance standards. Testing models against high-fidelity synthetic proxies allows teams to evaluate rare edge cases without risking sensitive enterprise data.

In production LLM and agentic workflows, synthetic data acts as a critical defensive barrier: - **Mitigating Memory and Prompt Leakage**: Because foundational models possess latent memory and can inadvertently surface ingested training points during prompt injection or extraction attacks, pre-training and fine-tuning on partial or pure synthetic datasets dramatically curtails blast radius. - **Virtual Reasoning Sandboxes**: Instead of letting LLMs iterate and reason directly over raw production tables, architectures can place a high-fidelity synthetic virtual layer in front of the model. The agent performs its intermediate iterative chain-of-thought against the synthetic proxy and only accesses production storage once the final query or action plan is validated.

[08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) Building a Self-Healing Data Pipeline Agent

Engineering reliable data pipelines is notoriously challenging. Srivastava captures the tension between engineering and business stakeholders with an analogy: building robust data pipelines is an exacting craft akin to "Da Vinci drawing something," whereas downstream business consumers behave "like my toddler drawing it on the wall—he wants it now, he wants it right, and everything should be accurate." Because distributed pipelines inevitably break due to infrastructure bottlenecks, schema drift, and brittle data contracts, General Mills designed an autonomous AI agent to diagnose and remediate pipeline failures in real time.

```mermaid flowchart TD accTitle: Architecture of the Self-Healing Data Pipeline Agent subgraph BuildPhase["1. Build Phase (Synthetic Generation)"] direction TB A1["Simulate Tech Failures (Pub/Sub, Dataflow, dbt, BigQuery)"] A2["Capture & Replay Logs"] A3["Format to Canonical JSON Schema"] A4["Vectorize into Case Library (RAG)"] A1 --> A2 --> A3 --> A4 end

subgraph RuntimePhase["2. Runtime Phase (Resolution & Feedback)"] direction TB B1["Live Pipeline Failure Logs"] B2["Agent RAG Retrieval (Top-K Matches)"] B3["LLM Reasoning & RCA Diagnosis"] B4["Human-in-the-Loop Approval"] B5["Action Executed & Post-Outcome Stored"] B1 --> B2 --> B3 --> B4 --> B5 end

A4 -.->|"Indexed Knowledge"| B2 B5 -.->|"Living Document Update"| A4 ```

To train the agent without waiting months for organic production failures, the engineering team synthetically simulated hundreds of technology failure modes across their ecosystem, including dbt, Google Cloud Dataflow, and Pub/Sub. The raw execution logs from these simulated failures were captured, replayed, and structured into a standardized canonical format optimized for LLM readability. These records were then compiled into an indexed vector **Case Library**.

When a live pipeline breaks in production, the agent compares the live diagnostic logs against the Case Library via Retrieval-Augmented Generation (RAG). By evaluating previously recorded failure signatures alongside past actions and their historical outcomes, the agent does not merely perform a naive keyword match—"a certain log retrieval can give you an action; a good case library gives you the reason to take an action." The agent formulates a root cause analysis (RCA), proposes a remediation strategy to a human engineer via a Human-in-the-Loop (HITL) interface, and writes the verified post-action outcome back into the Case Library.

[12:31](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=751s) Anatomy of the Case Library and Canonical Structuring

The Case Library forms the foundational intelligence layer for automated pipeline triage. It was created by systematically injecting synthetic bad data and artificial infrastructure constraints into core data technologies to record specific failure-action-outcome triads.

Feeding raw, unstructured error logs directly into an LLM causes severe latency spikes, context window bloat, and frequent hallucinations. To guarantee fast and deterministic RAG retrieval, all failure logs, remediation steps, and outcomes are normalized into a compact, standardized canonical format prior to vectorization:

```json { "failure_id": "ERR_DF_OOM_042", "technology": "Dataflow", "error_signature": "java.lang.OutOfMemoryError: Java heap space", "root_cause": "Unpartitioned high-volume streaming skew", "action_applied": "Added synthetic memory allocation and re-chunked partition keys", "action_outcome": "SUCCESS", "last_verified_timestamp": "2026-08-18T10:15:30Z" } ```

[16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) Runtime Execution, Architectural Lessons, and Single-Pane Vectorization

During live operations, the self-healing agent continuously monitors data pipelines. When a failure trigger occurs, the runtime orchestration executes a deterministic multi-step diagnostic workflow:

```mermaid flowchart LR accTitle: Runtime Reasoning and Execution Loop L["Live Error Log"] --> V["Vector Query (Top-K)"] V --> C["Case Library"] C --> R["LLM Reasoning & RCA"] R --> H["Human Operator Approval"] H --> E["Execute Fix"] E --> F["Feedback Loop: Append Outcome"] F --> C ```

- **Top-K Vector Retrieval**: The incoming error signature is transformed into an embedding to query the vectorized Case Library for top-k historical matches. - **Multi-Shot Agent Reasoning**: Rather than acting on static patterns, the LLM reasons across multiple historical resolutions: *"This failure happened previously; action X was taken, resulting in partial success, whereas action Y resolved it completely."* - **Accelerated RCA Delivery**: The agent synthesizes an accurate Root Cause Analysis in seconds, significantly shrinking triage time for on-call engineers. - **Human-in-the-Loop Execution**: The proposed fix is reviewed and confirmed by an engineer before remediation scripts run against production infrastructure. - **Living Document Feedback**: The final post-action result is written back into the vector store, ensuring continuous learning across deployment cycles.

A key architectural insight emerged regarding vector infrastructure. The team initially deployed a standalone third-party vector database service. However, cross-network retrieval latency and operational friction hindered pipeline triage SLAs. The team refactored the design to align with "Architecture 101: where if you have lesser knobs, you have more robust pipelines."

They consolidated the vector embeddings, pipeline execution logs, Case Library repositories, and compute workloads inside the same native cloud hyperscaler. Storing vector representations directly adjacent to the underlying data platform eliminated network overhead, minimized integration points, and established a scalable foundation for enterprise-grade autonomous data engineering.

[19:21](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1161s) Evolving from Cold Stores and Bounded Taxonomies to Living Case Libraries

Reflecting on the ongoing architectural evolution of their agentic framework, Sanchit Srivastava examines the limitations currently in place and where the engineering needs to go next. At present, the system operates largely as a "cold store"—a static repository capturing a catalog of historical failures, corresponding remediation actions, and recorded outcomes. The next architectural milestone is establishing an automated, live feedback loop back into the system whenever an action is executed in production, ensuring that resolution quality continuously sharpens over time. The fundamental challenge yet to be fully solved is eliminating this cold-start dynamic and transforming the setup into "a living and a hot case library."

Another key operational bottleneck stems from a "bounded taxonomy." Currently, the autonomous agent is strictly constrained to executing only the predefined actions manually cataloged by human engineers. However, practical observation has shown that the AI agent frequently reasons out and proposes better, more effective resolutions than those originally documented in the standard operating procedures. The team is actively engineering the next phase of the platform to dynamically evaluate these novel agent-generated proposals and incorporate superior solutions directly back into the core case library.

```mermaid flowchart TD accTitle: Transition from Cold Store to Hot Case Library subgraph Current ["Current Architecture (Cold Store)"] A["Failure Catalog"] --> B["Predefined Bounded Taxonomy"] B --> C["Static Remediation Action"] end

subgraph Future ["Target Architecture (Living Hot Library)"] D["Observed Incident"] --> E["Agent Proposes Resolution"] E --> F{"Is Proposal Better than SOP?"} F -->|"Yes"| G["Incorporate into Case Library"] F -->|"Standard"| H["Execute & Measure Outcome"] G --> H H -->|"Live Feedback Connect-Back"| D end ```

- **Cold Store vs. Hot Case Library:** Moving away from static lookup repositories toward continuously updated, closed-loop telemetry that updates case embeddings and outcome statistics dynamically. - **Overcoming Bounded Taxonomies:** Designing ingestion pathways so the system is not locked into hardcoded actions when an LLM agent discovers more optimal resolution strategies.

---

[20:13](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1213s) Statistical Governance, Drift Mitigation, and Final Takeaways

Turning to essential best practices for teams implementing synthetic data pipelines, Srivastava emphasizes the absolute necessity of rigorous, statistically governed generation. If synthetic data is generated naively, it introduces severe skewness and departs from real-world feature distributions. When synthetic training data deviates significantly from genuine data profiles, downstream AI models break down during training and evaluation: "Your classification will go all over the places."

Beyond initial baseline fidelity, ongoing drift management is critical. Teams must continuously monitor and align streaming synthetic datasets against incoming production distributions to ensure that distance metrics between real and synthetic data distributions remain tightly bounded across all target use cases. Concluding the session, Srivastava directs attendees to the project's GitHub repository for deep technical documentation, architectural blueprints, and sample code, extending an invitation for ongoing collaboration as the work continues to mature before wrapping up the presentation as time expires.

- **Statistical Governance:** Guard against artificial skewness and distortion to preserve feature co-dependencies required for downstream classification and modeling. - **Drift Prevention:** Ensure ongoing alignment between evolving real-world streaming data and synthetic generation parameters. - **Open-Source Artifacts:** Reference implementations, architectural documentation, and code samples are maintained on GitHub for practitioners scaling synthetic data engineering.

## Notable quotes > [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) synthetic data are we still talking about testing? — Sanchit Srivastava

> [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) Da Vinci drawing something — Sanchit Srivastava

> [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) like my toddler drawing it on the wall—he wants it now, he wants it right, and everything should be accurate. — Sanchit Srivastava

> [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) a certain log retrieval can give you an action; a good case library gives you the reason to take an action. — Sanchit Srivastava

> [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) This failure happened previously; action X was taken, resulting in partial success, whereas action Y resolved it completely. — Sanchit Srivastava

> [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) Architecture 101: where if you have lesser knobs, you have more robust pipelines. — Sanchit Srivastava

> [19:21](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1161s) cold store — Sanchit Srivastava

> [19:21](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1161s) a living and a hot case library. — Sanchit Srivastava

> [19:21](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1161s) bounded taxonomy — Sanchit Srivastava

> [20:13](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1213s) Your classification will go all over the places. — Sanchit Srivastava

## Mentioned Sanchit Srivastava, General Mills, Ai4, scikit-learn, PyOD, CTGAN, GDPR, CCPA, Google Cloud Pub/Sub, Google Cloud Dataflow, Google BigQuery, dbt, GitHub

## Video description Sanchit Srivastava - Sr. Manager Data Analytics & Architecture / Principal Architect, General Mills

What if the biggest bottleneck in your AI pipeline isn't your model — it's your data?

As AI systems grow more ambitious, real-world data is increasingly scarce, sensitive, or simply too expensive to collect at scale. This talk will explore how synthetic data generation has evolved from a niche workaround into a production-grade engineering discipline — one that lets teams build high-fidelity training sets without compromising user privacy or regulatory compliance.

We will walk through the core techniques, tradeoffs, and architectural patterns that separate synthetic data that works from synthetic data that misleads — covering everything from statistical fidelity and bias mitigation to privacy guarantees like differential privacy. Attendees will leave with a practical framework for deciding when and how to use synthetic data in their own pipelines, along with real-world lessons on scaling it responsibly.

If you're building AI systems where data quality, privacy, or scale is a constraint — this talk is your blueprint.

[Source](https://www.youtube.com/watch?v=THHxbNNVkTQ)

中文简报

通用磨坊利用合成故障训练自愈流水线AI 构建模拟基础设施故障的规范化案例库，使AI智能体能够在不泄露敏感数据的情况下，实时诊断并解决生产流水线崩溃问题。 Sanchit Srivastava将合成数据重新定义为贯穿六个编程类别的多维度运行时资产（涵盖从统计生成到基于模拟的代理），而非仅是构建期的测试构件。他解释了合成虚拟层如何保护敏感企业数据表免受LLM提示词泄露，并实现安全的智能体推理。结合通用磨坊（General Mills）的实际落地应用，他详细介绍了一个自主自愈数据流水线智能体，该智能体基于Pub/Sub、Dataflow、dbt和BigQuery上合成模拟故障模式的向量化案例库训练而成。演讲最后总结了关键架构经验，包括在原生云巨头平台中整合向量基础设施、从静态冷存储转向动态案例库，以及实施严格的统计治理以防止漂移。合成数据已从简单的QA Mock测试演变为涵盖部分合成生成和基于模拟系统的进阶工程学科。 (0:04) 编程式合成数据可分为零真实数据方法（统计、基于规则、异常注入）以及由数据驱动的生成方法（如CTGAN与模拟）。 (1:28) 合成数据现已作为运行时推理代理发挥作用，隔离敏感生产数据库与LLM的直接交互，并防止记忆或提示词泄露。 (4:45) 通用磨坊通过跨Pub/Sub、Dataflow、dbt和BigQuery等工具合成生成数百种故障模式，开发出了自愈流水线智能体。 (8:54) 将故障日志、修复措施和处理结果组织成规范的JSON格式，可防止上下文膨胀，并消除RAG驱动排障分流中的幻觉。 (12:31)

Reading summary

Sanchit Srivastava outlines how synthetic data has evolved from build-time testing into an active architectural layer that protects sensitive records during AI reasoning and powers autonomous, self-healing data pipelines. The Expanding Spectrum and Generation Categories (1:28) Synthetic data engineering has advanced beyond legacy dummy records into partial synthetic generation and simulation-based systems. Partial synthetic datasets preserve mathematical distributions and structural relationships to resolve sparse classes and feature blind spots in machine learning pipelines. Simulation-based systems generate deliberate failure modes, edge cases, and behavioral guardrails that production systems cannot safely execute, providing training material for autonomous reasoning agents. Programmatic generation divides into methods requiring zero exposure to production data and data-informed techniques. Zero-exposure methods eliminate leakage risks through statistical feature modeling (such as scikit-learn classification utilities), rule-based schemas matching data contracts, and deliberate outlier injection using frameworks like PyOD. Conversely, data-informed techniques—including Conditional Table GANs (CTGAN), perturbation-based data augmentation, and operational simulations—learn from seed samples to generate high-utility synthetic records for fraud detection and adversarial evaluation. Runtime Reasoning Proxies and Privacy Guardrails (4:45) In modern data architectures, synthetic data serves both as a build-time contract validation asset and a runtime operational barrier. Compliance mandates such as GDPR and CCPA restrict sensitive customer PII from touching non-production sandboxes or external AI vendor environments with weaker security controls. Testing against high-fidelity synthetic proxies allows organizations to safely evaluate rare edge cases without risking sensitive enterprise records. In production LLM and agentic workflows, synthetic datasets mitigate prompt injection and latent memory leakage, preventing models from surfacing ingested training data during extraction attacks. Instead of allowing autonomous agents to query raw production tables directly, systems can place a synthetic virtual layer in front of the model. The agent executes intermediate chain-of-thought iterations against the synthetic proxy and accesses production storage only after the final operational query or plan is validated. Self-Healing Pipelines via Simulated Failure Catalogs (8:54) To counter distributed pipeline failures caused by infrastructure limits, schema drift, and brittle data contracts, General Mills engineered an autonomous AI triage agent. Rather than waiting months to collect organic production incidents across Google Cloud Dataflow, Pub/Sub, BigQuery, and dbt, the team synthetically simulated hundreds of failure modes. Simulated disruptions—such as Dataflow out-of-memory errors and Pub/Sub subscriber backpressure—were captured, replayed, and compiled into an indexed vector Case Library. To prevent latency spikes, context window bloat, and hallucinations from unformatted logs, diagnostic records are normalized into a canonical JSON schema detailing error signatures, root causes, prescribed engineering actions, and verified outcomes. When live failures occur, the agent queries the Case Library via Retrieval-Augmented Generation (RAG) to conduct root cause analysis, presenting proposed remediation plans to a human-in-the-loop engineer before automated execution. Colocated Vector Storage and Living Case Libraries (19:21) Deploying the self-healing agent demonstrated the operational advantage of architectural consolidation. Srivastava notes that initial deployments using standalone third-party vector databases incurred cross-network retrieval latency that violated pipeline triage SLAs. In response, the team colocated vector embeddings, execution logs, and compute inside the native cloud hyperscaler housing the data platform, reducing operational knobs and integration points. The triage framework currently operates as a "cold store" constrained by a bounded taxonomy of human-authored resolutions. However, because the LLM agent frequently reasons out fixes superior to existing standard operating procedures, ongoing development focuses on transforming the platform into a "hot," living case library. This next phase requires engineering automated feedback loops that evaluate novel agent proposals and write post-action outcomes directly back into the vector store. Srivastava emphasizes that synthetic pipelines require strict statistical governance and continuous drift monitoring against production distributions to prevent downstream model failure, noting that automating the transition from static action catalogs to living feedback loops remains an ongoing architectural effort.

精炼阅读摘要

通用磨坊（General Mills）工程团队指出，合成数据已从传统的构建期测试工具演进为全周期核心资产，不仅能提供隐私屏障，还能作为大模型的运行时推理代理；通过主动模拟基础设施故障生成高保真案例库，企业可构建起具备自愈能力的自主数据管道智能体。合成数据范式演进与六大生成路径 (0:04) 合成数据的应用范畴已彻底超越基础 QA 与单元测试的传统认知，延伸至包含部分合成与系统仿真的全谱系架构。部分合成数据专门针对生产环境中具有严重稀疏特征或类别失衡的机器学习场景，在不使用真实用户数据的前提下严格保全数学分布与特征关联；仿真系统则主动生成极端的系统故障模式与行为边界，为自动化运维智能体提供训练基础。在程序化生成技术上，工程实践划分为两大流派共六种具体路径。第一类完全零接触真实数据，彻底消除泄漏风险，涵盖基于统计分布的生成（如借助 scikit-learn 的分类与回归生成器）、基于规则与数据契约的模板生成（如 Faker 生成符合特定校验规则的结构化实体），以及使用 PyOD 等框架主动按比例注入异常与污染指标的离群点注入技术；第二类则依赖真实样本驱动，包括利用条件生成对抗网络（CTGAN）对抗建模多维分布、通过数据增强执行语义扰动，以及面向非常规状态进行系统环境仿真。运行时推理代理与合规隔离屏障 (4:45) 合成数据在工程生命周期中的定位正发生结构性转变，从单纯的构建期静态验证拓展为运行时的动态推理代理。在 GDPR 与 CCPA 等合规法规约束下，企业严禁将包含真实个人可识别信息（PII）的生产数据暴露于防护相对薄弱的非生产沙盒或第三方 AI 服务商环境。高保真合成数据为评估罕见边界条件提供了安全可控的代用数据源。在以大语言模型（LLM）为核心的生产流程中，合成数据发挥着双重防护作用。一方面，针对模型潜在记忆效应所引发的提示词注入或提取攻击风险，采用纯合成或部分合成数据进行预训练与微调可显著收敛数据泄露的爆炸半径；另一方面，架构师能够在生产存储层与 LLM 之间架设一层合成虚拟数据层，使智能体仅在合成沙盒内完成中间链路的迭代推理与思维链验证，只有当最终查询指令或操作规划通过校验后，才获准调度底层生产环境执行。故障注入机制与规范化自愈案例库 (8:54) 面对分布式数据管道因基础设施瓶颈、Schema 漂移和脆弱契约频繁引发的运行中断，团队未选择耗费数月等待生产环境自然暴露错误，而是主动向技术栈注入合成异常。工程师针对 Google Cloud Pub/Sub 的背压限流、Google Cloud Dataflow 在突发流量下的内存溢出（OOM）、Google BigQuery 的全表扫描超时，以及 dbt 上游字段类型变更导致的漂移，程序化模拟了数百种软硬件故障模式并完整捕获运行日志。为避免将未经清洗的原始报错日志直接传入 LLM 造成上下文窗口膨胀、响应延迟与推理幻觉，系统将所有的故障特征、处置举措及最终成效统一重构为规范化的 JSON 格式，明确包含技术类型、错误签名、根本原因、执行动作与验证状态，随后将其向量化索引为检索增强生成（RAG）案例库。在生产运行时，智能体通过 Top-K 向量相似度检索历史方案，自主完成多轮根因分析（RCA）并生成修复策略，经人工确认（HITL）后下发执行，进而将验证结果写回案例库形成闭环。单平面架构整合与动态统计治理 (16:22) 在基础设施落地阶段，团队最初采用独立的第三方向量数据库，但跨网络检索带来的网络延迟与运维摩擦直接冲击了管道修复的服务等级协议（SLA）。依据减少控制节点以提高管道健壮性的工程原则，架构最终将向量嵌入、管道日志、案例知识库以及核心计算负载全面整合至同一原生云超大规模服务商（Hyperscaler）内部，通过就近部署消除了网络开销并降低了集成断点。针对当前系统仍属于静态查询“冷库”且受限于人工预设动作分类法的现状，团队正在推进向“动态热案例库”演进，旨在建立实时执行反馈链路，并使系统具备动态评估和采纳智能体自主生成的优选策略的能力。此外，Srivastava 强调了严谨统计治理的必要性：若合成生成过程缺乏严格的统计参数约束，极易引发人工偏度并破坏特征间的共依关系，导致下游分类模型在评估中全面崩溃，因此工程团队必须持续监控并严格收敛流式生产数据与合成数据分布之间的统计距离。该方案论证了通过受控合成故障构建高保真知识库以驱动数据管道自愈的可行性，但当前架构仍处于受限于预定义动作和静态冷库的初级阶段；如何在消除人工预设的同时实现智能体新策略的安全回流，并在持续数据漂移下维持合成数据分布的统计无偏性，仍是规模化落地中未完全解决的核心难题。

Detailed notes — zh

## Background 本内容由 Ai4 频道于 2026 年 8 月 18 日在 YouTube 平台发布，主讲人为通用磨坊（General Mills）高级数据分析与架构经理兼首席架构师 Sanchit Srivastava。演讲围绕合成数据如何从权宜之计演变为生产级数据工程学科展开，重点探讨如何在保障隐私合规的前提下构建高保真训练集与自愈数据管道。 ## TL;DR 演讲系统拆解了合成数据的演进历程与六大程序化生成方法，强调其已从构建期测试拓展至生产运行时的推理代理与安全护栏。针对脆弱的数据管道，团队利用合成脏数据与故障模拟构建了规范化的“案例库”，结合 RAG 与 AI Agent 实现了高效的管道自愈与精准根因分析（RCA）。同时，演讲者分享了避免跨服务分散、回归单云原生架构的工程经验，并指出了冷启动演进、动态分类法拓展以及严格统计学治理对防止数据漂移的关键作用。 ## Atmosphere & style 整场演讲充满务实且极具洞察力的工业界工程气息，主讲人语言凝练、节奏紧凑，将抽象的架构设计与生动的现实比喻（如“达芬奇作画与幼童涂鸦”）巧妙结合。分享兼具技术深度与实践反思，既有直击痛点的架构权衡，也有对前沿自愈系统的理性展望，给听众带来扎实、清醒且极具启发性的技术沉浸感。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) 合成数据已从单纯用于单元测试的完全合成数据，演进出保留分布结构的部分合成数据以及用作运行时安全护栏的模拟合成数据。 - [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) 统计驱动、基于规则模板与异常离群值注入等程序化生成方式无需真实数据介入，可从源头上消除数据泄露风险。 - [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) 生成式深度学习（如 CTGAN）能通过对抗网络从少量样本中学习真实分布，并同时生成正向与负向边缘场景数据以用于安全防御与欺诈检测。 - [04:46](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=286s) 合成数据的架构定位已延伸至生产环境运行时，可作为大模型与敏感生产数据之间的“推理代理”，有效防御 Prompt 注入与敏感数据记忆泄露。 - [04:46](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=286s) 高仿真合成数据有效化解了 GDPR/CCPA 合规限制、第三方 AI 沙箱数据泄露以及长尾极端异常样本稀缺等核心工程痛点。 - [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) 基于模拟合成数据构建的自愈数据管道（Self-Healing Pipelines）AI Agent，能够弥合数据管道精细构建与业务即时消费之间的断层。 - [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) 自愈 Agent 在构建期主动向技术栈注入故障并生成模拟日志，将其解析为高度结构化的规范格式（Canonical Format）以构建合成案例库。 - [11:29](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=689s) 合成案例库不仅提供处置动作，更通过为故障映射预期成败标签（成功、部分成功、失败）提供决策背后的完整推理依据，避免大模型幻觉。 - [11:29](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=689s) 自愈系统采用人在回路（Human-in-the-loop）机制，AI 代理给出根因分析和修复建议后由工程师确认执行，最终结果再回流沉淀至案例库形成闭环。 - [14:15](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=855s) 针对 Pub/Sub、Dataflow、BigQuery 和 dbt 等核心组件的常见故障进行模式化建模，未经清洗的原始日志直接输入大模型会导致幻觉激增与准确率下降。 - [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) 遵循“系统旋钮越少管道越稳健”的架构法则，将向量存储、案例库与数据底座收敛在同一主流云平台内，可显著降低跨服务网络检索延迟并提升鲁棒性。 - [19:17](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1157s) 自愈架构正在向突破冷启动限制的“动态热案例库”演进，并致力于打破预定义操作词典的受限分类法，以安全吸收大模型自主探索出的更优解。 - [19:17](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1157s) 合成数据生成必须遵循严格的统计学治理，防止数据偏斜（Skewness）与数据漂移（Data Drift），确保合成数据与真实业务数据分布高度一致。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) 合成数据的演进与六大程序化生成类别在过去，合成数据主要被视为仅用于单元测试和功能验证的“完全合成数据”（Fully Synthetic Data）。然而随着 AI 与高级机器学习的发展，合成数据的形态已经发生了深刻演化，不仅出现了用于填补真实数据空白的“部分合成数据”（Partial Synthetic Data，保留真实数据的特征与分布结构但非真实记录），还演化出了“基于模拟的合成数据”（Simulation-based Synthetic Data）。这种模拟数据完全由系统生成，能够作为应用运行时的安全护栏（Guardrails），防止 AI 做出预期之外的危险操作。 ```mermaid flowchart TD accTitle: 六大合成数据程序化生成方法分类 subgraph ZeroData ["无需真实数据 (零数据风险)"] A["1. 统计驱动生成 (scikit-learn: make_classification/regression)"] B["2. 基于规则/模板生成 (姓名、地址、邮箱等格式规范)"] C["3. 异常与离群值注入 (PyOD: 自定义污染率与边界测试)"] end subgraph DataDriven ["需要真实数据/规则驱动 (高级与模拟)"] D["4. 生成式深度学习 (CTGAN / 对抗网络正负样本生成)"] E["5. 数据增强 (转换规则与特征扩充)"] F["6. 基于模拟的生成 (构建系统护栏与代理推理环境)"] end ``` 生成合成数据主要涵盖六大程序化驱动的类别，彻底摆脱了传统的硬编码生成方式： - **统计驱动生成（Statistical-driven Generations）**：允许用户定义所需特征与分类维度，由程序生成符合统计分布的数据。典型工具库包括 scikit-learn 中的 `make_classification` 与 `make_regression`。 - **基于规则的模板数据（Rule-based Templated Data）**：生成姓名、物理地址、电子邮件等格式真实但内容虚构的数据。 - **异常与离群值注入（Anomaly & Outlier Injections）**：不用于生成“正常”业务数据，而是主动向数据集中注入异常，支持精确配置“污染程度”（Contamination Level），用于训练 AI 系统的鲁棒性与异常识别能力。Python 领域中最具代表性的是 `PyOD` 库。 - **生成式深度学习（Generative Deep Learning）**：基于生成对抗网络（GAN）架构与红蓝对抗思路，只需向系统提供少量真实数据样本，模型即可学习真实分布并同时生成正向与负向边缘场景数据。代表性工具库为 `CTGAN`（需注意使用时因输入了真实样本需做好隐私隔离），常用于欺诈检测系统或 AI 防火墙。 - **数据增强（Data Augmentation）**：基于转换规则对现有数据进行扩充与变形。 - **基于模拟的生成（Simulation-based Synthetic Data）**：通过建立业务仿真逻辑，生成原本在真实世界中难以收集或极度罕见的数据集。前三种方法的最大优势在于完全不需要真实数据介入，仅需以参数化形式提供数据 Schema 和规则定义即可生成，从而从根本上切断了数据泄漏风险。 --- ### [04:46](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=286s) 从构建期到运行时：合成数据在新时代的架构定位与工程价值合成数据的应用阶段正经历根本性的架构转变。过去它仅活跃在系统构建与开发期（如单元测试），如今已贯穿至生产环境的“运行时”（Runtime）。在构建期，合成数据用于保护隐私并校验数据契约（Data Contracts）；而在运行时，合成数据层则作为大模型与敏感生产数据之间的**“推理代理”（Reasoning Proxy）**，为 AI 决策提供隔离层。在数据工程实践中，合成数据解决了多个核心痛点： - **合规与非生产环境安全**：GDPR、CCPA 等法规对敏感数据与 PII（个人身份信息）有着严苛限制。缺乏严密安全护栏的开发、测试和非生产环境绝不能引入真实数据，高仿真合成数据能让研发团队在无合规风险的前提下开展全功能验证。 - **第三方 AI 工具与沙箱隔离**：接入第三方 AI 厂商往往需要将数据传输至外部沙箱，而这些沙箱的安全边界往往达不到企业内部的安全标准。使用合成数据测试 AI 系统，既能全面验证各类边界场景，又能规避真实核心资产外泄。 - **数据科学中的长尾异常建模**：真实世界的数据分布通常极度不均衡，数据科学家构建特征工程模型往往极度依赖罕见的异常样本。合成数据生成策略可以直接模拟这些罕见场景，使模型在上线前就具备应对极端情况的能力。 - **防御大模型记忆与 Prompt 泄漏**：AI 系统具有记忆能力，存在将训练数据或上下文泄漏给外部用户的潜在风险，同时还面临 Prompt 泄漏（Prompt Leakage）与 Prompt 注入攻击。通过构建一层无限逼近生产特征的虚拟合成数据层，大模型可以在合成层内完成复杂的思维链推理（Reasoning Loops），唯有最终形成明确行动决策后，才被允许访问真实的生产数据层执行动作。 --- ### [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) 自愈数据管道实践：基于模拟合成数据的 AI Agent 架构在现代数据栈中，“AI 的能力上限取决于数据质量”。然而，构建数据管道是一门需要精雕细琢的慢工艺术，正如演讲者所比喻的那样：“构建数据管道就像达芬奇在精心作画，需要耗费大量时间；而业务端消费数据却像我家刚学会走路的幼童在墙上涂鸦——他不仅立刻就要，而且要求一切必须绝对精准。”面对脆弱的数据契约和不可避免的管道故障，团队基于模拟合成数据构建了一款用于**“管道自愈”（Self-Healing Data Pipelines）**的 AI Agent，以弥合数据构建与即时消费之间的断层。 ```mermaid flowchart LR accTitle: 管道自愈 AI Agent 构建期案例库生成流程 subgraph FailureSimulation ["故障模拟与日志生成"] A["技术栈故障注入 (dbt / Dataflow 等)"] --> B["生成模拟合成日志 (Synthetic Logs)"] end subgraph Normalization ["标准化解析"] B --> C["解析为规范化格式 (Canonical Format)"] end subgraph KnowledgeBase ["代理知识库构建 (Build Phase)"] C --> D["构建合成案例库 (Synthetic Case Library)"] D --> E["索引关联: 故障特征 ➔ 处置动作 ➔ 预期结果"] end ``` 该 Agent 在构建期（Build Phase）的核心实现机制如下： - **技术栈故障模拟**：针对组织内部采用的核心组件（如 dbt、Dataflow 及各类 Pop 管道），主动注入各类预设故障并生成模拟合成错误日志（Synthetic Logs）。 - **规范化格式解析（Canonical Format）**：将分散杂乱的系统报错日志统一解析并存储为高度结构化的规范格式，这种格式不仅语义清晰，而且极大提升了 AI 系统的检索召回精度。 - **构建合成案例库（Synthetic Case Library）**：案例库充当 AI Agent 的核心索引知识库。它为每种特定故障建立索引档案，明确记录对应的处置动作、预期产生的业务影响，以及历史处置记录。在真实生产环境发生实时故障（Live Failure）时，Agent 将基于该案例库快速推理出根因并推荐修复指令。 ### [11:29](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=689s) 案例库（Case Library）架构与合成数据驱动的故障注入在数据管道运维中，AI 代理（Agent）持续监听并观察生产环境中的实时失败日志。当故障发生时，代理并非直接盲目报错，而是检索一个预先生成的丰富“案例库”（Case Library），在其中进行语义推理，找出当前故障最接近的历史场景并推导出正确的应对操作。普通的日志检索最多只能给出一种粗暴的处理动作，而一个高质量的案例库则能提供“采取该动作背后的推理依据”（the reason to take an action）。这正是合成数据发挥决定性价值的关键所在。当前流程中包含人在回路（Human-in-the-loop, HITL）机制：AI 代理给出建议动作后通知工程师确认，执行后的最终结果再回流沉淀至案例库中，形成闭环。 ```mermaid flowchart TD accTitle: 基于合成案例库的故障自愈推理流程 subgraph Build["构建阶段 (Build Phase)"] A["合成坏数据 / 异常基建"] --> B["触发管道故障并捕获日志"] B --> C["数据工程师映射解决方案 (Action & Outcome Mapping)"] C --> D["结构化规范案例库 (Canonical Case Library)"] D --> E["向量化存储 (Vector Storage)"] end subgraph Runtime["运行时推理 (Runtime Phase)"] F["生产环境实时故障日志"] --> G["AI 代理 (RAG 检索 Top-K 案例)"] E -.-> G G --> H["根因分析 (RCA) 与动作建议"] H --> I["工程师确认 (Human-in-the-Loop)"] I --> J["执行修复并评估最终结果"] J --> D end ``` 为了构建该案例库，团队针对数据管道依赖的核心技术栈，主动生成各种模拟故障。他们通过注入合成的“脏数据”（bad data）或人为制造劣质的底层基础设施支持来诱发错误。捕获真实报错日志后，团队反复重放这些故障场景，并由资深数据工程专家为每个故障分配固定的解决方案，同时明确标记该方案的预期结果——是“完全成功”、“部分成功”还是“执行失败”。随后，将所有故障模式、修复操作及对应结果全部向量化，形成供检索增强生成（RAG）调用的基石。 - **核心价值差异**：“日志检索只能给你一个动作，而优质的案例库能给你采取该动作的完整理由。” - **解决方案映射（Resolution Mapping）**：为每个合成故障打上预期成败标签（成功、部分成功、失败），避免模型在检索时产生幻觉。 --- ### [14:15](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=855s) 常见技术栈的故障模式与规范化数据结构演讲者深入拆解了案例库针对企业常见技术栈的具体故障建模。企业数据管道通常涉及实时摄取、流计算、数仓和数据转换等组件，每种组件都有典型且可预测的故障模式： | 技术组件 | 常见故障场景（Failure Scenario） | 合成模拟与应对操作（Action） | 预期结果（Outcome） | |:--- |:--- |:--- |:--- | | **Pub/Sub** | 实时摄取遭遇限流（Throttled），消息被拒绝或延迟堆积 | 触发退避重试或扩容摄取配额 | 消息吞吐恢复正常（Success） | | **Dataflow** | 流处理作业内存溢出（OOM） | 高吞吐合成数据压测重现；增加内存配置、降低单批次体量、重构分块（Chunking）或调整分区键 | 管道吞吐平稳，无内存崩溃 | | **BigQuery** | 分区不当或大查询超时（Timeout） | 在查询中增加分区（Partition）或聚簇（Clustering）条件 | 查询性能显著提升，超时消除 | | **dbt** | 上游表结构漂移（Schema Drift） | 自动回滚 Schema 或执行适配迁移操作 | 数据模型转换成功完成 | 在架构设计上，团队踩过的一个重要误区是直接将杂乱的原始日志塞给大模型。实践证明，未经清洗的原始日志会导致检索响应变慢、模型幻觉急剧上升且准确率大幅下跌。因此，必须将“故障模式-修复动作-执行结果”整理成极度清晰、统一的**规范化格式（Canonical Format）**，才能让 AI 代理高效向量化并进行精确推理。 --- ### [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) 运行时推理、架构精简与超大规模云原生落地在运行时，当某条生产管道发生崩溃，AI 代理会基于规范化的案例库执行 RAG 检索。如果仅给模型提供最新单次日志，它只能做浅层的模式匹配；但当案例库完整记录了历史故障、过往采取的动作及其实际成败时，代理就能够展开深度反思：“上次出现该错误时执行了操作 A，但结果只达到部分成功，因此本次应优先采取操作 B。” ```mermaid flowchart LR accTitle: 集中式单云架构 vs 跨服务分散架构 subgraph AntiPattern["高延迟架构（反面模式）"] direction TB A1["管道日志 / 数据源"] -->|跨网络调用| B1["独立第三方向量服务"] B1 -->|高网络延迟| C1["AI 推理引擎"] end subgraph Optimized["超大规模云原生架构（推荐）"] direction TB subgraph Cloud["统一云平台 (Hyperscaler)"] A2["管道日志 & 案例库"] --- B2["原生向量化存储"] B2 --- C2["AI 代理推理"] end end ``` 这种机制显著压缩了根因分析（RCA）的时间。考虑到现实中绝大部分管道中断都是由确定性、静态的原因导致，AI 代理给出的 RCA 准确率非常高。在基础设施选型方面，团队最初曾尝试采用独立的第三方向量数据库服务，但很快发现跨基础设施调用的网络检索延迟完全无法满足生产需求。最终，他们选择回归“架构基础法则 101”——**系统旋钮越少，管道就越稳健**（"if you have lesser knobs, you have more robust pipelines"）。他们将数据、管道日志、案例库与向量索引全部收敛在同一个主流云服务商（Hyperscaler）内部，大幅提升了检索性能与系统鲁棒性。 - **RCA 提效显著**：结构化案例检索直接去除了人工翻阅排查日志的繁琐流程。 - **架构收敛**：避免引入过多外部微服务，将向量存储与数据底座保持在同一平台，降低运维复杂度与延迟。 --- ### [19:17](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1157s) 冷启动突破、受限分类法与合成数据治理实践目前该体系仍处于演进阶段，团队正在着手解决两大核心挑战： 1. **冷启动与静态存储限制**：现有案例库在初始状态下属于“冷存储”（Cold Store），团队正致力于将其改造成实时连接、能够动态吸收生产反馈的“热案例库”（Hot Case Library）。 2. **打破受限分类法（Bounded Taxonomy）的枷锁**：当前 AI 代理的修复建议被严格限制在预定义的操作词典内。但在实际运行中，团队发现大模型有时能够提出比人类专家预设方案更优越的解决思路。如何安全地捕获并自动吸收模型自主探索出的更优解，是下一阶段的研发重点。最后，演讲者分享了合成数据工程的关键治理规范。在生成合成数据时，必须保持严格的统计学治理（Governed Statistical Manner），绝不能引入过度的数据偏斜（Skewness），必须确保合成数据与真实业务数据在分布上高度契合。一旦合成数据与真实分布脱节，下游 AI 分类与建模任务就会全盘崩溃。此外，还需建立持续的监控机制以防止数据漂移（Data Drift）。演讲者指出完整的原型代码已开源至 GitHub 仓库，欢迎业内同行通过 Gmail 深入交流与探讨。 ## Notable quotes > [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) 构建数据管道就像达芬奇在精心作画，需要耗费大量时间；而业务端消费数据却像我家刚学会走路的幼童在墙上涂鸦——他不仅立刻就要，而且要求一切必须绝对精准。 — Sanchit Srivastava > [08:54](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=534s) AI 的能力上限取决于数据质量。 — Sanchit Srivastava > [11:29](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=689s) 日志检索只能给你一个动作，而优质的案例库能给你采取该动作的完整理由。 — Sanchit Srivastava > [11:29](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=689s) the reason to take an action — Sanchit Srivastava > [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) 系统旋钮越少，管道就越稳健。 — Sanchit Srivastava > [16:22](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=982s) if you have lesser knobs, you have more robust pipelines — Sanchit Srivastava > [00:04](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=4s) 前三种方法的最大优势在于完全不需要真实数据介入，仅需以参数化形式提供数据 Schema 和规则定义即可生成，从而从根本上切断了数据泄漏风险。 — Sanchit Srivastava > [04:46](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=286s) 大模型可以在合成层内完成复杂的思维链推理，唯有最终形成明确行动决策后，才被允许访问真实的生产数据层执行动作。 — Sanchit Srivastava > [14:15](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=855s) 未经清洗的原始日志会导致检索响应变慢、模型幻觉急剧上升且准确率大幅下跌。 — Sanchit Srivastava > [19:17](https://www.youtube.com/watch?v=THHxbNNVkTQ&t=1157s) 在生成合成数据时，必须保持严格的统计学治理，绝不能引入过度的数据偏斜，必须确保合成数据与真实业务数据在分布上高度契合。 — Sanchit Srivastava ## Mentioned Sanchit Srivastava, General Mills, Ai4, scikit-learn, PyOD, CTGAN, dbt, Google Cloud Dataflow, Google Cloud Pub/Sub, BigQuery, GDPR, CCPA, GitHub, Gmail ## Video description Sanchit Srivastava - Sr. Manager Data Analytics & Architecture / Principal Architect, General Mills What if the biggest bottleneck in your AI pipeline isn't your model — it's your data? As AI systems grow more ambitious, real-world data is increasingly scarce, sensitive, or simply too expensive to collect at scale. This talk will explore how synthetic data generation has evolved from a niche workaround into a production-grade engineering discipline — one that lets teams build high-fidelity training sets without compromising user privacy or regulatory compliance. We will walk through the core techniques, tradeoffs, and architectural patterns that separate synthetic data that works from synthetic data that misleads — covering everything from statistical fidelity and bias mitigation to privacy guarantees like differential privacy. Attendees will leave with a practical framework for deciding when and how to use synthetic data in their own pipelines, along with real-world lessons on scaling it responsibly. If you're building AI systems where data quality, privacy, or scale is a constraint — this talk is your blueprint. [Source](https://www.youtube.com/watch?v=THHxbNNVkTQ)

Source quotes

[8:54] Sanchit Srivastava: 构建数据管道就像达芬奇在精心作画，需要耗费大量时间；而业务端消费数据却像我家刚学会走路的幼童在墙上涂鸦——他不仅立刻就要，而且要求一切必须绝对精准。 [8:54] Sanchit Srivastava: AI 的能力上限取决于数据质量。 [11:29] Sanchit Srivastava: 日志检索只能给你一个动作，而优质的案例库能给你采取该动作的完整理由。 [11:29] Sanchit Srivastava: the reason to take an action [16:22] Sanchit Srivastava: 系统旋钮越少，管道就越稳健。 [16:22] Sanchit Srivastava: if you have lesser knobs, you have more robust pipelines [0:04] Sanchit Srivastava: 前三种方法的最大优势在于完全不需要真实数据介入，仅需以参数化形式提供数据 Schema 和规则定义即可生成，从而从根本上切断了数据泄漏风险。 [4:46] Sanchit Srivastava: 大模型可以在合成层内完成复杂的思维链推理，唯有最终形成明确行动决策后，才被允许访问真实的生产数据层执行动作。 [14:15] Sanchit Srivastava: 未经清洗的原始日志会导致检索响应变慢、模型幻觉急剧上升且准确率大幅下跌。 [19:17] Sanchit Srivastava: 在生成合成数据时，必须保持严格的统计学治理，绝不能引入过度的数据偏斜，必须确保合成数据与真实业务数据在分布上高度契合。

Source quotes

[0:04] Sanchit Srivastava: synthetic data are we still talking about testing? [8:54] Sanchit Srivastava: Da Vinci drawing something [8:54] Sanchit Srivastava: like my toddler drawing it on the wall—he wants it now, he wants it right, and everything should be accurate. [8:54] Sanchit Srivastava: a certain log retrieval can give you an action; a good case library gives you the reason to take an action. [16:22] Sanchit Srivastava: This failure happened previously; action X was taken, resulting in partial success, whereas action Y resolved it completely. [16:22] Sanchit Srivastava: Architecture 101: where if you have lesser knobs, you have more robust pipelines. [19:21] Sanchit Srivastava: cold store [19:21] Sanchit Srivastava: a living and a hot case library. [19:21] Sanchit Srivastava: bounded taxonomy [20:13] Sanchit Srivastava: Your classification will go all over the places.

Original transcript

[0:04] Okay. So today we'll talk about synthetic data and I'll try to bring together synthetic data AI and [0:11] how it all helps in data engineering. Right? We all know the root cause still data. I was asked a [0:18] question today morning synthetic data are we still talking about testing? So that's why I added this [0:22] slide. I want to give very quickly very quickly I want to give this audience what synthetic data [0:27] is and how has it evolved in the recent world. We all know what synthetic data is. You know it [0:33] was fully synthetic back in the days when we were using it only for testing purposes but the world [0:39] has changed now. We are seeing the evolution of partial synthetic and sometimes stimulations based [0:44] synthetic data. Now partial synthetic data synthetic data is used very heavily in your [0:49] advanced ML and in your advanced AI use cases where you're trying to fill some gaps in the [0:54] data right so this is the synthetic data where data is not real but it has a character and the [1:00] structure of real data and stimulated stimulation based synth data is the data which is absolutely [1:07] generated by you but has value it is a guardrail for you which you don't want your applications [1:15] to be doing. We'll talk a lot more a lot about simulated data today. Uh because the agent we are [1:21] going to talk about is going to be built on that simulated data. Um I also want to give this team [1:28] six key categories you can generate synthetic data. Right? These are all programmatic driven [1:32] categories. Do not want you to go back to old traditional methods. The six key categories [1:36] to generate synthetic data are statisticaldriven generations. Here is where you can define the kind [1:42] of features you want, the kind of classifications you want and programs will build your statist [1:47] statisticaldriven data for you. In the next slide, you will see the libraries you can use. If you are [1:53] familiar of scikit libraries, make classification and make regressions are very good libraries to [1:58] generate this kind of data. And then there are rule-based templated data. This is a data where [2:04] the values are not real but the structure around the data is um addresses, names, email addresses [2:10] etc. Right? So there are very good libraries out there. In the next slide I'll show you that [2:15] u you can use those for your programs. And then anomaly and outly injections these are [2:21] Python based [clears throat] libraries. They are special. These are not the data which will help [2:27] you to generate synthetic data for your good use cases. But you can self-inflict anomalies [2:34] in those data elements. You can determine how much of contamination you want and how much of [2:39] contamin contamination level you want your AI systems to be trained on. So very good a very [2:45] good library is Python OD. It's a great library to use. Next I will have the format too. The top [2:52] the benefit of top three uh categories are you do not need your real data for it. So with AI we know [2:59] the rest comes along. So for the first three you don't need your real data. You just have [3:03] to give definition of your data in a very simple parameterized manner and the applications will do [3:08] that for you. The bottom three are interesting. Generative deep learning right? So this is based [3:14] on adversial network uh generative adversial network strategy. This is where a lot of red team [3:20] blue teams uh pictures comes in. There you give a sample of your data to programs and then they [3:28] learn from your data and generate both positive and negative scenarios data for you. Very useful [3:34] uh a very useful library is CTJ. Uh it's it's very useful and we will see how we have used this uh [3:41] in our use cases. And then data augmentation and simulation based are the app categories [3:48] where you can generate the data. You're actually generating the data either you're augmenting it [3:53] by using transformation rules or you are actually generating the data which usually is not available [3:58] to you by generating stimulations. This slide has some very popular libraries among all these [4:06] categories and what are the key points to watch out for these libraries are. I'll just cover here [4:12] um CTJ right. So this is a library where you have to be a bit cautious because you are giving [4:17] uh inputs of your real data to AI systems right but this helps this [4:22] uh this application because it learns the trends of your data fill the gaps based on the [4:28] inputs you have given and if you want certain negative scenarios can generate can generate [4:32] those scenarios for you too very useful if you are building some fraud detection systems very [4:38] useful if you're bringing some firewall based AI systems this this is a very good library to [4:45] uh this is just a code base. So now I'm going to switch gears right we spoke about synthetic [4:49] data and how can you build it what are the key libraries to build it now we are going to see [4:53] where do they really need to be in the new real world right in the earlier world synthetic data [4:59] was very very heavy when we were developing our systems right you know unit testing and all kind [5:05] of testing used to happen on those datas I think that that world has moved over now synthetic data [5:11] now is a key validation phase not just during your build time but during your runtime time too and [5:17] I'll show you an example how we have done that by by putting synthetic data during runtime too. What [5:23] it does is during your build time using synthetic data not only you're ensuring your privacies not [5:28] only you're ensuring your data contracts but during runtime you're using the synthetic data [5:33] layer as a reasoning mechanism as a reasoning proxy for AI before it actually hits your [5:39] sensitive data. So this is the new place I feel synthetic data is and will belong going forward. [5:46] quickly touch upon why engineering teams need synthetic data. As we move over to [5:51] data engineering aspects of it u traditional reasons right sensitive data GDPR CCPA talk [5:58] about data sensitivity and PII you do not want your real data in your non-prod environments where [6:04] guardrails are not proper you want your teams your development teams your testing teams your [6:09] application development teams to be using your synthetic data which has very close resemblance [6:13] of your actual data and they can build the applications around it a lot of AI applications [6:20] and third party vendors are coming in right they do not follow your your security architecture we [6:25] know AI is coming up with sandbox concepts where their data will be moved over there and security [6:31] periphery around those sandbox is sandboxes may or may not be as per your standards and may or [6:37] may not be what your data deserves. So testing your AI systems over synthetic data gives you the [6:44] flexibility of trying out every scenario versus taking the risk of working on your actual data. [6:51] And data scientists usually found scenarios which are not quite common in actual data in real world [6:57] data but they like to build feature engineers feature models and feature engineering models [7:03] over exceptions. So you can use synthetic data uh strategies which discussed before to build the [7:09] model over those synthetic proxies and have your applications ready to handle those scenarios in [7:14] real time real time world and then there is new beast in the game called AI why do you need for [7:20] AI well we know AI has a memory and AI can leak it out definitely don't want your data to be going to [7:26] AI system where it has capacity to share it out to external world we have seen what has happened to [7:32] organizations and you don't want to repeat that. So train your LLMs, train your AI systems as much [7:38] as possible over partial or pure synthetic data before you unleash them into your into your uh [7:44] real real production data. Prompt leakage is becoming quite common, right? So you want to [7:50] avoid that by putting a synthetic data layer over your AI system. Hence ensuring that nothing goes [7:57] out as a prompt leakage or a prompt injection mechanism from your production data. And most [8:02] importantly right you want to have AI agents a good accurate answer to be provided by them right [8:11] for that reasoning is extremely important but you don't want reasoning to be happening actually on [8:18] your actual production data all the time there is where the sandbox boundaries comes into picture. [8:24] So you can build a virtual data layer which is very close which very closely resembles your [8:31] production data for reasoning loops and when the decision has been made by AI LLMs then they get [8:38] to go to the production data to actually surface out the action. I think this is the place where [8:43] synthetic data is going to play a big role and we have tried in our next slides you will see we [8:47] have tried doing it um in one of the project which in one of the product which we have built. Okay, [8:54] we spoke about synthetic data. We spoke about why do we need it? We spoke about how can we [8:58] generate it and now let's bring it home with a AI agent helping our data engineering teams. So [9:06] what we have done using synthetic data um and and simulated data scenario is we have tried [9:11] to build an agent which helps in healing the data pipelines. We all know AI is as good as [9:17] your data is. I think that is out of question now and we all agree on that and I think uh most of us [9:24] really agree data building data pipelines is like an art right uh it's like it takes time right it's [9:31] like Davinco drawing something versus consuming a data is like my toddler drawing it on the wall he [9:37] wants it now he wants it right and everything should be accurate there is a paramount shift [9:43] here you're building something which takes time you want something and you're using it in a way [9:48] The time is the factor. You want it super fast. So with this agent, [9:52] we have tried to narrow that gap. Pipelines will fail. Data contracts are weak. There's no doubt [9:58] why will they not fail. So this agent essentially is trying is a is an effort to narrow that gap by [10:05] giving this artist a visual what could have gone wrong when the pipeline failed and a very clear [10:11] instruction that this could be the action leading to a certain outcome. This is a highle flowchart, [10:20] flow diagram, architecture diagram you can think of. What we did, we use certain technologies in [10:27] our organization. We generated synthetic failures over those technologies. This is all stimulation [10:35] based synthetic strategy. We had DBT failures. We created synthetic locks for them. We had data [10:42] flow failures, pop for failures, lot of technology failures we can generate. We generated locks for [10:47] them and then we passed them and stored them into a canonical format. Canonical formats are highly [10:54] liked by AI systems because they are much readable and they are much accurately uh retrievable. And [11:00] we created something called as case library and we'll go in depth what case library looks like. [11:06] This case library is nothing but think of as a indexed information for agent to know for [11:12] a certain failure what actions you can take and what could be the possible outcome and what was [11:18] the last action which was taken when that failure happened. This is the build phase of this agent [11:24] and you build a synthetic case library during the build phase. When the live failure happens, [11:29] this agent is observing the logs of of life failures and then it goes to this synthetically [11:35] generated case library and it evaluates using this rich library and it reasons with this library that [11:45] what could be the late what could be the closest retrieval and write action possible on a certain [11:52] failure. A certain log retrieval can give you an action. A good case library gives you the reason [12:00] to take an action. And that's the key difference here. And that's why synthetic data is supremely [12:05] important. Once an action is determined by this case library and LDM interaction then obviously [12:12] we have a human in the loop. The human is is uh obviously informed that this could be the action [12:18] you can take and then a post action outcome is moved back into this case library. So this is [12:25] our flow diagram and we'll go into depth of each and every component. Let's go to case library. I [12:31] think this is the most important part and this is where synthetic data plays a significant role. As [12:35] I mentioned right for all the technologies we have we picked some certain key technologies [12:40] using which we create our data pipelines and we generated failures. We give we generated [12:45] failures by giving it synthetic bad data. We generated failure by giving it synthetic bad [12:50] infrastructure support. We captured those actual failures logs and we replayed them again and again [13:00] and then with the expertise of data engineer teams we assigned certain fixed actions to each of those [13:08] failures and then we we said this action is going to take success outcome this action could be a [13:15] partial success this action could be a failure success. So that is resolution mapping step. [13:21] And now we have our document or our table or you know our vector storage where we have possibly all [13:29] the failures which we are trying to solve possibly all the actions you can take on those failures and [13:35] possibly all the outcomes of those failures. Now it's just a matter of retrieving and retrieving [13:39] it fast and not hallucinating while retrieving it. So we vectorized it. We vectorized it and [13:46] we made it as a rag mechanism for our AI agent to pick it up. This is the build part and when [13:53] the runtime happens then our agent goes to this case library retrieve the the best similarity [14:01] option and execute or we have right now human in the loop propose a human to execute that whatever [14:08] the outcome is goes back to this library to make it as a living document. Let's unwrap case library [14:15] what it looks like. What you're seeing right now are the key technologies we use and I'm sure many [14:20] of the data teams use uh in the room right you use popsups right for your real-time injection. Common [14:26] failure in popsups is you're getting throttled right your messages are either getting you know [14:31] rejected or messages are not coming. We know the success criteria there what to do over there. So [14:37] you have a failure scenario you decided an action and you know the outcome of it. If you're using [14:42] streaming, if you're using data flow, you can run out of memory. We stimulated those scenarios by [14:47] giving it high volume synthetic data. Capture the log. Your action potentially is to add memory or [14:53] reduce the volume or do some chunking there or add some partition keys there and then you know [14:58] what the outcome could be. If you are using a database like BigQuery, you can run out of [15:03] uh you know your queries are not performing because a partition and system is timing out. [15:08] Well, now you have lo you now you have mentioned that in such cases either add a partition or a [15:13] clustering equation and the possible outcome could be performance could get better and in [15:18] case of DBT schema drift are very common. So you now you know the failure reason the outcome could [15:23] be roll out the schema whatever outcome works for your organization right in our certain cases [15:28] we decided to roll it out and then you know the outcome would be success. So what you're seeing [15:33] here is is nothing but association of common failure and data pipeline. What possible actions [15:39] you can take and the possible outcomes of that right sitting together in a in a library format. [15:48] Now important thing here based on our learning is you can do all of it in logs throw the logs [15:53] and things will work out. That does not work out very well right. So that's why what you're [15:57] seeing here is a very clear canonical format which is very easy to vectorize and very easy [16:03] for AIs to understand and reason around. If you throw a bunch of logs in there, we have [16:09] seen response time reducing hallistation creeping in and accuracy was not not just there. So a very [16:16] good canonical format is extremely important for this uh case library. This is just trying [16:22] to say that. Now let's move on to the runtime. What happens in runtime? When a pipeline fails, [16:29] you have a case library where all your possible actions and outcomes are already logged in. Your [16:36] AI agent has ragged has been ragged on that case library. It starts to reason among itself. If [16:45] you just give it the latest logs, it will just retrieve and take the action. But when you're [16:50] maintaining a library of all the failures, all the actions, all the outcomes and constantly feeding [16:56] it again and again, now it can reason it that this failure, the last effort I did was this action was [17:04] taken last time and the possible outcome was that and that gives it extreme power to make [17:09] much more accurate actions. So now this agent is essent essentially going to our case library [17:15] which has been vectorized to retrieve top K use cases and then reasoning around it is happening [17:21] and finally it comes out with hey you know this is your best possible root cause analysis take the [17:26] necessary action in here right now we have a human in loop we do not allow it to take uh action right [17:33] immediately by itself but in here we come a human comes in and you know determine uh whether um you [17:40] know the root cause analysis right or not. What we have seen is this has reduced our RC analysis [17:46] significantly and often times right pipelines fails because of very static reasons. Often times [17:52] the RC analysis by the agent is pretty accurate. This is just a code piece. The entire code is I'm [17:59] going to share the GitHub repository so you guys can go and take a look there. Right? So this is [18:03] another watch out which you guys you know which we realized when we were building this agent [18:08] u we have an option of vectorizing it and we have an option of using a vector service. We first went [18:14] on with using a vector service. Um but we realized the retrieval time was just not in was just not [18:23] working out for us if we are storing the vectors in a different uh infrastructure. So we decided to [18:28] vectorize it in a hyperscaler where both our data our pipeline logs our case case repository and [18:37] this vector will be will be stored. This is going back to architecture 101 where if you have lesser [18:42] knobs you have more robust pipelines. So strongly recommend you to see if you can store your case [18:48] library and vectorization of case library in the same place that will go a long way. So finally [18:55] what we just saw is synthetic data um actually landed this the created the whole concept using [19:03] synthetic data we created a bedrock of resolution and possible outcomes and then AI using that rag [19:11] mechanism is able to reason it properly and give us an accurate root cause analysis. Important [19:17] thing to call out is and this is something we're working out right now is how to keep it going. [19:21] Right now it's a cold store for us. We keep a we have a list of failures, action out outcomes. We [19:27] want to have a live connect back whenever the action is taken so that it gets more accurate. [19:36] I think we already spoke about that. So yeah, I mean what we have not solved is a cold start, [19:39] right? We want it to be a living and a hot case library. And also I think right it [19:45] is limited sometime by bounded texonomy. And what does that mean is agent can do only the [19:52] action which we have determined as the actions. Sometimes we have seen that agent proposes better [19:58] uh better actions better resolutions than we have documented. So we want to [20:03] take it to the next level where we want to see that if an if an action proposed [20:07] by agent is better than ours how do we make that as part of case library too. [20:13] Certain best practices you want to focus on are when you are creating your synthetic data, right? [20:18] Make sure that you're not you're doing it in a governed statistical manner, right? You don't want [20:25] too much skewess in your data. You don't want your data to look very different from your real data. [20:31] That creates issue in AI and in your modeling phases. Your classification will go all over [20:36] the places. Also also ensure once you have you know stream once you have synthetic data as you [20:44] are moving ahead right it is not causing a lot of drift right so this you know your synthetic data [20:50] aligning with real data they need to be closed for most of your real use cases um a part of this code [20:58] is is uh in this GitHub location you guys can take a look at it there are much more deeper technical [21:03] documents there um this is a work in progress right so if someone wants to talk more about [21:08] it. Uh this is my Gmail account. Thanks a lot, guys. I think I've exhausted my time. Thank you.