Chicory AI Uses Ontology Graphs to Stop LLM Hallucinations in DataOps

Unlocking Data Engineering with Generative AI With Chicory AI — Ai4

Ai4

Standalone language models fail in enterprise data operations because stale catalogs force them to guess, requiring multi-agent workflows grounded in unified code and runbook ontologies to deliver deterministic automation.

AI-assisted summary of Unlocking Data Engineering with Generative AI With Chicory AI — original source. Check the original for full context.

Source published:

Detailed notes

## Background Published on YouTube by Ai4 on August 24, 2025, this presentation features Saurabh Sarkar, Co-founder and CTO of Chicory AI. The session explores how enterprise data teams can overcome complex ETL pipelines and schema inconsistencies by using generative AI, agentic workflows, and AI-assisted metadata management to automate DataOps.

## TL;DR Saurabh Sarkar highlights that data teams spend the vast majority of their time firefighting pipeline failures due to broken lineage and fragmented tribal knowledge. Because standalone LLMs lack operational context and resort to guessing, Chicory AI provides a zero-orchestration multi-agent platform that aggregates code, documentation, and communication threads into a structured context layer. By grounding generative AI in a deterministic data warehouse ontology and knowledge graph, the system automates metadata cataloging, anomaly debugging, schema harmonization, and BI traceability. This structured approach replaces manual grunt work with confident, reproducible automation across modern data stacks.

## Atmosphere & style The talk has a crisp, pragmatic, and authoritative tech-conference feel geared directly toward overburdened data engineers and architects. Sarkar delivers a focused, problem-solving narrative that moves from the relatable frustrations of enterprise data firefighting to a structured, architectural solution. The tone remains practical and earnest throughout, emphasizing reliable engineering over generative AI hype.

## Key takeaways - [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) Enterprise data teams spend the majority of their time firefighting broken pipelines rather than innovating, largely due to fragmented documentation and tribal knowledge. - [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) Broken data lineage and ungrounded troubleshooting slow development cycles and bottleneck downstream machine learning initiatives. - [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) Standalone LLMs fail at data engineering tasks because passive, outdated data catalogs cause models to guess and hallucinate. - [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) Reliable data automation requires providing AI models with grounded, up-to-date schema definitions, code context, and operational business logic. - [03:02](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=182s) Chicory AI provides an out-of-the-box multi-agent framework tailored for DataOps that removes the need for manual agent orchestration and fine-tuning. - [03:02](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=182s) The platform captures and structures operational context from diverse sources, including code repos, data notebooks, documentation, and communication threads like emails. - [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) Core automated use cases include automated catalog generation, root-cause pipeline debugging, feature engineering, and cross-system data harmonization. - [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) A structured data warehouse ontology introduces determinism to agent workflows by decomposing complex tasks based on historical human runbooks. - [06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) Interactive knowledge graphs unify lakehouse platforms like Databricks with external documentation to ground generative AI in real structural relationships.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) The Reality of Modern Data Engineering: Firefighting and Grunt Work

Saurabh Sarkar, co-founder and CTO of Chicory AI, introduces his team's mission as a three-year-old startup tackling the persistent bottlenecks in data engineering through generative AI. While the promise of modern data stacks is substantial, the everyday reality inside enterprise data teams is dominated by repetitive, exhausting "grunt work." Instead of building innovative data products or driving analytics forward, data practitioners spend the overwhelming majority of their work hours "firefighting than innovating."

This operational drag stems from fragmented organizational context. Critical system knowledge remains trapped as tribal wisdom in the minds of individual engineers, while documentation is scattered unpredictably across disconnected tools. When pipelines inevitably break, the lack of end-to-end lineage turns debugging into an expensive and painfully slow troubleshooting exercise. Downstream machine learning teams suffer direct consequences, finding their development cycles stalled due to the unavailability or unreliability of high-quality data.

- **Enterprise Reality**: Data teams spend most of their time firefighting pipeline failures rather than innovating on new data features. - **Root Causes**: - Tribal knowledge locked in individual employees' heads. - Fragmented documentation scattered across disparate platforms. - Broken data lineage that makes troubleshooting costly and slow. - ML bottlenecks caused by inconsistent and low-quality data feeds.

---

[01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) Why Standalone LLMs Fail: The Need for Rich Context

When generative AI emerged, the initial assumption was that large language models would immediately solve these data pipeline headaches. However, real-world customer implementations quickly demonstrated that an "LLM alone isn't enough." In typical data warehouse environments, data catalogs are passive, stale, and out-of-date. Without explicit access to schema definitions, lineage maps, transformation code, and operational business logic, an AI model cannot reason accurately.

Without grounded and up-to-date context, an AI system is reduced to "just guessing"—a pitfall that affects human engineers just as much as machines. When deprived of necessary context, models hallucinate incorrect assumptions about tables and data flows. To unlock true automation, teams must move past naive prompting and address the core question: "What if AI actually understood your data?" and "What if AI knew that what your data actually meant?" Providing fresh, structured data context transforms AI from an unpredictable guesser into a system capable of delivering "automation with confidence."

```mermaid flowchart TD accTitle: The Context Gap in Data AI subgraph G1["Without Context (Failure Mode)"] A1["Outdated Catalogs & Missing Lineage"] --> B1["LLM Guessing / Hallucination"] B1 --> C1["Broken Pipelines & Costly Errors"] end

subgraph G2["With Grounded Context (Chicory AI)"] A2["Schemas, Code, Docs & Lineage"] --> B2["Structured Context Layer & Ontology"] B2 --> C2["Automation with Confidence"] end ```

---

[03:02](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=182s) Chicory AI's Multi-Agent Architecture and Data Context Layer

To bridge the context divide, Chicory AI built an agentic platform designed specifically for DataOps. Rather than requiring users to manually configure orchestrations, fine-tune models, or construct bespoke agent scaffolding, the platform provides a multi-agent framework out of the box. It automates tedious data operational tasks—such as extracting metadata, writing documentation, and mapping lineage—that previously consumed hours of human engineering time.

The platform unifies structured, semi-structured, and unstructured data by reading directly from existing source code repositories, databases, notebooks, and static documentation. Crucially, it also incorporates external communication channels—such as engineering emails and message threads where operational context frequently hides—and normalizes this knowledge into a single, structured context layer. This guarantees that downstream AI agents execute tasks with reliable, consistent, and accurate domain awareness.

- **Zero-Orchestration Multi-Agent System**: Eliminates manual agent tuning and complex pipeline setup out of the box. - **Universal Ingestion**: Pulls context from structured tables, semi-structured formats, code repositories, data notebooks, documentation, and operational emails. - **Grounded Context Layer**: Serves verified schema and business logic to downstream AI agents to eliminate hallucinations during automated maintenance.

---

[04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) Core DataOps Use Cases and Deterministic Workflows

Chicory AI addresses several acute enterprise data bottlenecks, starting with foundational data understanding and automated catalog generation. Beyond initial cataloging, the platform's multi-agent workflows automate pipeline optimization, fault debugging, feature engineering, exploratory data research, cross-system data harmonization and mapping, and end-to-end business intelligence (BI) traceability.

```mermaid flowchart LR accTitle: Chicory AI Core Architecture and Workflow subgraph Ingestion["Context Aggregation"] I1["Databricks / Warehouses"] I2["Notebooks & Code"] I3["Docs & Runbooks"] end

subgraph Core["Chicory AI Platform"] C1["Data Warehouse Ontology"] C2["Deterministic Agent Workflows"] end

subgraph Tasks["Automated DataOps"] T1["Catalog Generation"] T2["Pipeline Optimization"] T3["Data Harmonization"] T4["BI Traceability"] end

Ingestion --> Core Core --> Tasks ```

What distinguishes the platform is its construction of a unified "data warehouse ontology" that serves as the architectural backbone for all subsequent engineering tasks. Rather than allowing agents to execute open-ended, non-deterministic actions, the system brings "determination to the agentic workflows." It breaks large engineering challenges into discrete sub-problems and aligns execution with historical company runbooks and human troubleshooting patterns. By referencing how human data engineers previously resolved issues, the agents continuously improve, scale complex lineage mapping, and resolve intricate data analysis dependencies.

| Use Case Category | Operational Capability | Impact | |:--- |:--- |:--- | | **Catalog & Understanding** | Automated metadata extraction and catalog generation | Creates a living, real-time context layer across all assets | | **Pipelines & Debugging** | Automated optimization, anomaly detection, and root-cause debugging | Drastically cuts human time spent on firefighting | | **Harmonization & Mapping** | Cross-source schema alignment and automated transformation logic | Eliminates manual transformation bottlenecks | | **Traceability & Governance** | End-to-end BI lineage and deep feature exploration | Accelerates ML deployments and ensures compliance |

---

[06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) Knowledge Graph Grounding and Modern Data Stack Support

Demonstrating the platform's integration capabilities, Sarkar highlights environments like Databricks where operational documentation frequently lives outside the core lakehouse platform. Chicory AI bridges these silos by ingesting disparate catalog definitions and external documentation to construct an interactive knowledge graph. This graph serves as the deterministic grounding layer for generative AI, ensuring agents reason over real structural relationships rather than probabilistic guesses.

With an accurate knowledge graph in place, teams can scale automated documentation and maintenance without manual intervention while debugging and accelerating production ML pipelines. Chicory AI integrates across the modern data stack, supporting various protocols, storage layers, and infrastructure environments. Wrapping up his presentation with a call to visit Booth K33 and try the online application at `app.chicory.ai`, Sarkar emphasizes the core mission: "With Chicory, I think it's we help your AI not guess, actually understand."

## Notable quotes > [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) firefighting than innovating — Saurabh Sarkar

> [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) LLM alone isn't enough — Saurabh Sarkar

> [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) just guessing — Saurabh Sarkar

> [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) What if AI actually understood your data? — Saurabh Sarkar

> [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) What if AI knew that what your data actually meant? — Saurabh Sarkar

> [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) automation with confidence — Saurabh Sarkar

> [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) data warehouse ontology — Saurabh Sarkar

> [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) determination to the agentic workflows — Saurabh Sarkar

> [06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) With Chicory, I think it's we help your AI not guess, actually understand. — Saurabh Sarkar

## Mentioned Saurabh Sarkar, Chicory AI, Ai4, Databricks

## Video description Saurabh Sarkar, Cofounder & CTO, Chicory AI

Abstracts

Data engineering is evolving rapidly, but teams still face significant challenges—complex ETL pipelines, schema inconsistencies, and time-consuming data transformations. Generative AI is changing the game by automating these tasks, reducing manual effort, and accelerating data pipeline development.

This talk explores how LLM-powered agentic workflows can automate schema mapping, anomaly detection, feature engineering, and transformation logic, enabling faster and more efficient data processing. We will dive into real-world implementations of Generative AI in data engineering, showcasing how adaptive retrieval-augmented generation (aRAG) and AI-assisted metadata management enhance data quality, governance, and lineage tracking.

Whether you're a data engineer, architect, or AI practitioner, this talk will equip you with insights, best practices, and actionable steps to unlock the full potential of Generative AI in data engineering.

[Source](https://www.youtube.com/watch?v=Ot9uCz2IbHI)

中文简报

Chicory AI利用本体图遏制DataOps中的LLM幻觉 独立语言模型在企业数据运维中表现不佳，因为过时的数据目录迫使其进行猜测；这需要植根于统一代码与运行手册本体的多智能体工作流，以实现确定性自动化。 Saurabh Sarkar指出，由于数据血缘断裂和隐性知识碎片化，数据团队将绝大部分时间用于应对数据管道故障。鉴于独立LLM缺乏运维上下文且倾向于凭空猜测，Chicory AI提供了一个无需人工编排的多智能体平台，将代码、文档和沟通记录整合为结构化上下文层。通过将生成式AI锚定在确定性数据仓库本体与知识图谱中，该系统实现了元数据编目、异常调试、模式协调以及BI可追溯性的自动化。这种结构化方法以高置信度、可复现的自动化，取代了现代数据栈中繁琐的人工作业。 企业数据团队目前将绝大部分精力耗费在管道故障排查而非业务创新，根本原因在于系统隐性知识割裂、文档碎片化以及数据血缘断裂拖慢了下游 ML 项目。 (0:04) 独立 LLM 难以直接胜任数据工程任务，由于传统数据目录被动且容易过时，模型在缺乏实时 Schema、代码上下文及业务逻辑的情况下极易产生错误幻觉。 (1:22) Chicory AI 推出了开箱即用的 DataOps 多智能体框架，无需手动编排与微调，即可自动从代码库、数据 Notebook、文档乃至工程邮件中整合出统一的结构化上下文层。 (3:02) 平台借助构建数据仓库本体（Ontology）与知识图谱为 Agent 工作流引入确定性，结合历史运行手册将复杂任务分解，实现自动化目录生成、根因排查与跨系统数据调和。 (4:41)

Reading summary

Saurabh Sarkar, co-founder and CTO of Chicory AI, argues that generative AI can only resolve chronic data engineering bottlenecks if models are grounded in an operational context layer and deterministic ontologies rather than naive prompting. Context Fragmentation and the Firefighting Bottleneck (0:04) Modern enterprise data teams frequently spend the majority of their working hours firefighting pipeline breakdowns instead of building data products or advancing analytics. According to Sarkar, this operational overhead is driven by fragmented organizational knowledge: critical system architecture remains trapped as tribal knowledge within individual engineers' minds, while technical documentation is dispersed across disconnected platforms. Because end-to-end data lineage is typically absent, diagnosing pipeline failures becomes a slow, costly troubleshooting process. This operational drag cascades downstream, leaving machine learning teams stalled as they wait for dependable, high-quality data feeds. The Context Deficit in Standalone LLMs (1:22) Early assumptions that off-the-shelf large language models would independently resolve pipeline challenges failed in practice. Sarkar notes that enterprise data catalogs are often passive, static, and outdated, leaving models without real-time schema definitions, transformation code, or operational business logic. Deprived of grounded system context, language models are reduced to probabilistic guessing and hallucinating erroneous assumptions about tables and data flows. Sarkar asserts that achieving trustworthy data automation requires moving beyond prompt engineering to supply structured, up-to-date context reflecting what data actually represents. Multi-Source Ingestion and Unified Context Layers (3:02) To supply missing context, Chicory AI built a multi-agent framework intended to operate without manual orchestration or agent scaffolding. The architecture automates operational overhead, including metadata extraction, catalog documentation, and lineage mapping, which previously required hours of human engineering. The system unifies structured, semi-structured, and unstructured data by indexing code repositories, databases, notebooks, and runbooks. It also ingests external communication channels, such as engineering emails and message threads where troubleshooting history is often preserved, converting fragmented records into a unified context layer to prevent agent hallucinations. Deterministic Execution and Knowledge Graph Grounding (4:41) To prevent open-ended or non-deterministic agent errors, Chicory AI structures metadata into a data warehouse ontology and interactive knowledge graph. Demonstrating with platforms like Databricks—where documentation often sits outside lakehouse storage—the knowledge graph establishes structural relationships that enable agents to reason deterministically over system dependencies. The platform decomposes complex DataOps workflows into discrete tasks informed by historical runbooks and human troubleshooting patterns. This deterministic execution spans automated catalog generation, root-cause pipeline debugging, cross-system schema harmonization, and end-to-end business intelligence traceability. Sarkar contends that bridging the gap between passive catalogs and automated DataOps requires structuring scattered organizational knowledge into interactive ontologies, enabling AI agents to act deterministically rather than relying on ungrounded inference.

精炼阅读摘要

Chicory AI 联合创始人兼首席技术官 Saurabh Sarkar 指出，现代企业数据团队普遍陷入管道排障与繁琐运维的泥潭；单靠大语言模型无法解决问题，唯有通过构建覆盖代码、文档与业务逻辑的统一上下文层及知识图谱，结合确定性多智能体框架，才能实现可靠的数据工程自动化。 企业数据工程的运维困境与隐性知识瓶颈 (0:04) Saurabh Sarkar 指出，尽管现代数据技术栈发展迅速，但企业数据团队的日常工作仍被大量繁琐且重复的基础性运维所占据。工程师们将绝大部分工时耗费在处理管道故障的“救火”工作中，无法专注于高价值的数据产品研发与前沿分析。 这种低效现象的根源在于组织上下文的严重割裂。大量关键系统的运行逻辑仅作为“隐性知识”存在于个别资深工程师的脑海中，各类技术文档杂乱分散在彼此孤立的协作工具里。当数据管道发生故障时，端到端血缘关系的缺失导致排障过程极其耗时且成本高昂。这种不稳定性直接波及下游的机器学习团队，因高质量数据供给的延迟与质量缺陷而延缓了模型开发迭代。 独立大模型的局限与上下文断层危机 (1:22) 生成式 AI 兴起之初，行业普遍预期大语言模型能够迅速接管数据管道的运维难题，但实际客户落地证明，仅依靠单一模型并不足以解决工程痛点。Sarkar 分析称，在典型的数据仓库环境中，现有的数据目录往往是被动且陈旧的，既无法动态跟踪 Schema 变更，也缺乏对血缘图谱、数据转换代码和实际业务逻辑的实时感知。 在缺乏充分且最新上下文的支撑下，AI 模型与缺乏系统全貌的人类工程师一样，只能依赖概率“进行猜测”，进而对数据表结构与数据流产生严重的幻觉和误判。Sarkar 强调，必须让 AI 准确理解数据的真实语义，唯有依托经过验证的结构化上下文，才能消除不可控的模型推断，建立高确定性的自动化工程能力。 多智能体架构与全源上下文层的构建 (3:02) 为了消除上下文断层，Chicory AI 开发了一套专用于 DataOps 的开箱即用多智能体系统。该平台无需用户手动配置复杂的智能体编排流程、微调模型或搭建自定义脚手架，即可自动化执行元数据提取、文档生成和血缘映射等耗时任务。 在底层数据摄取方面，该架构直接打通了结构化、半结构化与非结构化资产的连接边界。系统能够自动读取现有的源代码仓库、数据库、数据分析 Notebook、静态技术文档，甚至连同工程电子邮件和消息会话中潜藏的操作记录一并整合。这些多源信息被集中归一化为一个结构化上下文层，从而确保下游执行任务的各个智能体均具备一致且准确的领域认知。 数仓本体与确定性工作流的核心落地场景 (4:41) 该平台的核心架构依托于一套统一的“数据仓库本体”，将其作为所有后续工程操作的执行基准。不同于开放式、具有随机性的智能体行为，系统为工作流注入了确定性机制：它将庞杂的工程挑战拆解为离散的子任务，并严格参考企业历史运维手册和人类专家的排障路径进行推演与执行，从而实现自我迭代与依赖解析。 在具体业务场景中，该平台覆盖了从基础数据理解到高阶治理的闭环链条。这包括实时目录与元数据自动生成、管道性能优化、异常检测与自动化根因调试；同时支持跨异构源的数据协调映射与模式对齐，并建立从底层仓库到商业智能（BI）报表的端到端血缘追踪，以加速机器学习特征研发并满足企业合规需求。 知识图谱锚定与现代数据技术栈协同 (6:35) 针对 Databricks 等典型湖仓体系中运维文档往往与核心计算环境相脱离的痛点，Chicory AI 通过汇聚各处离散的目录配置与外部说明，构建起具备交互能力的知识图谱。该图谱作为生成式 AI 的确定性事实底座，促使智能体依靠客观的实体结构关系进行推理，彻底摆脱概率性猜测。 Sarkar 表示，该架构目前已全面适配现代数据技术栈中的主流传输协议、存储引擎与云基础设施。通过将分散上下文沉淀为可计算的图谱层，企业能够在减少人工干预的前提下，加速生产级数据管道的运维效率，使 AI 真正从盲目推测迈向精确理解。 Sarkar 强调，Chicory AI 的核心目标是促使 AI 从“凭空猜测数据”转变为“真正理解数据语义”。然而，该模式在实际企业复杂异构环境中的全面普及，仍取决于非结构化通信内容清洗的精准度、专有历史运维手册的标准化程度，以及跨多云深层旧系统的兼容适应能力。

Detailed notes — zh

## Background 该内容由 Ai4 于 2025 年 8 月 24 日发布在 YouTube 平台。视频记录了 Chicory AI 联合创始人兼首席技术官 Saurabh Sarkar 的演讲，主要探讨如何利用基于大语言模型的智能体工作流、自适应检索增强生成（aRAG）及 AI 辅助元数据管理，自动化应对数据清洗、模式映射与血缘追踪等数据工程挑战。 ## TL;DR 演讲深入剖析了当前数据工程师陷入机械杂务与频繁排障的困境，指出单纯依赖大语言模型因缺乏上下文极易导致“幻觉”。为此，Chicory AI 打造了专为 DataOps 设计的多智能体架构，通过统一代码、文档及业务沟通记录来构建数据仓库本体与上下文层。系统通过拆解任务并学习专家运维经验（Runbooks），以知识图谱为基准实现确定性自动化，覆盖数据目录生成、流水线排障与端到端血缘追踪等核心场景。 ## Atmosphere & style 整场演讲节奏明快、直击行业痛点，洋溢着科技峰会现场浓厚的技术探讨与务实氛围。主讲人 Saurabh Sarkar 表达生动干练，以直面一线工程师“救火”苦力的共鸣开篇，随后层层递进地展开系统架构与解决方案。尽管受限于现场时间限制，演讲依然条理分明、逻辑严密，展现出对现代数据栈痛点的深刻洞察与强大自信。 ## Key takeaways - [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) 数据工程师普遍陷入繁重的机械杂务与日常救火状态，文档碎片化和知识孤岛严重阻碍了创新与机器学习团队的落地。 - [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 单纯依靠大语言模型（LLM）无法解决数据工程难题，缺乏准确的架构定义与数据血缘会导致 AI 产生严重幻觉。 - [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 传统数据目录普遍处于被动更新且内容滞后的状态，无法为生成式 AI 提供严谨的推理上下文支撑。 - [03:00](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=180s) Chicory AI 打造了基于多智能体（Multi-Agent）框架的 DataOps 平台，无需复杂微调即可自动化低效工程流程。 - [03:00](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=180s) 系统通过整合结构化/非结构化数据、业务代码、Notebook 及往来邮件，建立统一的上下文层以提升 AI 执行精度。 - [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) 平台构建数据仓库本体（Data Warehouse Ontology），作为多智能体执行任务的核心基石与语义标准。 - [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) 引入确定性执行机制，通过拆解复杂问题并吸收人类专家的 Runbooks 和排障经验，使智能体能够持续演进。 - [06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) 系统在接入企业数据后首先自动构建知识图谱，通过结构化上下文锚定生成式 AI，彻底杜绝盲目猜测与幻觉风险。 - [06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) 该方案无缝兼容现代数据栈生态，能够深度集成 Databricks 数据目录等主流基础设施与外部工具。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) 企业数据工程的现状：疲于“救火”与杂务缠身 演讲在热烈的现场氛围中开场。Chicory AI 联合创始人兼首席技术官 Saurabh Sarkar 首先登台向观众问好，并介绍了 Chicory AI 这家成立三年的初创公司。团队的核心使命是通过生成式 AI（Generative AI）自动化数据工程中最繁重、最机械重复的环节，从而彻底释放数据工程的真正潜力。 Saurabh Sarkar 指出，当前大多数企业内部的数据工程现状几乎等同于“繁重的苦力活（grunt work）”。工程师们每天绝大部分的工作时间并没有用来推动业务创新，而是陷入了无休止的“救火（firefighting）”状态中。导致这一困境的核心痛点在于：大量的关键数据逻辑作为“隐性经验/部落知识（tribal knowledge）”仅留存在个别员工的大脑里，相关文档零散分布在不同的工具与系统之中；同时，数据血缘（lineage）频繁断裂，导致排查系统故障的成本极其高昂，下游的机器学习（ML）团队也因数据不可用或质量不可靠而步履维艰。 - Chicory AI 是一家专注于利用生成式 AI 解决数据工程痛点的初创企业。 - 当前数据工程师的日常工作主要被应急排障与繁琐杂务占据，缺乏创新空间。 - 文档碎片化、知识孤岛、血缘断裂以及不可靠的数据质量，构成了阻碍企业 ML 团队推进的主要瓶颈。 --- ### [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 为何单纯依靠 LLM 行不通：缺乏上下文导致 AI 产生幻觉 业界普遍期望 AI 与数据的结合能够迎刃而解上述问题，但 Chicory 团队在与大量客户的实际合作中发现了一个核心痛点——“仅仅依靠大语言模型（LLM alone）根本不够”。 当前 AI 与数据技术栈之间的脱节十分严重。以数据仓库生态为例，现有的数据目录通常处于被动记录且内容陈旧过时的状态。如果缺乏精准的架构定义（schema）、完整的数据血缘、底层代码以及转换逻辑（transformation logic），AI 根本无法展开严谨的推理分析。Saurabh Sarkar 强调，在缺乏最新且正确的信息支撑时，AI 实际上“只是在瞎猜（AI is just guessing）”。这种情况对人类数据工程师同样适用：如果缺乏上下文，人类会无从下手，而在 AI 世界里，盲目猜测的直接后果就是产生严重的“幻觉”。因此核心问题演变为：“如果 AI 能真正理解你的数据，甚至明白这些数据背后的实际业务含义，将会怎样？” - 传统数据仓库的数据目录普遍存在被动更新、严重滞后且与实际业务脱节的问题。 - 缺少 schema、血缘与转换逻辑时，大语言模型无法进行逻辑推理，只会导致幻觉生成。 - 赋予 AI 鲜活、结构化的全局上下文，是实现高置信度自动化（automation with confidence）的先决条件。 --- ### [03:00](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=180s) Chicory AI 的多智能体架构：专为 DataOps 打造的上下文引擎 为了从根本上解决数据理解与上下文缺失的问题，Chicory 推出了专为 DataOps 打造的智能体平台。该平台采用多智能体框架（Multi-Agent Framework），能够自动完成元数据提取、文档生成与数据血缘梳理。它开箱即用，无需用户进行复杂的编排或微调，即可快速将原本需要人工耗费数小时的低效工作流实现自动化。 ```mermaid flowchart TD accTitle: Chicory AI 自动化 DataOps 架构流程图 subgraph DataSources["多模态上下文接入"] A["结构化/半结构化/非结构化数据"] B["业务代码与 Notebooks"] C["分散文档与外部沟通邮件"] end subgraph ChicoryEngine["Chicory 统一上下文与智能体层"] D["数据仓库本体与统一上下文层"] E["多智能体框架 (Multi-Agent)"] end subgraph Operations["DataOps 自动化执行"] F["元数据提取与文档生成"] G["复杂数据血缘与架构映射"] H["高置信度自动化与精准排障"] end A --> D B --> D C --> D D --> E E --> F E --> G E --> H ``` 在典型的数据仓库环境中，数据往往跨越结构化、半结构化和非结构化等多种形态。Chicory 的核心能力在于将这些分散的信息统一沉淀在单一的上下文层中。它能直接连接企业的现有代码库、数据资产、Jupyter Notebook 以及各类技术文档，甚至能够吸纳包含关键背景信息的往来邮件。通过将这些非底层数仓内的业务信息全部纳入体系，Chicory 为每个数据任务输出高度结构化的上下文，从根本上消除了工程师的杂务负担，确保 AI 在执行数据任务时具备极高的可靠性、一致性与精准度。 - 平台专为数据运维（DataOps）场景定制，采用 Multi-Agent 架构。 - 无缝整合代码、文档、Notebook 及外部沟通邮件，建立统一上下文层。 - 无需复杂的模型编排或微调，即可直接替换耗时冗长的人工操作。 --- ### [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) 核心应用场景与执行确定性：从数据本体到运维自动化 Chicory 平台能够伴随企业业务需求持续演进。目前平台已成功落地的核心场景包括：底层数据理解与自动化数据目录生成（作为支撑上层 AI 任务的基础骨干）、数据流水线（Pipeline）优化与排障调试、特征工程、深度探索式研究，以及长期困扰大型企业的数据调和与架构映射（Data Harmonization & Mapping），最终实现端到端的商业智能（BI）可追溯性。 Saurabh Sarkar 详细阐述了 Chicory 的独特优势：平台能为所有的工程任务构建出最优的“数据仓库本体（Data Warehouse Ontology）”，并作为 AI 智能体执行任务的核心基石。为了确保执行过程具备“确定性（determination）”，Chicory 会将复杂问题拆解为多个子问题，同时主动学习并提取企业内部人类专家沉淀的操作手册（Runbooks）与历史排障经验。通过学习人类解决特定问题的步骤，AI 能够不断参考、优化并持续演进，进而在超大规模场景下稳定解决复杂的数据血缘关联与深度数据分析难题。 - **核心应用场景**：数据目录生成、流水线优化/调试、特征工程、深度探索、数据调和映射与 BI 溯源。 - **构建数仓本体**：为所有数据工程任务建立统一的语义与逻辑本体层。 - **引入确定性机制**：拆解复杂任务，复用企业现有的 Runbooks 与专家操作经验，让 AI 智能体持续学习进化。 --- ### [06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) 系统架构展示与现场收尾 由于现场演讲时间所剩无几（仅剩约 2 分钟），Saurabh Sarkar 快速为观众梳理了技术架构与核心逻辑。虽然未能展开完整的动态演示，但他通过技术截图展示了 Chicory 如何与 Databricks 数据目录以及外部文档系统实现无缝对接。 Chicory 接入企业上下文后的第一步是自动构建知识图谱（Knowledge Graph）。这一图谱构建了对企业数据的全景理解，通过真实、结构化的上下文将生成式 AI 牢牢锚定（grounding），彻底根除幻觉风险。基于这套图谱，企业无需投入人工即可大规模扩展文档编写及各类 DataOps 自动化任务，显著加速机器学习模型的开发与部署调试周期。Saurabh Sarkar 总结道，Chicory 专为现代数据栈（Modern Data Stack）设计，支持多种底层协议、基础设施与外部集成。演讲最后，他强调 Chicory 的核心使命就是“让 AI 不再凭空瞎猜，而是真正理解你的数据”，并热情邀请参会者前往 K33 展位交流，或访问官方平台 `app.chicory.ai` 体验产品。 - 系统首要任务是基于企业上下文自动构建结构化知识图谱，为 GenAI 提供锚定基准。 - 架构兼容现代数据栈生态，支持主流协议、底层基础设施与 Databricks 等数据目录工具。 - 听众可前往 Ai4 展会的 K33 展位现场交流，或通过 `app.chicory.ai` 体验产品演示。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) 当前大多数企业内部的数据工程现状几乎等同于“繁重的苦力活”，工程师们每天绝大部分的工作时间并没有用来推动业务创新，而是陷入了无休止的“救火”状态中。 — Saurabh Sarkar > [00:04](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=4s) 大量的关键数据逻辑作为“隐性经验/部落知识”仅留存在个别员工的大脑里，相关文档零散分布在不同的工具与系统之中。 — Saurabh Sarkar > [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 仅仅依靠大语言模型（LLM alone）根本不够。 — Saurabh Sarkar > [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 在缺乏最新且正确的信息支撑时，AI 实际上“只是在瞎猜”。 — Saurabh Sarkar > [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 如果 AI 能真正理解你的数据，甚至明白这些数据背后的实际业务含义，将会怎样？ — Saurabh Sarkar > [01:22](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=82s) 缺少 schema、血缘与转换逻辑时，大语言模型无法进行逻辑推理，只会导致幻觉生成。 — Saurabh Sarkar > [03:00](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=180s) 将原本需要人工耗费数小时的低效工作流实现自动化。 — Saurabh Sarkar > [04:41](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=281s) 平台能为所有的工程任务构建出最优的“数据仓库本体”，并作为 AI 智能体执行任务的核心基石。 — Saurabh Sarkar > [06:35](https://www.youtube.com/watch?v=Ot9uCz2IbHI&t=395s) 让 AI 不再凭空瞎猜，而是真正理解你的数据。 — Saurabh Sarkar ## Mentioned Saurabh Sarkar, Chicory AI, Ai4, Databricks, Jupyter Notebook, Modern Data Stack, Multi-Agent Framework, DataOps ## Video description Saurabh Sarkar, Cofounder & CTO, Chicory AI Abstracts Data engineering is evolving rapidly, but teams still face significant challenges—complex ETL pipelines, schema inconsistencies, and time-consuming data transformations. Generative AI is changing the game by automating these tasks, reducing manual effort, and accelerating data pipeline development. This talk explores how LLM-powered agentic workflows can automate schema mapping, anomaly detection, feature engineering, and transformation logic, enabling faster and more efficient data processing. We will dive into real-world implementations of Generative AI in data engineering, showcasing how adaptive retrieval-augmented generation (aRAG) and AI-assisted metadata management enhance data quality, governance, and lineage tracking. Whether you're a data engineer, architect, or AI practitioner, this talk will equip you with insights, best practices, and actionable steps to unlock the full potential of Generative AI in data engineering. [Source](https://www.youtube.com/watch?v=Ot9uCz2IbHI)

Source quotes

[0:04] Saurabh Sarkar: 当前大多数企业内部的数据工程现状几乎等同于“繁重的苦力活”，工程师们每天绝大部分的工作时间并没有用来推动业务创新，而是陷入了无休止的“救火”状态中。 [0:04] Saurabh Sarkar: 大量的关键数据逻辑作为“隐性经验/部落知识”仅留存在个别员工的大脑里，相关文档零散分布在不同的工具与系统之中。 [1:22] Saurabh Sarkar: 仅仅依靠大语言模型（LLM alone）根本不够。 [1:22] Saurabh Sarkar: 在缺乏最新且正确的信息支撑时，AI 实际上“只是在瞎猜”。 [1:22] Saurabh Sarkar: 如果 AI 能真正理解你的数据，甚至明白这些数据背后的实际业务含义，将会怎样？ [1:22] Saurabh Sarkar: 缺少 schema、血缘与转换逻辑时，大语言模型无法进行逻辑推理，只会导致幻觉生成。 [3:00] Saurabh Sarkar: 将原本需要人工耗费数小时的低效工作流实现自动化。 [4:41] Saurabh Sarkar: 平台能为所有的工程任务构建出最优的“数据仓库本体”，并作为 AI 智能体执行任务的核心基石。 [6:35] Saurabh Sarkar: 让 AI 不再凭空瞎猜，而是真正理解你的数据。

Source quotes

[0:04] Saurabh Sarkar: firefighting than innovating [1:22] Saurabh Sarkar: LLM alone isn't enough [1:22] Saurabh Sarkar: just guessing [1:22] Saurabh Sarkar: What if AI actually understood your data? [1:22] Saurabh Sarkar: What if AI knew that what your data actually meant? [1:22] Saurabh Sarkar: automation with confidence [4:41] Saurabh Sarkar: data warehouse ontology [4:41] Saurabh Sarkar: determination to the agentic workflows [6:35] Saurabh Sarkar: With Chicory, I think it's we help your AI not guess, actually understand.

Original transcript

[0:04] Glad to be here and [0:06] like woo like we have good audience [0:08] here. So let's get started. So I am [0:11] solubar and co-founder Chikree and we [0:14] are a three-year-old startup who is [0:16] trying to solve data engineering using [0:18] generative AI. And today I want to share [0:20] like uh how we explore generative AI to [0:24] unlock data the true potential of data [0:27] engineering by automating the hardest [0:29] and the most repetitive [0:32] parts of the job. [0:36] So if you think like in the enterprises [0:39] today and what we have observed is the [0:41] state of the data engineering today is [0:43] mostly equal to grunt work. And what I [0:46] mean is like the reality is that we [0:49] spend most of our job and time in [0:53] firefighting [0:54] than innovating. And why this happen? [0:57] Because the pain points actually truly [1:00] lies in the tribal knowledge of uh in [1:03] the heads of the people the scattered [1:06] documentation across tools broken [1:08] lineage which actually uh leads to very [1:11] costly troubleshooting and ML teams they [1:14] are getting slowed because of [1:16] unavailability or unreliability of [1:18] quality data. [1:22] Now we thought that AI will help. Now AI [1:25] and data will be able to help and solve [1:28] this problem overall. But something [1:31] which we learned working with a customer [1:33] that LLM alone isn't enough. [1:37] What's broken in the AI and data stack [1:39] is like for example in the data data [1:41] warehouse ecosystem if you don't have [1:43] cataloges updated and which are usually [1:46] passive and out ofdated and you have [1:50] kind of the AI cannot reason well [1:53] without proper schema lineage code or [1:56] transformation logic. What we have found [1:59] is like without all this uh correct [2:02] information and upto-date uh context [2:06] uh AI is just guessing and I would say [2:09] actually that applies to human as well [2:11] like if you don't provide a correct [2:12] contest to any data engineer today they [2:16] in the AI world they might hallucinate [2:18] right because they are just guessing and [2:20] not actually understanding the data and [2:23] that's where the question comes from [2:24] that what if AI actually understood your [2:27] data [2:29] What if AI knew that what your data [2:31] actually meant? [2:33] So [2:35] what we did is like and learned over the [2:39] period of time uh is like in the data [2:42] infrastructure world not able to provide [2:46] the right schema lineage or the fresh [2:49] context [2:51] actually makes your AI suffer. But if [2:54] you can bring the AI with the right [2:56] context, it actually provides with [2:58] automation with confidence [3:00] and that's where [3:02] Chry comes into the picture. So we are [3:05] the first agentic platform which is purp [3:08] purpose-built for data ops and what I [3:10] mean is like think of all the complex [3:13] challenges challenges which you face in [3:15] your data operations that's where the [3:17] platform can actually help solve. It [3:20] automates metadata extraction, [3:22] documentations and lineage uh using [3:25] multi- aent framework and you know no [3:27] orchestration or tuning needed like you [3:29] can bring your model versus not but what [3:32] it gives you out of the box is able to [3:34] automate the workflows which would have [3:38] required you know uh hours of human time [3:42] which was wasteless wasteful [3:45] u and let's talk about kind of an data [3:48] warehous setup Right? If you think there [3:50] are structured data, unstructured data, [3:52] semi-structured data to be able to bring [3:55] all that context under one, you know, [3:58] one layer, that's where Chikree helps [4:00] you. It kind of it works directly with [4:02] your existing code, data, notebooks, [4:05] documentation. Think of, you know, all [4:10] emails where you have the uh like the [4:13] relevant information which is not a part [4:16] of your data infrastructure. But that [4:18] kind of adds values also like it helps [4:21] you remove that grunt work which I [4:23] mentioned and delivers structured [4:25] context to every task which enables AI [4:28] to be able to provide you the most [4:30] reliable consistent and you know the [4:34] most accurate uh and able to perform the [4:37] job for you. [4:41] Let's talk about the specific use cases. [4:43] So our platform is built uh evolve over [4:47] time so that we can evolve and we keep [4:50] going and evolve with uh kind of the [4:51] needs. uh what Chry is able to automate [4:54] uh and the you know some of the use [4:56] cases which we have uh solved today is [4:59] data understanding [5:01] um and you know if I have to call it [5:03] like as a catalog generation as well and [5:05] which I mentioned like the understanding [5:07] context layer the base basic kind of the [5:09] backbone of all um AI to be able to uh [5:13] do its job better. uh think of pipeline [5:15] optimizations or debugging, feature [5:18] engineering or deep research [5:19] exploration, data harmonization or [5:21] mapping which actually is a very you [5:24] know it's a big pain for a lot of [5:26] enterprise companies and all comes down [5:29] to the business intelligence [5:30] traceability. So with Chikree uh we are [5:33] able to kind of automate all these task [5:35] with you know accuracy and what makes uh [5:39] Chikree different is [5:42] we like you know we uh help uh create [5:46] the best data warehouse ontology for all [5:50] engineering tasks and that kind of you [5:51] know becomes the uh backbone for able to [5:54] perform all the other um AI you know [5:56] agentic tasks there. We brings [5:58] determination to the agentic workflows [6:00] by breaking the problem into small [6:02] problems and also picking up the [6:05] experiences or runbooks for the [6:08] companies and able to kind of execute on [6:10] that because that's where the AI learns [6:12] from that how human has kind of already [6:15] uh able to uh solve a problem using uh [6:20] certain steps. So it kind of refers and [6:22] uh improves and evolves over time. which [6:24] solves complex lineage and relationship [6:26] and data analysis problem at scale as [6:28] well. So that's how kind of chry enables [6:31] the automation for the AI um agents. So [6:35] now I'll just quickly go through a demo [6:38] and the goal of the demo is kind of [6:40] trying to show you that when you have [6:42] okay so I think we just have two minutes [6:45] left. So I'll I'll try to make it very [6:47] quick. So what I wanted to show here is [6:49] like you know data uh so we have a setup [6:52] where you can show uh have databicks uh [6:56] you know uh cataloges with documentation [7:00] which is kind of outside of databicks [7:02] and that's where the beauty of it like [7:03] we pro we bring all the data together [7:05] and able to provide you u a context [7:09] layer which powers and empowers the AI [7:11] agent built on top of our platform. [7:14] So I'll actually skip through the demo [7:18] so that we have time. Uh but I can kind [7:21] of show you some snippets here which can [7:23] help you understand how Chry works. So [7:25] the first thing you do is like you bring [7:27] all the like you integrate with your uh [7:29] context to chry and it helps create a [7:31] knowledge graph for you and that [7:33] knowledge graphs u is kind of you know [7:35] what provides the data understanding and [7:39] uh and why it matters is because this [7:41] real and structured context is what [7:43] ground geni to help it avoid [7:45] hallucination. So now it kind of [7:47] understands your data real well. you can [7:50] scale uh in this case it's documentation [7:52] but you can scale any kind of automation [7:54] uh without any manual effort because our [7:56] agents kind of uh are on top of it. Uh [8:00] and it can of course accelerate your and [8:02] uh de debug your ML deployment and [8:04] development cycle. Uh that said we are [8:08] built for the you know modern data [8:10] stack. We support uh you know different [8:12] layers. I can give you a very quick tech [8:15] architecture. So we kind of support [8:17] different protocols uh infrastructures [8:19] and also integrations. Uh and that would [8:24] be my time. So with Chikree I think it's [8:27] we help your AI not guess actually [8:29] understands. So uh please come say hi to [8:33] us at booth K33 and the demo which I [8:36] couldn't show but you can go and check [8:38] it out at app.chickree.ai. [8:40] Thank you everyone.