Databricks Unveils Genie ZeroOps to Repair Broken Data Pipelines

Unlocking agentic data engineering with Lakeflow + Genie — Databricks

Databricks

External coding agents fail at data operations because verifying fixes requires production data access, forcing remediation models directly into the governed data plane to prevent silent corruption.

AI-assisted summary of Unlocking agentic data engineering with Lakeflow + Genie — original source. Check the original for full context.

Source published:

Key ideas

  • Within three months of release, 60% of all Lakeflow pipelines are already being generated directly by Genie Code, accelerating the volume of data pipelines organizations must manage. (2:21)
  • Databricks introduced Zerobus Ingest, a fully managed service that is wire-compatible with the Apache Kafka protocol, eliminating broker operations and streaming up to 12 GB/s directly into the lakehouse. (5:36)
  • Lakeflow Designer is now generally available as a no-code visual preparation tool that natively compiles workflows into open Apache Spark declarative pipelines rather than proprietary formats. (4:49)
  • Databricks unveiled Genie ZeroOps, an autonomous background operational agent running inside the data plane that detects pipeline anomalies, identifies root causes via Unity Catalog lineage, and validates code fixes on isolated shallow clones. (12:32)

Detailed notes

## Background Published on YouTube by Databricks on June 23, 2026, this keynote session features Bilal Aslam, Sr. Director of Product Management, presenting at the 2026 Data + AI Summit. The talk addresses the mounting complexity of enterprise data engineering and details how Databricks' Lakeflow and the newly introduced Genie ZeroOps autonomous agent simplify pipeline construction and operations.

## TL;DR Bilal Aslam examines the grueling 'messy middle' of data engineering, warning that the unchecked explosion of AI coding agents risks overwhelming brittle, legacy data infrastructure. To counter this, Databricks unifies ingestion, transformation, and orchestration on an open foundation through Lakeflow—spanning Spark declarative pipelines, Lakeflow Designer, Zerobus Ingest, and serverless Lakeflow Jobs. Addressing the operational maintenance bottleneck, Aslam introduces Genie ZeroOps, an autonomous agent living natively inside the Databricks data plane. Driven by Unity Catalog lineage and zero-copy shallow clones, Genie ZeroOps autonomously triages pipeline failures, tests candidate code fixes against isolated production data, and enforces governance policies to eliminate data downtime.

## Atmosphere & style The presentation blends energetic keynote showmanship with an empathetic, candid peer-to-peer conversation, opening playfully with an inside joke about Bilal's signature purple jacket before validating the genuine burnout of overworked data engineers. Aslam builds narrative tension by contrasting the deterministic simplicity of software engineering against the insidious, silent failures endemic to data pipelines. The mood shifts from realistic wariness to triumphant relief during the live demonstration, where chaotic operational fire drills are replaced by a calm, inbox-style interface that handles end-to-end incident triage with effortless precision.

## Bulletin - [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) Within three months of release, 60% of all Lakeflow pipelines are already being generated directly by Genie Code, accelerating the volume of data pipelines organizations must manage. - [05:36](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=336s) Databricks introduced Zerobus Ingest, a fully managed service that is wire-compatible with the Apache Kafka protocol, eliminating broker operations and streaming up to 12 GB/s directly into the lakehouse. - [04:49](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=289s) Lakeflow Designer is now generally available as a no-code visual preparation tool that natively compiles workflows into open Apache Spark declarative pipelines rather than proprietary formats. - [12:32](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=752s) Databricks unveiled Genie ZeroOps, an autonomous background operational agent running inside the data plane that detects pipeline anomalies, identifies root causes via Unity Catalog lineage, and validates code fixes on isolated shallow clones.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=0s) Modern AI agents and operational workflows require high-quality data, but enterprise source systems remain fragmented across legacy databases, SaaS platforms, and event streams. - [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) The rapid adoption of AI coding agents increases the risk of operational overload by generating more pipelines on top of already fragile infrastructure. - [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) Databricks is anchoring Lakeflow on open frameworks, formats, and APIs to avoid proprietary vendor lock-in across ingestion, transformation, and orchestration. - [03:37](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=217s) Apache Spark declarative pipelines unify Python and SQL workflows while integrating an open real-time mode engine for millisecond-level streaming. - [04:49](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=289s) Lakeflow Designer eliminates shadow IT by compiling visual analyst workflows directly into standard, version-controlled Spark declarative pipelines. - [05:36](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=336s) Lakeflow Connect features over 100 enterprise connectors that execute as standard Spark declarative pipelines rather than proprietary runtimes. - [06:18](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=378s) Zerobus Ingest allows engineering teams to point existing Kafka producers directly to the lakehouse, avoiding Kafka broker management and resolving small-file issues. - [06:57](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=417s) Lakeflow Jobs modernizes orchestration with serverless execution, pure Python DAG support, and over 50 open-source operators to trigger external systems like Snowflake. - [06:57](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=417s) Spark declarative pipelines process more than 200 trillion rows daily, and Lakeflow Jobs executes roughly 1.7 billion monthly job runs globally. - [08:48](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=528s) Unlike software engineering where failures produce loud errors, data engineering failures are often silent and permanent, corrupting downstream analytical and operational systems. - [10:39](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=639s) General-purpose coding agents cannot safely operate outside the data plane because testing data fixes requires direct access to sensitive production data and end-to-end lineage. - [12:32](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=752s) Genie ZeroOps operates directly in the data plane across a five-phase loop: detection via ML models, lineage assessment, multi-agent root cause analysis, code remediation, and validation on shallow clones. - [14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) Genie ZeroOps correlates related batch job alerts into unified incidents and ranks them based on downstream business criticality using Unity Catalog lineage graphs. - [14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) To verify remediations without production risk, Genie ZeroOps executes proposed fixes on zero-copy shallow clones of production tables before generating pull requests. - [14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) Beyond pipeline breaks, Genie ZeroOps continuously scans datasets for compliance risks, such as exposed PII, and automatically provisions Unity Catalog governance policies to resolve them.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=0s) The Purple Jacket and the “Messy Middle” of Data Engineering

Bilal Aslam takes the stage to enthusiastic applause, kicking things off with a sartorial callback to prior conferences. Two years ago, he wore a distinct purple jacket while talking about Lakeflow, decided against wearing it the following year, and heard unanimous feedback from attendees demanding its return. After thanking earlier speakers—including Mukesh, Arsalan, Ali Ghodsi, and Kenan—for hyping up the future of artificial intelligence and autonomous agents, Bilal acknowledges the anxiety brewing among data practitioners in the audience. Everyone is talking about agents, apps, and real-time operations, but every single one of those systems relies strictly on high-quality data. For data engineers, that means their jobs suddenly became significantly harder.

The fundamental tension lies between where teams want to go and where they must begin. The business destination on the right is straightforward: autonomous agents, automated workflows, and operational intelligence. The starting point on the left is chaotic: raw operational sources scattered across Kafka, Salesforce, NetSuite, and legacy on-premises databases like Oracle and Microsoft SQL Server. The space between them is what Bilal calls the “messy middle”—the grueling, unglamorous data engineering work of ingesting, cleaning, and transforming that raw data into consumable insights. Over decades, organizations have stitched together brittle architectures crammed with dozens of disparate vendor logos, tools, and fragile point-to-point connections. While teams have made peace with this sprawl under the rationalization that “at least it works,” the reality is fraught with severe compromises. Practitioners cannot achieve uniform version control—they can manage their Apache Spark code in Git, but their underlying Kafka infrastructure remains decoupled. Broad governance is virtually unattainable because data reads and writes traverse dozens of disconnected endpoints, and cost-effective scaling fails because isolated layers of the stack scale unpredictably.

Bilal delivers a blunt assessment to the data engineers in the room: “I'm here to tell you that your job and my job as a data engineer is about to get quite a bit harder.”

- **Fragmented source systems:** Production data remains siloed across modern SaaS platforms (Salesforce, NetSuite), event streams (Kafka), and legacy relational on-premises databases (Oracle, SQL Server). - **Core architectural shortcomings:** Disjointed legacy stacks lack universal version control, make enterprise-wide data governance nearly impossible, and suffer from uneven scaling bottlenecks.

---

[02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) The Rise of Coding Agents and Lakeflow's Open Foundation

Bilal points to Genie Code, Databricks’ purpose-built coding agent tailored specifically for data engineering, data science, and machine learning. While the tool delivers immense productivity, it also surfaces an alarming operational reality. In just three months since its release, an astonishing 60% of all Lakeflow pipelines are already being generated directly by Genie Code. Bilal admits that while he loves agentic tools, the sheer velocity at which agents can spin up pipelines poses an existential threat to already fragile data infrastructure: “Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure.” Without architectural intervention, engineering teams will simply be swamped.

Rather than piling new tooling onto existing complexity, Databricks aims to simplify the stack by grounding it entirely in open formats, open frameworks, and open-source tools. Bilal asserts that Lakeflow provides this exact unified foundation, offering native interoperability across hundreds of partner ecosystems and lakehouse tools so teams can build resilient pipelines without vendor lock-in.

```mermaid flowchart LR accTitle: Lakeflow Unified Open Architecture accDescr: Diagram illustrating how Lakeflow replaces fragmented ingestion, transformation, and orchestration tools with unified open frameworks.

subgraph Legacy["Legacy Fragmented Stack"] direction TB L1["SaaS / Kafka / CDC"] --> L2["Custom Code / dbt / Flink"] --> L3["Unmanaged Airflow"] end

subgraph Lakeflow["Lakeflow Unified Open Foundation"] direction TB K1["Lakeflow Connect & Zerobus"] --> K2["Spark Declarative Pipelines\n(Batch, SQL/Python, Real-Time Mode)"] --> K3["Serverless Lakeflow Jobs"] end

Legacy -.->|"Consolidates into"| Lakeflow ```

---

[03:37](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=217s) Unifying ETL: Spark Declarative Pipelines and Real-Time Streaming

Diving into transformation and ETL—the layer where engineering teams sink the vast majority of their time and budget—Bilal tackles the fragmentation between legacy Spark scripts, custom Amazon EMR distributions, and standalone SQL engines. Lakeflow replaces these disparate engines with Apache Spark declarative pipelines. Because these pipelines are open and declarative, developers merely specify the desired end state, and the engine handles execution. This structure also makes them uniquely suited for AI agents like Genie to generate, optimize, and maintain without human friction. Furthermore, declarative pipelines run anywhere, from production clusters straight down to a local laptop.

Spark declarative pipelines bridge two historical divides in data engineering: language fragmentation and latency differences. Where teams previously had to bifurcate workloads—relying on dbt for SQL workflows and raw Spark for Python—declarative pipelines unify SQL and Python in one environment. Simultaneously, teams that adopted Apache Flink specifically to achieve millisecond stream processing can now stay on Spark. Databricks has integrated its open-source "real-time mode" engine directly into Spark declarative pipelines, delivering ultra-low-latency, millisecond-level streaming on an open engine.

Bilal highlights the core value of this unification: “So you can get millisecond streaming on an open framework that runs anywhere.”

---

[04:49](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=289s) Eliminating Shadow IT with Lakeflow Designer

Beyond formal data engineering teams, enterprises routinely struggle with widespread shadow IT. Bilal notes a common organizational imbalance: a company might employ 20 centralized data engineers, yet host hundreds of business analysts independently assembling ad-hoc workflows using visual drag-and-drop tools. These pipelines live on personal laptops, rely on proprietary visual formats, and inevitably break or become completely unmaintainable the moment their creator leaves the company.

To solve this divide without alienating non-technical users, Bilal announces the General Availability (GA) of Lakeflow Designer, a no-code data preparation environment powered natively by Genie. What sets Lakeflow Designer apart from traditional proprietary visual tools is its compilation target. Instead of locking business logic inside proprietary metadata formats, Lakeflow Designer automatically compiles visual workflows into standard Apache Spark declarative pipelines. Analysts obtain the visual ease they require, while data engineers inherit clean, open, version-controlled code that runs anywhere across the enterprise.

Bilal sums up the architectural philosophy: “And by the way, one super cool thing is that Lake Flow Designer doesn't build any proprietary pipelines. It just builds Spark declarative pipelines under the hood.”

---

[05:36](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=336s) Ingestion at Scale: Lakeflow Connect and Zerobus Ingest

Addressing the ingestion boundary, Bilal critiques traditional third-party SaaS ingestion platforms. While they advertise hundreds of pre-built connectors, they typically drop records into proprietary storage formats. Because these systems lack standard interfaces, AI coding agents cannot inspect or manage them, leaving pipelines unobservable, impossible to automate, and detached from unified governance. In response, Lakeflow Connect has expanded its library to more than 100 native connectors, supported by an open-source community ecosystem where engineers can build custom connectors. Crucially, Lakeflow Connect does not introduce custom ingestion runtimes; it executes as standard Spark declarative pipelines under the hood.

For real-time telemetry, data engineers have long depended on Apache Kafka as a high-volume ingestion buffer, despite universally disliking the operational overhead of running Kafka brokers. Databricks introduces Zerobus Ingest, a fully managed service that is 100% wire-compatible with the Kafka protocol. Teams can redirect existing Kafka producer code directly to a Zerobus endpoint without modifying upstream applications. Zerobus Ingest streams data straight into the lakehouse in open formats, eliminating the classic "small files problem" by ingesting millions of small records seamlessly at throughputs reaching 12 gigabytes per second.

Bilal emphasizes how clean this transition is for engineering teams: “So, you can take your Kafka producing code, point it at Zero Bus Ingest, and you can land data into the lakehouse in open formats, ready for AI.”

- **Lakeflow Connect scope:** Over 100 enterprise connectors available, open-source connector tooling for internal engineering teams, and native execution via Spark declarative pipelines. - **Zerobus Ingest capabilities:** Full Kafka API wire compatibility, direct ingestion into lakehouse storage without broker management, eliminates small-file bloat, and handles sustained throughput of 12 GB/sec.

---

[06:57](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=417s) Modernizing Orchestration with Lakeflow Jobs and Production Milestones

The final piece of the legacy stack is the orchestrator, historically dominated by Apache Airflow. Many teams cling to Airflow solely because they want to define Directed Acyclic Graphs (DAGs) in Python. However, self-managed Airflow comes with severe operational liabilities; an internal Databricks customer survey revealed that 80% of organizations run outdated Airflow distributions riddled with unpatched security vulnerabilities and lacking modern orchestration features. Lakeflow Jobs directly addresses this by offering fully serverless orchestration with native support for pure Python DAGs alongside visual point-and-click workflow builders.

Historically, Databricks orchestration was criticized for excelling only at internal Databricks tasks while struggling with external enterprise systems. Bilal announces over 50 open-source integrations and operators for Lakeflow Jobs, enabling unified orchestration across third-party data platforms—including competing warehouses like Snowflake.

Closing out the architectural overview, Bilal shares production metrics demonstrating that Lakeflow is running at immense enterprise scale rather than serving as an experimental prototype:

Bilal underscores the battle-tested nature of the platform: “This is not a brand new product. We've been at this for a while and it's ready for the world's toughest data workloads.”

- **Spark declarative pipelines throughput:** Currently processes over 200 trillion rows of data every single day. - **Lakeflow Jobs execution volume:** Powering roughly 1.7 billion job runs each month across global enterprise environments. - **Serverless compute adoption:** 50% of Databricks customers have opted into serverless compute to eliminate cluster configuration and VPC provisioning. - **Orchestration breadth:** 50+ open-source operators added to Lakeflow Jobs to trigger external systems and platforms like Snowflake.

[08:48](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=528s) The Data Operations Bottleneck: Why Coding Agents Fail

Bilal Aslam points out an uncomfortable irony: making it radically easier to build data pipelines with AI and open stacks actually threatens to make the operational burden far worse. If anyone across an organization can effortlessly spin up pipelines, the inevitable byproduct is more outages, more broken jobs, and more operational chaos. According to a partner survey Aslam cites, data engineering teams already spend more than 50% of their working hours purely on pipeline maintenance, yet organizations still endure up to 60 hours of data downtime every single month. While teams might hope that general-purpose coding agents can inspect API endpoints or table schemas and solve these incidents automatically, Aslam stresses that data engineering is fundamentally different from traditional software engineering.

``` +---------------------------+-----------------------------------+-----------------------------------+ | Dimension | Software Engineering | Data Engineering | +---------------------------+-----------------------------------+-----------------------------------+ | Composition | Code is self-describing | Code + Data | | Testing | Fully deterministic (pass/fail) | Statistical evaluation | | Failure Modes | Loud exceptions, stack traces | Silent and permanent corruption | +---------------------------+-----------------------------------+-----------------------------------+ ```

In software engineering, code is self-describing and tests are deterministic—code either compiles and passes unit tests, or it does not. In data engineering, pipelines are an inseparable union of code and data, meaning tests cannot simply evaluate static logic; they must evaluate statistical distributions across dynamic datasets. Furthermore, software failures announce themselves loudly through explicit exceptions and error traces that allow immediate deployment rollbacks. In data engineering, failures occur silently and permanently, quietly corrupting downstream systems until an executive discovers bad numbers days or weeks later. As Bilal Aslam puts it: “In data engineering, failures are silent and permanent.”

[10:39](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=639s) The Trust Dilemma and the Data Plane Breakthrough

General coding agents stumble across the entire operational lifecycle because they lack the proper context and data plane integration. For incident detection, coding agents lack direct telemetry; feeding them multi-megabyte Spark traces quickly exhausts LLM context windows. For root cause assessment, they lack end-to-end data lineage unless an engineer manually exports catalog graphs out of the system. Even if an agent writes a code fix, verifying that fix presents an even steeper barrier: an engineer cannot simply run a unit test in an isolated sandbox. Validating a data pipeline fix requires running code against actual production data.

This verification requirement creates an acute security and governance dilemma. To identify root causes, an autonomous agent needs read access to production tables. To verify that its fix actually resolves data discrepancies without introducing regressions, the agent needs write access to production data. Aslam polls the audience directly, asking who would willingly hand uncontrolled read and write privileges over their production crown jewels to a third-party coding agent they did not build; not a single hand goes up in the room. Aslam observes that granting arbitrary agents write permissions invites disaster because an untrusted model could easily drop critical tables. The Databricks team realized that operations cannot be solved by an external tool sitting outside the perimeter: “The big realization we came to was that your operations agent actually needs to live in the data plane.” Only the data plane natively encompasses the underlying data, the Unity Catalog lineage graph, and the required governance boundaries.

[12:32](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=752s) Architecture of Genie ZeroOps

To solve operational automation safely within the governance boundary, Aslam introduces Genie ZeroOps, an autonomous background agent designed to place data and AI operations on autopilot. Rather than relying on generic prompt engineering over external logs, Genie ZeroOps embeds directly into the Databricks platform across five distinct operational phases:

```mermaid flowchart LR accTitle: Genie ZeroOps Five-Stage Operational Loop accDescr: Diagram illustrating the five continuous operational stages of Genie ZeroOps within the Databricks data plane.

subgraph Sense["Detection & Assessment"] A["Autonomous ML Models<br/>(Metrics & Logs)"] --> B["Graph Ranking<br/>(Unity Catalog Lineage)"] end

subgraph Investigate["Root Cause & Remediation"] B --> C["Supervisor & Sub-Agents<br/>(Consensus Engine)"] C --> D["Genie Code<br/>(Ticketing & Git PRs)"] end

subgraph Validate["Safe Verification"] D --> E["Shallow Clones<br/>(Isolated Data Branches)"] end ```

- **Detection:** Genie ZeroOps autonomously generates and continuously fine-tunes per-table machine learning models. With native internal access to metrics, runtime events, and compute logs, Databricks already runs tens of thousands of these operational ML models in active production without requiring user-built monitoring plumbing. - **Assessment:** When an anomaly surfaces, the agent performs graph ranking algorithms across Unity Catalog data lineage, systematically traversing upstream dependencies and downstream consumers. - **Root Cause Analysis:** A dedicated supervisor agent coordinates a fleet of specialized sub-agents. These sub-agents research potential failure points along the graph and deliberate until reaching consensus on the most probable root cause. - **Remediation:** Genie ZeroOps collaborates with Genie Code, connecting to version control repositories and enterprise ticketing systems to file and update incident tickets throughout the debugging lifecycle. - **Verification:** Leveraging a decade of open lakehouse engineering, the agent creates shallow clones of production tables—fast, zero-copy metadata branches similar to Lakeflow branching. It executes the proposed remediation inside Databricks' native network and code isolation boundaries, validating output distributions before touching real data.

As Bilal Aslam announces to the audience: “I'm super, super excited to announce Genie ZeroOps, a new background Genie that puts your data and AI operations on autopilot.”

[14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) Live Demo: Autopilot Incident Triage and Autonomous PII Remediation

Transitioning to a live demonstration, Aslam reveals that the Genie ZeroOps interface deliberately avoids looking like a standard observability dashboard packed with flashing red lights, confusing graphs, and alert fatigue. Instead, it mirrors a prioritized email inbox where incidents are ranked by business severity. The system intelligently groups related failures; in one example shown on screen, Genie ZeroOps deduces that 16 distinct failing batch jobs all stem from a single shared upstream incident, collapsing 16 noisy alerts into one actionable thread.

Opening an incident, Aslam examines an issue where an upstream table named `top_fan_voters` suffered an abrupt drop in row counts approximately ten minutes earlier. Aslam emphasizes that engineers do not have to start an investigation from scratch: “And what Genie ZeroOps does is by the time you log in, your background agent has actually done all the thinking for you.” The system walks downstream lineage to determine that `top_fan_voters` feeds a critical executive-facing fan engagement dashboard, automatically elevating the incident priority. Simultaneously, the agent's root-cause sub-agents fan backward along the Directed Acyclic Graph (DAG), bypassing healthy upstream nodes to pinpoint the exact failure inside an upstream `fan_interactions` table.

``` [Upstream Root Cause] [Detected Anomaly] [Downstream Impact] fan_interactions --------(lineage)-------> top_fan_voters --------(lineage)------> Fan Engagement Dashboard (Faulty logic/data) (Sudden row drop) (Critical business asset) | | +---- Genie ZeroOps writes fix +---- Verified on shallow clone & opens Pull Request before PR creation ```

Genie ZeroOps writes a source code fix, but it does not apply it directly to production. After receiving explicit administrative permission, it provisions a shallow clone of the petabyte-scale production table, applies the candidate fix in complete isolation, and verifies that the output produces the expected row counts. With the fix verified against real-world data patterns, the engineer can review the work and open a Git pull request in one click.

Aslam showcases a second capability: autonomous governance monitoring. When granted scan permissions, Genie ZeroOps continuously audits production tables for regulatory and security risks, such as exposed personally identifiable information (PII). In the demo, the agent detects an application accidentally exposing the PII of roughly 1,000 users. It compiles an impact report detailing the specific fields leaked, analyzes downstream table lineage, and generates a ready-to-deploy Unity Catalog governance policy fix rather than an application code patch. Aslam concludes that by automating deep triage, safe verification, and policy enforcement, Genie ZeroOps frees data engineers from endless break-fix maintenance so they can focus on building high-value systems.

## Notable quotes > [08:48](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=528s) In data engineering, failures are silent and permanent. — Bilal Aslam

> [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure. — Bilal Aslam

> [10:39](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=639s) The big realization we came to was that your operations agent actually needs to live in the data plane. — Bilal Aslam

## Mentioned Bilal Aslam, Mukesh, Arsalan, Ali Ghodsi, Kenan, Databricks, Data + AI Summit, Lakeflow, Lakeflow Designer, Lakeflow Connect, Lakeflow Jobs, Genie, Genie Code, Genie ZeroOps, Zerobus Ingest, Unity Catalog, Apache Spark, Apache Kafka, Apache Flink, Apache Airflow, Salesforce, NetSuite, Oracle, Microsoft SQL Server, Amazon EMR, dbt, Snowflake, Git

## Video description Data engineering is a messy process, and the growth of AI apps and agents is adding more complexity. The data that AI needs is stored all over the enterprise.

Previously, companies had to rely on brittle, expensive pipelines to move this information. But Lakeflow make it easy to turn raw data stored in these systems into insights and action.

At the 2026 Data + AI Summit, Bilal Aslam, Sr. Director of Product Management at Databricks, shared updates to Lakeflow and introduced Genie ZeroOps, an AI agents for data and AI operations.

Learn more: https://www.databricks.com/blog/introducing-genie-zeroops

00:00 —Data engineering and the "messy middle" 02:21 — Simpliyfing the data stack 03:18 — Building an open data foundation for AI with Lakeflow 04:50 — Now available: Lakeflow Designer 05:40 — Lakeflow Connect's growing ecosystem 06:18 — Ditching Kafka with Zerobus Ingest 06:57 — Updates to Lakeflow Jobs 08:47 — Challenges to using AI in data operations 12:34 — Introducing Genie ZeroOps with live demo 17:23 — What Genie ZeroOps means for data engineers

Presenters: Bilal Aslam, Sr. Director of Product Management, Databricks

[Source](https://www.youtube.com/watch?v=0WDGu-IPmZM)

Reading summary

Databricks presenter Bilal Aslam argues that while AI agents require high-quality data, the rapid proliferation of automated pipelines threatens to overwhelm fragile, fragmented data architectures. To prevent operational gridlock, Databricks proposes consolidating data stacks onto open declarative pipelines and embedding an autonomous operations agent, Genie ZeroOps, directly within the data plane. Pipeline Proliferation Exacerbates the Data Maintenance Crisis (0:00) Data engineers are trapped in the "messy middle," tasked with extracting, cleaning, and transforming raw records scattered across Kafka, Salesforce, NetSuite, and legacy relational databases into reliable assets for downstream AI systems. Historically, teams stitched together dozens of point tools under fragile architectures that lack unified version control, consistent governance, and predictable scaling. While AI coding agents like Genie Code provide immense authoring speed—generating 60% of all Lakeflow pipelines within three months of release—Aslam warns that this velocity poses an existential threat to already fragile infrastructure. Citing a partner survey, Aslam notes that data teams already allocate over 50% of their working hours strictly to pipeline maintenance, yet enterprises still suffer up to 60 hours of data downtime per month. Accelerating pipeline creation without standardizing the underlying architecture inevitably multiplies operational failures. Unifying Ingestion, Processing, and Orchestration Under Lakeflow (3:37) To eliminate fragmented runtimes, Lakeflow consolidates workloads using open Apache Spark declarative pipelines, where users specify the target data state and the engine handles execution across environments from local laptops to cloud clusters. The declarative approach unifies SQL and Python workflows, while an integrated open-source "real-time mode" engine brings millisecond stream processing to Spark, removing the need for separate stream engines like Apache Flink. For business analysts, the generally available Lakeflow Designer provides a no-code visual interface that compiles directly into standard Spark declarative pipelines, curbing proprietary shadow IT. At the ingestion boundary, Lakeflow Connect provides over 100 native connectors running as declarative pipelines. For high-volume streaming, Databricks introduced Zerobus Ingest, a fully managed service that is 100% wire-compatible with the Apache Kafka protocol. Zerobus lands data directly into open lakehouse storage without broker management, handling sustained throughputs of up to 12 gigabytes per second while eliminating small-file storage bloat. Orchestration is addressed by Lakeflow Jobs, a serverless engine managing 1.7 billion monthly runs with native Python DAG support and over 50 open-source operators that can trigger third-party warehouses such as Snowflake. Why Generalist Coding Agents Stumble on Data Operations (8:48) Aslam stresses that general-purpose coding agents cannot resolve pipeline failures because data engineering differs fundamentally from software engineering. Software code is self-describing, tests are deterministic, and failures produce explicit exceptions. Conversely, data pipelines represent an inseparable union of code and dynamic data where validation requires statistical evaluation, and errors manifest as silent, permanent data corruption. External coding agents also face technical and security constraints. Multi-megabyte Spark error logs quickly overwhelm model context windows, and external models lack catalog data lineage. Crucially, verifying a pipeline fix requires running candidate code against production data. Granting third-party agents uncontrolled write permissions to production environments introduces intolerable security and governance risks, such as accidental table drops. Consequently, Aslam asserts that automated operational agents must reside inside the data plane where security perimeters and lineage graphs already exist. Autonomous Triage and Safe Verification via Genie ZeroOps (12:32) Databricks introduced Genie ZeroOps to place pipeline monitoring and remediation on autopilot within the data plane. The system operates across five stages: anomaly detection powered by continuously fine-tuned per-table machine learning models, graph ranking across Unity Catalog data lineage, multi-agent consensus coordinated by a supervisor agent, code remediation linked to version control, and isolated verification. To remediate failures safely, Genie ZeroOps provisions shallow clones—fast, zero-copy metadata branches of production tables—to test code fixes within network isolation before modifying live datasets. During a live demonstration, the agent triaged an anomaly in an upstream interactions table that caused a row-count collapse in an executive dashboard, grouped 16 related batch failures into a single incident, verified the fix on a shallow clone, and prepared a Git pull request. The agent also demonstrated continuous compliance auditing, detecting exposed personally identifiable information across roughly 1,000 users and generating ready-to-deploy Unity Catalog governance policies. Aslam concludes that scaling enterprise AI requires freeing data engineers from break-fix maintenance through unified open standards and in-engine automation. While Lakeflow and Genie ZeroOps demonstrate high throughput and automated triage in production, the model's broader impact depends on organizations trusting autonomous background agents to audit and remediate production data assets.

Source quotes

[8:48] Bilal Aslam: In data engineering, failures are silent and permanent. [2:21] Bilal Aslam: Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure. [10:39] Bilal Aslam: The big realization we came to was that your operations agent actually needs to live in the data plane.

Original transcript

[0:00] Welcome to the stage, Databricks senior [0:02] director of product management, Bilal [0:05] Aslam. [0:09] >> [screaming] [0:11] >> Good morning. Good morning. [0:14] I'm going to talk to you about Lakeflow, [0:16] but just a little quick note first. I [0:18] wore this I wore this purple jacket 2 [0:20] years ago, talked about Lakeflow. I did [0:22] not wear it last year, and the biggest [0:24] piece of feedback was bring it back. So, [0:26] here it is. [0:27] Uh my name is Bilal Aslam. I'm a senior [0:29] director of product management at [0:30] Databricks. Super excited to talk to you [0:31] about Lakeflow. Now, uh first of all, [0:34] thank you to Mugesh and Arsalan and Ali [0:36] and Kenan and everybody who's come here [0:37] and talked about AI and agents. Now, uh [0:40] super exciting future, uh but if you're [0:42] anything like me, if you're a data [0:43] engineer, uh you're probably sitting [0:45] there thinking, "This is great, but all [0:46] of this needs data." Uh all these apps, [0:49] agents, AI, everything needs data, and [0:52] you might be thinking, "My job got a [0:53] little bit harder." [0:55] Uh [0:55] and that's because uh [0:57] with data engineering, the destination [0:58] is simple. The destination is get to the [1:00] right. Get to the business outcomes we [1:02] want. We want agents, applications, we [1:04] want real-time operations, but we're [1:06] actually starting from the left. We're [1:08] starting from raw data sources. We have [1:10] data in Kafka, Salesforce, NetSuite. We [1:13] have on-premise databases like Oracle [1:14] and SQL Server. And the messy middle, [1:17] that's data engineering. That's our job [1:19] to take that raw data, turn it into [1:20] insights. [1:22] And over the years, we have built [1:24] architectures like these. I put this up [1:26] here because it's representative. You [1:28] don't have to look at every box here. [1:29] It's a lot of logos. It's a lot of [1:31] boxes. It's a lot of lines between [1:33] boxes. And that's just to get data from [1:36] one place to the other. [1:37] And you know, we've made peace with [1:39] complexity. Sure, there's a lot of [1:40] tools, but you're probably sitting in [1:41] here thinking, "Okay, at least it [1:43] works." [1:44] Uh even though you know instinctively [1:46] that this complicated, not unified [1:49] architecture is missing a bunch of [1:50] critical things, we want to version [1:52] control everything as data engineers, [1:54] but in this stack, you can version [1:55] control some things, not the others. Uh [1:57] you can version version control your [1:58] Spark code, but not your Kafka [2:00] infrastructure. [2:01] You want governance everywhere, but it's [2:02] pretty much impossible. You're reading [2:04] and writing to too too many different [2:05] places, and there are parts of the [2:07] stack, as we will see soon, that are [2:08] just locked up. And finally, you want [2:10] everything to scale, and you want to [2:12] save money. Uh so you can build more [2:14] applications and outcomes for your [2:15] business, but not all parts of the stack [2:17] scale equally well. [2:20] I'm here to tell you that your job and [2:22] my job as a data engineer is about to [2:24] get quite a bit harder. [2:26] What you're seeing here is Genie code. [2:28] It's a purpose-built coding agent for [2:29] data science, machine learning, and data [2:31] engineering. [2:32] Uh if you're using it, that's awesome [2:34] because uh 60% of Lake Flow pipelines, [2:37] this is a crazy statistic, are already [2:39] written by Genie code in just 3 months. [2:43] We just released this. [2:44] It's pretty awesome. [2:46] On the other hand, this statistic scares [2:47] me a little bit as a data engineer. [2:49] And that's because I have a crush on [2:51] agents. Agents are great, but all agents [2:53] mean new data, new pipelines, more to [2:56] build, more to manage on that fragile [2:58] infrastructure. We're going to get [3:00] swamped. [3:02] So what do we do? We start by removing [3:04] the moving pieces. And actually, it's [3:05] super important. We don't just want to [3:07] remove complexity with more complexity. [3:09] We want to replace that with open [3:11] formats, open open frameworks, open [3:13] tools. We ultimately need to simplify [3:15] the data stack. [3:17] I'm here to make a pretty bold claim, [3:19] and my bold claim is that Lake Flow is [3:21] that unified stack. That it gives you [3:23] that open foundation for AI and agents [3:26] so that you can build the future that [3:28] your company deserves. And now, we work [3:30] with hundreds of partners, lots of [3:31] tools. They're all interoperable with [3:33] Lake Flow and the lakehouse. You can use [3:35] them as well. [3:36] So let's get started. I'm going to do a [3:37] little bit of a speed run through this. [3:39] Uh how do you simplify data [3:40] transformation? ETL. This is where [3:42] you're spending most of your time and [3:43] money. [3:44] Uh you have Spark. You have lots of [3:46] legacy Spark jobs. You may have a Spark [3:49] distribution on EMR or some other [3:51] vendor. [3:52] I'm going to replace that with Apache [3:53] Spark declarative pipelines. If you're [3:55] not using them, please try them out on [3:57] Lake Flow. They're open, you can run [3:59] them anywhere including on your laptop, [4:01] and they're declarative so you can just [4:03] focus on the work to be done. And [4:05] because they're open, agents are really [4:07] really good at writing these pipelines. [4:09] They unify batch and streaming. [4:11] Many years ago, DBT was the only game in [4:12] town when it came to SQL. I like writing [4:14] ETL in SQL, but Spark declarative [4:16] pipelines unifies that as well. It has [4:18] SQL and Python. You don't have to [4:20] choose. And finally, if you're using [4:22] Flink for real-time streaming, that's [4:24] probably because actually low latency [4:25] streaming was really hard on Databricks [4:28] and Spark. But now, last year we open [4:30] sourced real-time mode, which is a brand [4:32] new engine for low latency stream [4:34] streaming, and I'm super excited to [4:35] share that it's now available inside [4:37] Spark declarative pipelines. So you can [4:39] get millisecond streaming on an open [4:41] framework that runs anywhere. [4:43] Awesome. So we're taking care of this [4:45] complexity, we're getting rid of this, [4:46] we're building an open foundation for [4:47] agents and AI. [4:49] Well, what about all the shadow IT [4:51] that's happening? So you see, you may [4:53] have 20 data engineers in your company, [4:55] but you have hundreds of analysts who [4:56] are building pipelines in drag and drop [4:58] tools. These pipelines are built on [5:00] their laptops, proprietary formats, [5:03] proprietary tools, and if somebody [5:05] leaves the team, you're out of luck. [5:08] Super excited to share that Lake Flow [5:09] Designer, the no-code data prep tool [5:13] powered by Genie, is now available. It's [5:15] generally available. You can try today, [5:17] you can try it right now. [5:20] >> [applause] [5:20] >> Thank you. [5:23] Awesome. And by the way, one super cool [5:25] thing is that Lake Flow Designer doesn't [5:27] build any proprietary pipelines. It just [5:29] builds Spark declarative pipelines under [5:31] the hood. Again, you can just run them [5:33] anywhere. [5:34] I think you're starting to see a [5:34] pattern. All right, we're going to go a [5:36] little faster. I'm going to go ahead and [5:37] simplify data ingestion. If you're using [5:40] a SaaS that has hundreds of connectors, [5:42] that's great, but it's probably landing [5:44] data in proprietary formats. Your agents [5:47] don't know how to operate this SAS. You [5:49] cannot automate this. You can barely [5:51] monitor it or observe it or govern it. [5:53] As Ali mentioned, I'm super excited to [5:55] share that LakeFlow Connect now has more [5:57] than 100 connectors. If there's a [5:59] connector you're thinking of, we've [6:01] either built it, we're building it, or [6:03] we're going to build it. And there's a [6:04] new community that's building [6:06] open-source connectors. You can even [6:07] build your own. And guess what? I think [6:09] you're going to guess what I'm going to [6:10] say next. This isn't a special type of [6:12] pipeline or ETL. It's just Spark [6:14] declarative pipelines under the hood. [6:16] That was pretty awesome. [6:18] All right. Let's talk about Kafka. As [6:20] data engineers, we use Kafka day in and [6:22] day out for high-volume telemetry. We're [6:24] bringing that in. Kafka's a buffer, but [6:26] we also know Kafka's a real pain to [6:29] manage, okay? [6:31] Super excited to share that Zero Bus [6:33] Ingest is a fully managed service that [6:35] is now 100% wire compatible with Kafka. [6:38] So, you can take your Kafka producing [6:39] code, point it at Zero Bus Ingest, and [6:42] you can land data into the lakehouse in [6:44] open formats, ready for AI. So, no small [6:47] files, millions of small files at 12 GB [6:50] a second. [6:51] So, it's pretty awesome. You can get rid [6:53] of Kafka for your stack. [6:55] Great. [6:56] Uh okay, so we're almost there, and [6:58] you're going to see we're still building [7:00] everything on an open ETL framework [7:02] that's declarative. But you see you see [7:05] this little uh Airflow icon hanging out [7:07] there. That's your orchestrator. You [7:08] have to use that to trigger things, to [7:10] schedule things. Let's simplify that as [7:12] well. So, you're probably using Airflow [7:14] because you like writing workflows in [7:15] Python. Well, LakeFlow Jobs supports [7:18] writing workflows in Python now. It's [7:19] just uh pure Python. You can also build [7:22] point-and-click DAGs. [7:24] It's fully serverless, so you don't have [7:25] to manage any infrastructure. And one [7:27] interesting fact is that we did a survey [7:30] of our customers, and 80% of our [7:31] customers are using old Airflow [7:33] distributions. So, they're managing [7:35] them, maintaining them, they have [7:36] security vulnerabilities, and they're [7:38] not even getting all the new features, [7:40] okay? Uh and I'm the last thing I'm [7:42] really excited about because for the [7:43] longest time, let's face it, with jobs, [7:45] it was really good at orchestrating [7:46] Databricks, not so great at [7:48] orchestrating other systems. So, super [7:50] excited. This probably one of my [7:51] favorites in this conference is we're [7:52] releasing 50 plus integrations. So, you [7:54] can orchestrate everything. You can [7:56] think of these as Airflow operators. [7:57] They're open source and you can [7:59] orchestrate everything including [8:00] Snowflake. Okay. So, where where does [8:02] this bring us? This brings us to a an [8:05] open stack. Everything is pretty much [8:07] building open source declarative [8:09] pipelines. [8:10] And [8:12] this is not a brand new product. We've [8:15] been at this for a while and it's ready [8:17] for the world's toughest data workloads. [8:19] I want to share some statistics with [8:20] you. Spark declarative pipelines now [8:23] process 200 trillion rows of data every [8:25] single day. [8:27] Lake Flow jobs is [8:29] running the world's biggest data [8:30] workloads. It's about 1.7 billion job [8:33] runs per month. And finally, serverless [8:36] compute is really popular. You don't [8:37] want to manage clusters. You don't want [8:39] to set up VPCs. 50% of our customers [8:41] have now opted in to use serverless [8:43] compute. That's pretty awesome. [8:46] Great. So, we've I hopefully I have [8:48] convinced you to some degree that we [8:50] made it easier to build. And now you [8:52] might be thinking, "Okay, we have an [8:53] open stack. I can start building AI can [8:56] write the code." Maybe you're going to [8:57] go try out Genie and it's going to build [8:59] something for you. [9:00] But what about operations? I haven't [9:03] really solved that for you. Actually, [9:04] I've kind of made the problem worse, [9:05] right? Because now you're going to say, [9:07] "Great. Now more people can build more [9:09] pipelines, [9:10] more outages, more problems." [9:13] And here's the reality. Simplifying [9:15] pipeline creation is actually the easy [9:17] part. Although this was pretty tough, [9:18] right? [9:19] It's data operations that are really, [9:21] really hard. Here's a survey that is by [9:24] one of our partners and it shows that [9:26] data engineering teams spend more than [9:28] 50% of their time on maintenance. So, [9:31] that's where your time is going. It's [9:32] only going to get worse. And even with [9:34] this time spent on maintenance, you [9:36] still see up to 60 hours of downtime [9:39] every month. So, this sucks. [9:42] Now, you might be thinking, uh but [9:43] Bilal, you just told me that this is a [9:45] unified stack. Surely, it has APIs and [9:47] endpoints and tables. Maybe a coding [9:49] agent can just fix it. And [9:51] unfortunately, not really. Let me walk [9:52] you very quickly through why. And the [9:54] big idea here is that data engineering [9:56] is not software engineering. It's [9:57] something a bit more. [9:59] In software engineering, your code is [10:01] self-describing. [10:02] You It's It just literally describes [10:04] itself. And in data engineering, it's [10:06] data and code. [10:08] In software engineering, your tests [10:10] either work or they don't. They're fully [10:12] deterministic. In data engineering, you [10:14] have code and data. You can't just test [10:16] data. It's statistical. [10:19] And finally, in software engineering, [10:21] and this is the big one, failures are [10:23] loud. When something goes wrong, you're [10:25] going to see a trace. You're going to [10:26] see an exception. And then you can roll [10:27] back your deployment. In data [10:29] engineering, failures are silent and [10:31] permanent. You're probably going to get [10:33] an angry phone call from a CEO or a [10:34] stakeholder saying, "Hey, how come this [10:36] data is bad?" Maybe a week later. [10:39] Uh so, why do coding agents trip up [10:41] here? Well, that's because when it comes [10:43] to detection, they're just missing data. [10:46] They're missing telemetry. And you can [10:48] give them Spark logs, for example, but [10:50] these are multi-megabyte traces. You [10:52] have to be very careful to manage the [10:53] infra- the the context window there. [10:55] And similarly, when it comes to [10:57] assessing the root cause, they're [10:58] missing lineage. Now, you have to export [11:00] your lineage out of the system. And to [11:02] some degree, you can do this. Coding [11:03] agents are getting better and better and [11:05] context windows are getting better. And [11:07] then you have to remediate. They have to [11:09] write the code fix. But it's actually [11:11] the final step, which I'm going to be a [11:13] little pedantic about and call it [11:14] verify. [11:15] You can't with data, you can't just run [11:17] your unit tests. You actually have to [11:19] take your code fix, and you have to run [11:21] it on production data. Okay. And if this [11:24] is giving you shivers, that's exactly [11:26] right. It really should. So, what does [11:27] your agent need? Your operations agent [11:30] needs to combine code and data. That's [11:32] tools and skills to some degree it can [11:34] do it. [11:36] To figure out the root cause, it needs [11:39] read access to production data. [11:41] Uh, who's comfortable with that? It's an [11:43] agent you didn't write and you don't [11:45] trust. I I don't see a single hand [11:47] raised here. [11:49] But actually, your coding agent also [11:51] needs write access to production data. [11:54] That's the only way it can verify the [11:56] fix. Like who's here wants to give a [11:58] random agent write access to their [12:00] production data? There's got to be one [12:02] person. [12:03] Okay, there's no one who wants to do [12:05] that, right? Because it can cause chaos. [12:07] It can drop everything. [12:10] And this is the big insight. So, we've [12:11] been looking at this problem, it's a [12:12] tough problem, and the big realization [12:14] we came to was that your operations [12:16] agent actually needs to live in the data [12:19] plane. It cannot live outside of the [12:21] data plane. And if you think about it, [12:22] that makes sense because the data plane [12:24] has the data. It has the lineage. It's [12:27] also the right governance boundary. [12:29] So, [12:32] I'm super, super excited to announce [12:34] Genie Zero Ops, a new background Genie [12:37] that puts your data and AI operations on [12:39] autopilot. [12:41] Great. I'm going to give you a demo, but [12:43] let me quickly walk you through how it [12:44] works. [12:45] So, for detection, Genie Zero Ops [12:48] autonomously builds per table machine [12:50] learning models, and it continuously [12:52] fine-tunes them. And it has native [12:54] access to metrics, events, and logs. So, [12:56] you don't have to build this plumbing [12:57] yourself. We already have tens of [12:59] thousands of these machine learning [13:00] models in production with Genie Zero [13:02] Ops. [13:04] To assess, it does graph ranking on data [13:07] lineage in Unity Catalog. So, I'll show [13:09] you how it walks the lineage forward and [13:11] backwards. And then to figure out the [13:13] root cause, and this is where you spend [13:14] a lot of your time. The root cause is [13:15] basically what went wrong. It actually [13:17] has a supervisor agent and a fleet of [13:19] sub agents who do research and then come [13:22] to a consensus on what is the likeliest [13:23] cause. [13:25] For remediation, Genie Zero Ops works [13:27] with Genie code. It cooperates. If Genie [13:29] code has access to your code, to your [13:31] ticketing system, to your version [13:33] control, it uses all that. So, it can [13:35] also update tickets for you as it goes [13:36] through the life cycle. [13:38] And finally, verification is the [13:40] toughest step. To verify, this is the 10 [13:42] years of building an open lakehouse. It [13:44] builds shallow clones of your production [13:46] data. These are super cheap, super fast [13:49] branches, similar to what we do in [13:50] Lakebase. And then it utilizes native [13:53] network and code isolation in [13:54] Databricks. All right. So, I'm going to [13:56] show you a quick demo of what Genie Zero [13:58] Ops actually does. [14:01] Great. So, let's see. [14:04] All right. So, you're going to notice a [14:06] couple of things here. First of all, [14:08] this does not look anything like a [14:10] dashboard. We're not flooding you with [14:12] metrics, events, alerts, light uh you [14:15] know, red lights and all that. What you [14:17] see is something that looks and feels a [14:19] lot like an email inbox. It's actually, [14:21] we modeled it after a prioritized email [14:23] inbox. So, for example here, one cool [14:26] thing is that you get to see severity. [14:28] So, every incident is ranked by [14:30] severity, so you can only you have you [14:31] can just focus on what really matters. [14:33] And also, this isn't just a list of jobs [14:36] and pipelines and failures and things [14:37] like that. Over here, for example, [14:39] here's an alert and Genie Zero Ops [14:41] figured out that hey, 16 jobs are [14:43] failing, they're actually the same [14:45] incident. So, the natural grouping saves [14:47] you time. [14:49] All right. So, I'm going to go ahead and [14:50] look at an actual incident. And what's [14:52] happening here is that there's an [14:54] upstream table called top fan voters, [14:56] and something's going wrong with it [14:58] about 10 minutes or so ago, [15:00] uh the row count dropped quite a bit. [15:03] And what Genie Zero Ops does is by the [15:05] time you log in, your background agent [15:08] has actually done all the thinking for [15:10] you. It has done all the investigation [15:11] for you. And as a data engineer, I can [15:14] start looking at things like what's the [15:15] impact. Now, when it comes to impact, [15:18] not all tables are important, not all [15:20] pipelines are important. So, it walks [15:21] the lineage forward, but in this case it [15:23] figures out that, "Hey, top fan voters [15:25] is a pretty important table. Downstream [15:27] tables depend on it, and this fan fan [15:29] engagement dashboard depends on it." So, [15:31] it marks it as critical. [15:33] Then, it deploys the root cause agents [15:35] to figure out what's going on. And what [15:37] these agents do is this is where they're [15:39] saving you hours of time. [15:41] They start from this table, and they fan [15:43] back using lineage, and they go [15:45] investigate potential causes upstream. [15:47] So, over here, this is a fairly simple [15:49] DAG. For example, you know, it could be [15:52] any one of these tables, but really it [15:53] finds this one table, this fan [15:55] interactions table, that's the one [15:57] that's problematic. [15:58] Okay. Then, it went ahead and [16:00] autonomously wrote a fix for you, but [16:03] this fix is not deployed to production. [16:05] You're in control. What it did do, and I [16:07] want to share this, this is super cool, [16:09] is that it went and created a shallow [16:11] clone. It's like the lake-based branch [16:12] of your production data. By the way, you [16:14] have to give it permission. It won't do [16:16] it automatically. You have to tell it [16:17] you can do it with this pipeline. [16:18] And this could be petabytes of data. [16:21] It's a shallow clone. It deployed the [16:23] fix, and it actually verified that the [16:24] right number of rows are returned. This [16:26] is awesome. This is hours, if not days [16:29] worth of work. And now at this point, [16:31] I'm just going to go ahead and create a [16:32] pull request, and I can follow my [16:34] software development life cycle. [16:38] Uh let me show you one other super cool [16:39] thing that Genie Zero Ops can do. The [16:41] one other thing Genie Zero Ops does is [16:44] if you give it permission, it will [16:46] autonomously scan tables for things like [16:48] PII. So, in this case, our application [16:50] is unwillingly exposing the PII of about [16:53] a thousand users. Okay? That's pretty [16:55] bad. [16:56] And Genie Zero Ops creates a report for [16:57] me. It finds out what the PII, the [16:59] personally identifiable information is. [17:02] It gives me a per-table breakdown. It [17:05] looks at lineage to figure out this is [17:06] important. Yeah, this is pretty bad. And [17:08] then in this case, it comes up with a [17:10] proposed fix that's not a code fix. It's [17:12] a It's a Unity Catalog a policy fix, and [17:15] I can deploy it. [17:17] Okay, so that's Genie Zero Ops. We're [17:19] super excited about it. [17:21] And for data engineers like me, and [17:23] hopefully for you, it means far less [17:26] time building, far less time [17:28] maintaining, and much more time building [17:31] on a unified and simple data stack. [17:33] Thank you very much.