Databricks
External coding agents fail at data operations because verifying fixes requires production data access, forcing remediation models directly into the governed data plane to prevent silent corruption.
AI-assisted summary of Unlocking agentic data engineering with Lakeflow + Genie — original source. Check the original for full context.
Source published:
Key ideas
- Within three months of release, 60% of all Lakeflow pipelines are already being generated directly by Genie Code, accelerating the volume of data pipelines organizations must manage. (2:21)
- Databricks introduced Zerobus Ingest, a fully managed service that is wire-compatible with the Apache Kafka protocol, eliminating broker operations and streaming up to 12 GB/s directly into the lakehouse. (5:36)
- Lakeflow Designer is now generally available as a no-code visual preparation tool that natively compiles workflows into open Apache Spark declarative pipelines rather than proprietary formats. (4:49)
- Databricks unveiled Genie ZeroOps, an autonomous background operational agent running inside the data plane that detects pipeline anomalies, identifies root causes via Unity Catalog lineage, and validates code fixes on isolated shallow clones. (12:32)
Detailed notes
## Background Published on YouTube by Databricks on June 23, 2026, this keynote session features Bilal Aslam, Sr. Director of Product Management, presenting at the 2026 Data + AI Summit. The talk addresses the mounting complexity of enterprise data engineering and details how Databricks' Lakeflow and the newly introduced Genie ZeroOps autonomous agent simplify pipeline construction and operations.
## TL;DR Bilal Aslam examines the grueling 'messy middle' of data engineering, warning that the unchecked explosion of AI coding agents risks overwhelming brittle, legacy data infrastructure. To counter this, Databricks unifies ingestion, transformation, and orchestration on an open foundation through Lakeflow—spanning Spark declarative pipelines, Lakeflow Designer, Zerobus Ingest, and serverless Lakeflow Jobs. Addressing the operational maintenance bottleneck, Aslam introduces Genie ZeroOps, an autonomous agent living natively inside the Databricks data plane. Driven by Unity Catalog lineage and zero-copy shallow clones, Genie ZeroOps autonomously triages pipeline failures, tests candidate code fixes against isolated production data, and enforces governance policies to eliminate data downtime.
## Atmosphere & style The presentation blends energetic keynote showmanship with an empathetic, candid peer-to-peer conversation, opening playfully with an inside joke about Bilal's signature purple jacket before validating the genuine burnout of overworked data engineers. Aslam builds narrative tension by contrasting the deterministic simplicity of software engineering against the insidious, silent failures endemic to data pipelines. The mood shifts from realistic wariness to triumphant relief during the live demonstration, where chaotic operational fire drills are replaced by a calm, inbox-style interface that handles end-to-end incident triage with effortless precision.
## Bulletin - [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) Within three months of release, 60% of all Lakeflow pipelines are already being generated directly by Genie Code, accelerating the volume of data pipelines organizations must manage. - [05:36](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=336s) Databricks introduced Zerobus Ingest, a fully managed service that is wire-compatible with the Apache Kafka protocol, eliminating broker operations and streaming up to 12 GB/s directly into the lakehouse. - [04:49](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=289s) Lakeflow Designer is now generally available as a no-code visual preparation tool that natively compiles workflows into open Apache Spark declarative pipelines rather than proprietary formats. - [12:32](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=752s) Databricks unveiled Genie ZeroOps, an autonomous background operational agent running inside the data plane that detects pipeline anomalies, identifies root causes via Unity Catalog lineage, and validates code fixes on isolated shallow clones.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=0s) Modern AI agents and operational workflows require high-quality data, but enterprise source systems remain fragmented across legacy databases, SaaS platforms, and event streams. - [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) The rapid adoption of AI coding agents increases the risk of operational overload by generating more pipelines on top of already fragile infrastructure. - [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) Databricks is anchoring Lakeflow on open frameworks, formats, and APIs to avoid proprietary vendor lock-in across ingestion, transformation, and orchestration. - [03:37](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=217s) Apache Spark declarative pipelines unify Python and SQL workflows while integrating an open real-time mode engine for millisecond-level streaming. - [04:49](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=289s) Lakeflow Designer eliminates shadow IT by compiling visual analyst workflows directly into standard, version-controlled Spark declarative pipelines. - [05:36](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=336s) Lakeflow Connect features over 100 enterprise connectors that execute as standard Spark declarative pipelines rather than proprietary runtimes. - [06:18](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=378s) Zerobus Ingest allows engineering teams to point existing Kafka producers directly to the lakehouse, avoiding Kafka broker management and resolving small-file issues. - [06:57](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=417s) Lakeflow Jobs modernizes orchestration with serverless execution, pure Python DAG support, and over 50 open-source operators to trigger external systems like Snowflake. - [06:57](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=417s) Spark declarative pipelines process more than 200 trillion rows daily, and Lakeflow Jobs executes roughly 1.7 billion monthly job runs globally. - [08:48](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=528s) Unlike software engineering where failures produce loud errors, data engineering failures are often silent and permanent, corrupting downstream analytical and operational systems. - [10:39](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=639s) General-purpose coding agents cannot safely operate outside the data plane because testing data fixes requires direct access to sensitive production data and end-to-end lineage. - [12:32](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=752s) Genie ZeroOps operates directly in the data plane across a five-phase loop: detection via ML models, lineage assessment, multi-agent root cause analysis, code remediation, and validation on shallow clones. - [14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) Genie ZeroOps correlates related batch job alerts into unified incidents and ranks them based on downstream business criticality using Unity Catalog lineage graphs. - [14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) To verify remediations without production risk, Genie ZeroOps executes proposed fixes on zero-copy shallow clones of production tables before generating pull requests. - [14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) Beyond pipeline breaks, Genie ZeroOps continuously scans datasets for compliance risks, such as exposed PII, and automatically provisions Unity Catalog governance policies to resolve them.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=0s) The Purple Jacket and the “Messy Middle” of Data Engineering
Bilal Aslam takes the stage to enthusiastic applause, kicking things off with a sartorial callback to prior conferences. Two years ago, he wore a distinct purple jacket while talking about Lakeflow, decided against wearing it the following year, and heard unanimous feedback from attendees demanding its return. After thanking earlier speakers—including Mukesh, Arsalan, Ali Ghodsi, and Kenan—for hyping up the future of artificial intelligence and autonomous agents, Bilal acknowledges the anxiety brewing among data practitioners in the audience. Everyone is talking about agents, apps, and real-time operations, but every single one of those systems relies strictly on high-quality data. For data engineers, that means their jobs suddenly became significantly harder.
The fundamental tension lies between where teams want to go and where they must begin. The business destination on the right is straightforward: autonomous agents, automated workflows, and operational intelligence. The starting point on the left is chaotic: raw operational sources scattered across Kafka, Salesforce, NetSuite, and legacy on-premises databases like Oracle and Microsoft SQL Server. The space between them is what Bilal calls the “messy middle”—the grueling, unglamorous data engineering work of ingesting, cleaning, and transforming that raw data into consumable insights. Over decades, organizations have stitched together brittle architectures crammed with dozens of disparate vendor logos, tools, and fragile point-to-point connections. While teams have made peace with this sprawl under the rationalization that “at least it works,” the reality is fraught with severe compromises. Practitioners cannot achieve uniform version control—they can manage their Apache Spark code in Git, but their underlying Kafka infrastructure remains decoupled. Broad governance is virtually unattainable because data reads and writes traverse dozens of disconnected endpoints, and cost-effective scaling fails because isolated layers of the stack scale unpredictably.
Bilal delivers a blunt assessment to the data engineers in the room: “I'm here to tell you that your job and my job as a data engineer is about to get quite a bit harder.”
- **Fragmented source systems:** Production data remains siloed across modern SaaS platforms (Salesforce, NetSuite), event streams (Kafka), and legacy relational on-premises databases (Oracle, SQL Server). - **Core architectural shortcomings:** Disjointed legacy stacks lack universal version control, make enterprise-wide data governance nearly impossible, and suffer from uneven scaling bottlenecks.
---
[02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) The Rise of Coding Agents and Lakeflow's Open Foundation
Bilal points to Genie Code, Databricks’ purpose-built coding agent tailored specifically for data engineering, data science, and machine learning. While the tool delivers immense productivity, it also surfaces an alarming operational reality. In just three months since its release, an astonishing 60% of all Lakeflow pipelines are already being generated directly by Genie Code. Bilal admits that while he loves agentic tools, the sheer velocity at which agents can spin up pipelines poses an existential threat to already fragile data infrastructure: “Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure.” Without architectural intervention, engineering teams will simply be swamped.
Rather than piling new tooling onto existing complexity, Databricks aims to simplify the stack by grounding it entirely in open formats, open frameworks, and open-source tools. Bilal asserts that Lakeflow provides this exact unified foundation, offering native interoperability across hundreds of partner ecosystems and lakehouse tools so teams can build resilient pipelines without vendor lock-in.
```mermaid flowchart LR accTitle: Lakeflow Unified Open Architecture accDescr: Diagram illustrating how Lakeflow replaces fragmented ingestion, transformation, and orchestration tools with unified open frameworks.
subgraph Legacy["Legacy Fragmented Stack"] direction TB L1["SaaS / Kafka / CDC"] --> L2["Custom Code / dbt / Flink"] --> L3["Unmanaged Airflow"] end
subgraph Lakeflow["Lakeflow Unified Open Foundation"] direction TB K1["Lakeflow Connect & Zerobus"] --> K2["Spark Declarative Pipelines\n(Batch, SQL/Python, Real-Time Mode)"] --> K3["Serverless Lakeflow Jobs"] end
Legacy -.->|"Consolidates into"| Lakeflow ```
---
[03:37](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=217s) Unifying ETL: Spark Declarative Pipelines and Real-Time Streaming
Diving into transformation and ETL—the layer where engineering teams sink the vast majority of their time and budget—Bilal tackles the fragmentation between legacy Spark scripts, custom Amazon EMR distributions, and standalone SQL engines. Lakeflow replaces these disparate engines with Apache Spark declarative pipelines. Because these pipelines are open and declarative, developers merely specify the desired end state, and the engine handles execution. This structure also makes them uniquely suited for AI agents like Genie to generate, optimize, and maintain without human friction. Furthermore, declarative pipelines run anywhere, from production clusters straight down to a local laptop.
Spark declarative pipelines bridge two historical divides in data engineering: language fragmentation and latency differences. Where teams previously had to bifurcate workloads—relying on dbt for SQL workflows and raw Spark for Python—declarative pipelines unify SQL and Python in one environment. Simultaneously, teams that adopted Apache Flink specifically to achieve millisecond stream processing can now stay on Spark. Databricks has integrated its open-source "real-time mode" engine directly into Spark declarative pipelines, delivering ultra-low-latency, millisecond-level streaming on an open engine.
Bilal highlights the core value of this unification: “So you can get millisecond streaming on an open framework that runs anywhere.”
---
[04:49](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=289s) Eliminating Shadow IT with Lakeflow Designer
Beyond formal data engineering teams, enterprises routinely struggle with widespread shadow IT. Bilal notes a common organizational imbalance: a company might employ 20 centralized data engineers, yet host hundreds of business analysts independently assembling ad-hoc workflows using visual drag-and-drop tools. These pipelines live on personal laptops, rely on proprietary visual formats, and inevitably break or become completely unmaintainable the moment their creator leaves the company.
To solve this divide without alienating non-technical users, Bilal announces the General Availability (GA) of Lakeflow Designer, a no-code data preparation environment powered natively by Genie. What sets Lakeflow Designer apart from traditional proprietary visual tools is its compilation target. Instead of locking business logic inside proprietary metadata formats, Lakeflow Designer automatically compiles visual workflows into standard Apache Spark declarative pipelines. Analysts obtain the visual ease they require, while data engineers inherit clean, open, version-controlled code that runs anywhere across the enterprise.
Bilal sums up the architectural philosophy: “And by the way, one super cool thing is that Lake Flow Designer doesn't build any proprietary pipelines. It just builds Spark declarative pipelines under the hood.”
---
[05:36](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=336s) Ingestion at Scale: Lakeflow Connect and Zerobus Ingest
Addressing the ingestion boundary, Bilal critiques traditional third-party SaaS ingestion platforms. While they advertise hundreds of pre-built connectors, they typically drop records into proprietary storage formats. Because these systems lack standard interfaces, AI coding agents cannot inspect or manage them, leaving pipelines unobservable, impossible to automate, and detached from unified governance. In response, Lakeflow Connect has expanded its library to more than 100 native connectors, supported by an open-source community ecosystem where engineers can build custom connectors. Crucially, Lakeflow Connect does not introduce custom ingestion runtimes; it executes as standard Spark declarative pipelines under the hood.
For real-time telemetry, data engineers have long depended on Apache Kafka as a high-volume ingestion buffer, despite universally disliking the operational overhead of running Kafka brokers. Databricks introduces Zerobus Ingest, a fully managed service that is 100% wire-compatible with the Kafka protocol. Teams can redirect existing Kafka producer code directly to a Zerobus endpoint without modifying upstream applications. Zerobus Ingest streams data straight into the lakehouse in open formats, eliminating the classic "small files problem" by ingesting millions of small records seamlessly at throughputs reaching 12 gigabytes per second.
Bilal emphasizes how clean this transition is for engineering teams: “So, you can take your Kafka producing code, point it at Zero Bus Ingest, and you can land data into the lakehouse in open formats, ready for AI.”
- **Lakeflow Connect scope:** Over 100 enterprise connectors available, open-source connector tooling for internal engineering teams, and native execution via Spark declarative pipelines. - **Zerobus Ingest capabilities:** Full Kafka API wire compatibility, direct ingestion into lakehouse storage without broker management, eliminates small-file bloat, and handles sustained throughput of 12 GB/sec.
---
[06:57](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=417s) Modernizing Orchestration with Lakeflow Jobs and Production Milestones
The final piece of the legacy stack is the orchestrator, historically dominated by Apache Airflow. Many teams cling to Airflow solely because they want to define Directed Acyclic Graphs (DAGs) in Python. However, self-managed Airflow comes with severe operational liabilities; an internal Databricks customer survey revealed that 80% of organizations run outdated Airflow distributions riddled with unpatched security vulnerabilities and lacking modern orchestration features. Lakeflow Jobs directly addresses this by offering fully serverless orchestration with native support for pure Python DAGs alongside visual point-and-click workflow builders.
Historically, Databricks orchestration was criticized for excelling only at internal Databricks tasks while struggling with external enterprise systems. Bilal announces over 50 open-source integrations and operators for Lakeflow Jobs, enabling unified orchestration across third-party data platforms—including competing warehouses like Snowflake.
Closing out the architectural overview, Bilal shares production metrics demonstrating that Lakeflow is running at immense enterprise scale rather than serving as an experimental prototype:
Bilal underscores the battle-tested nature of the platform: “This is not a brand new product. We've been at this for a while and it's ready for the world's toughest data workloads.”
- **Spark declarative pipelines throughput:** Currently processes over 200 trillion rows of data every single day. - **Lakeflow Jobs execution volume:** Powering roughly 1.7 billion job runs each month across global enterprise environments. - **Serverless compute adoption:** 50% of Databricks customers have opted into serverless compute to eliminate cluster configuration and VPC provisioning. - **Orchestration breadth:** 50+ open-source operators added to Lakeflow Jobs to trigger external systems and platforms like Snowflake.
[08:48](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=528s) The Data Operations Bottleneck: Why Coding Agents Fail
Bilal Aslam points out an uncomfortable irony: making it radically easier to build data pipelines with AI and open stacks actually threatens to make the operational burden far worse. If anyone across an organization can effortlessly spin up pipelines, the inevitable byproduct is more outages, more broken jobs, and more operational chaos. According to a partner survey Aslam cites, data engineering teams already spend more than 50% of their working hours purely on pipeline maintenance, yet organizations still endure up to 60 hours of data downtime every single month. While teams might hope that general-purpose coding agents can inspect API endpoints or table schemas and solve these incidents automatically, Aslam stresses that data engineering is fundamentally different from traditional software engineering.
``` +---------------------------+-----------------------------------+-----------------------------------+ | Dimension | Software Engineering | Data Engineering | +---------------------------+-----------------------------------+-----------------------------------+ | Composition | Code is self-describing | Code + Data | | Testing | Fully deterministic (pass/fail) | Statistical evaluation | | Failure Modes | Loud exceptions, stack traces | Silent and permanent corruption | +---------------------------+-----------------------------------+-----------------------------------+ ```
In software engineering, code is self-describing and tests are deterministic—code either compiles and passes unit tests, or it does not. In data engineering, pipelines are an inseparable union of code and data, meaning tests cannot simply evaluate static logic; they must evaluate statistical distributions across dynamic datasets. Furthermore, software failures announce themselves loudly through explicit exceptions and error traces that allow immediate deployment rollbacks. In data engineering, failures occur silently and permanently, quietly corrupting downstream systems until an executive discovers bad numbers days or weeks later. As Bilal Aslam puts it: “In data engineering, failures are silent and permanent.”
[10:39](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=639s) The Trust Dilemma and the Data Plane Breakthrough
General coding agents stumble across the entire operational lifecycle because they lack the proper context and data plane integration. For incident detection, coding agents lack direct telemetry; feeding them multi-megabyte Spark traces quickly exhausts LLM context windows. For root cause assessment, they lack end-to-end data lineage unless an engineer manually exports catalog graphs out of the system. Even if an agent writes a code fix, verifying that fix presents an even steeper barrier: an engineer cannot simply run a unit test in an isolated sandbox. Validating a data pipeline fix requires running code against actual production data.
This verification requirement creates an acute security and governance dilemma. To identify root causes, an autonomous agent needs read access to production tables. To verify that its fix actually resolves data discrepancies without introducing regressions, the agent needs write access to production data. Aslam polls the audience directly, asking who would willingly hand uncontrolled read and write privileges over their production crown jewels to a third-party coding agent they did not build; not a single hand goes up in the room. Aslam observes that granting arbitrary agents write permissions invites disaster because an untrusted model could easily drop critical tables. The Databricks team realized that operations cannot be solved by an external tool sitting outside the perimeter: “The big realization we came to was that your operations agent actually needs to live in the data plane.” Only the data plane natively encompasses the underlying data, the Unity Catalog lineage graph, and the required governance boundaries.
[12:32](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=752s) Architecture of Genie ZeroOps
To solve operational automation safely within the governance boundary, Aslam introduces Genie ZeroOps, an autonomous background agent designed to place data and AI operations on autopilot. Rather than relying on generic prompt engineering over external logs, Genie ZeroOps embeds directly into the Databricks platform across five distinct operational phases:
```mermaid flowchart LR accTitle: Genie ZeroOps Five-Stage Operational Loop accDescr: Diagram illustrating the five continuous operational stages of Genie ZeroOps within the Databricks data plane.
subgraph Sense["Detection & Assessment"] A["Autonomous ML Models<br/>(Metrics & Logs)"] --> B["Graph Ranking<br/>(Unity Catalog Lineage)"] end
subgraph Investigate["Root Cause & Remediation"] B --> C["Supervisor & Sub-Agents<br/>(Consensus Engine)"] C --> D["Genie Code<br/>(Ticketing & Git PRs)"] end
subgraph Validate["Safe Verification"] D --> E["Shallow Clones<br/>(Isolated Data Branches)"] end ```
- **Detection:** Genie ZeroOps autonomously generates and continuously fine-tunes per-table machine learning models. With native internal access to metrics, runtime events, and compute logs, Databricks already runs tens of thousands of these operational ML models in active production without requiring user-built monitoring plumbing. - **Assessment:** When an anomaly surfaces, the agent performs graph ranking algorithms across Unity Catalog data lineage, systematically traversing upstream dependencies and downstream consumers. - **Root Cause Analysis:** A dedicated supervisor agent coordinates a fleet of specialized sub-agents. These sub-agents research potential failure points along the graph and deliberate until reaching consensus on the most probable root cause. - **Remediation:** Genie ZeroOps collaborates with Genie Code, connecting to version control repositories and enterprise ticketing systems to file and update incident tickets throughout the debugging lifecycle. - **Verification:** Leveraging a decade of open lakehouse engineering, the agent creates shallow clones of production tables—fast, zero-copy metadata branches similar to Lakeflow branching. It executes the proposed remediation inside Databricks' native network and code isolation boundaries, validating output distributions before touching real data.
As Bilal Aslam announces to the audience: “I'm super, super excited to announce Genie ZeroOps, a new background Genie that puts your data and AI operations on autopilot.”
[14:01](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=841s) Live Demo: Autopilot Incident Triage and Autonomous PII Remediation
Transitioning to a live demonstration, Aslam reveals that the Genie ZeroOps interface deliberately avoids looking like a standard observability dashboard packed with flashing red lights, confusing graphs, and alert fatigue. Instead, it mirrors a prioritized email inbox where incidents are ranked by business severity. The system intelligently groups related failures; in one example shown on screen, Genie ZeroOps deduces that 16 distinct failing batch jobs all stem from a single shared upstream incident, collapsing 16 noisy alerts into one actionable thread.
Opening an incident, Aslam examines an issue where an upstream table named `top_fan_voters` suffered an abrupt drop in row counts approximately ten minutes earlier. Aslam emphasizes that engineers do not have to start an investigation from scratch: “And what Genie ZeroOps does is by the time you log in, your background agent has actually done all the thinking for you.” The system walks downstream lineage to determine that `top_fan_voters` feeds a critical executive-facing fan engagement dashboard, automatically elevating the incident priority. Simultaneously, the agent's root-cause sub-agents fan backward along the Directed Acyclic Graph (DAG), bypassing healthy upstream nodes to pinpoint the exact failure inside an upstream `fan_interactions` table.
``` [Upstream Root Cause] [Detected Anomaly] [Downstream Impact] fan_interactions --------(lineage)-------> top_fan_voters --------(lineage)------> Fan Engagement Dashboard (Faulty logic/data) (Sudden row drop) (Critical business asset) | | +---- Genie ZeroOps writes fix +---- Verified on shallow clone & opens Pull Request before PR creation ```
Genie ZeroOps writes a source code fix, but it does not apply it directly to production. After receiving explicit administrative permission, it provisions a shallow clone of the petabyte-scale production table, applies the candidate fix in complete isolation, and verifies that the output produces the expected row counts. With the fix verified against real-world data patterns, the engineer can review the work and open a Git pull request in one click.
Aslam showcases a second capability: autonomous governance monitoring. When granted scan permissions, Genie ZeroOps continuously audits production tables for regulatory and security risks, such as exposed personally identifiable information (PII). In the demo, the agent detects an application accidentally exposing the PII of roughly 1,000 users. It compiles an impact report detailing the specific fields leaked, analyzes downstream table lineage, and generates a ready-to-deploy Unity Catalog governance policy fix rather than an application code patch. Aslam concludes that by automating deep triage, safe verification, and policy enforcement, Genie ZeroOps frees data engineers from endless break-fix maintenance so they can focus on building high-value systems.
## Notable quotes > [08:48](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=528s) In data engineering, failures are silent and permanent. — Bilal Aslam
> [02:21](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=141s) Agents are great, but all agents mean new data, new pipelines, more to build, more to manage on that fragile infrastructure. — Bilal Aslam
> [10:39](https://www.youtube.com/watch?v=0WDGu-IPmZM&t=639s) The big realization we came to was that your operations agent actually needs to live in the data plane. — Bilal Aslam
## Mentioned Bilal Aslam, Mukesh, Arsalan, Ali Ghodsi, Kenan, Databricks, Data + AI Summit, Lakeflow, Lakeflow Designer, Lakeflow Connect, Lakeflow Jobs, Genie, Genie Code, Genie ZeroOps, Zerobus Ingest, Unity Catalog, Apache Spark, Apache Kafka, Apache Flink, Apache Airflow, Salesforce, NetSuite, Oracle, Microsoft SQL Server, Amazon EMR, dbt, Snowflake, Git
## Video description Data engineering is a messy process, and the growth of AI apps and agents is adding more complexity. The data that AI needs is stored all over the enterprise.
Previously, companies had to rely on brittle, expensive pipelines to move this information. But Lakeflow make it easy to turn raw data stored in these systems into insights and action.
At the 2026 Data + AI Summit, Bilal Aslam, Sr. Director of Product Management at Databricks, shared updates to Lakeflow and introduced Genie ZeroOps, an AI agents for data and AI operations.
Learn more: https://www.databricks.com/blog/introducing-genie-zeroops
00:00 —Data engineering and the "messy middle" 02:21 — Simpliyfing the data stack 03:18 — Building an open data foundation for AI with Lakeflow 04:50 — Now available: Lakeflow Designer 05:40 — Lakeflow Connect's growing ecosystem 06:18 — Ditching Kafka with Zerobus Ingest 06:57 — Updates to Lakeflow Jobs 08:47 — Challenges to using AI in data operations 12:34 — Introducing Genie ZeroOps with live demo 17:23 — What Genie ZeroOps means for data engineers
Presenters: Bilal Aslam, Sr. Director of Product Management, Databricks
[Source](https://www.youtube.com/watch?v=0WDGu-IPmZM)