Databricks
Chief Executive Officer Ali Ghodsi warns runaway token costs will become prohibitive within twelve months unless enterprises abandon frontier model worship to ground autonomous agents in proprietary lakehouse context.
AI-assisted summary of Data + AI Summit Keynote 2026 | Day 2 — original source. Check the original for full context.
Source published:
Key ideas
- Databricks introduced Omnigent, an open-source meta-harness abstraction layer that standardizes cross-harness coordination, session handoffs, contextual policy enforcement, and financial guardrails across proprietary frameworks like Claude Code and OpenAI Codex. (18:56)
- The company launched Agent Bricks alongside Unity AI Gateway to centralize enterprise governance over models, agents, and MCP tools, using an 'advisor pattern' and dynamic data tainting to optimize inference costs and prevent privileged data leaks. (40:00)
- Databricks announced Genie ZeroOps for Machine Learning to act as an autonomous primary responder for production model operations, reversing traditional on-call workflows to automatically diagnose, retrain, and queue drifted models for deployment. (102:43)
- Addressing domain-specific agentic workflows, Databricks unveiled CustomerLake for 1-to-1 Infinity Campaigns and Lakewatch, an open agentic security lakehouse accelerated by the acquisition of Panther Labs. (120:00)
Detailed notes
## Background Published on YouTube by Databricks on June 18, 2026, this keynote captures Day 2 of the Data + AI Summit before an audience of over 30,000 data professionals. According to the description, the session covers new open-source innovations, major updates across the Databricks Platform, and autonomous agentic solutions tailored for enterprise marketing and cybersecurity.
## TL;DR Databricks leadership argues that raw model intelligence is no longer the bottleneck in enterprise AI, framing proprietary context, cost control, and avoiding vendor lock-in as the true production hurdles. To bridge fragmented agent frameworks, Databricks unveils Omnigent, an open-source meta-harness that standardizes multi-agent orchestration, contextual security policies, and live collaboration. The presentation introduces Agent Bricks with Unity AI Gateway for intelligent model routing, Databricks Apps to govern enterprise 'vibe coding,' and Genie Code and Genie ZeroOps for autonomous machine learning lifecycle management. The keynote concludes with major domain-specific platforms: CustomerLake, an agentic Customer Data Platform for hyper-personalized marketing, and Lakewatch, an open agentic security lakehouse built in tandem with the acquisition of Panther Labs.
## Atmosphere & style The presentation delivers the polished, high-energy cadence typical of a flagship tech keynote, alternating between visionary executive manifestos and dense, live terminal and UI demonstrations. Speakers maintain an urgent, pragmatic tone, steering clear of passive model hype to focus on the unglamorous mechanics of production deployment—such as token budgets, sandboxing, data tainting, and on-call fatigue. The narrative carries a strong collaborative rhythm as platform leads and engineering founders hand off seamless demonstrations, cultivating a sense of technical momentum and enterprise inevitability around agentic systems.
## Bulletin - [18:56](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1136s) Databricks introduced Omnigent, an open-source meta-harness abstraction layer that standardizes cross-harness coordination, session handoffs, contextual policy enforcement, and financial guardrails across proprietary frameworks like Claude Code and OpenAI Codex. - [40:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2400s) The company launched Agent Bricks alongside Unity AI Gateway to centralize enterprise governance over models, agents, and MCP tools, using an 'advisor pattern' and dynamic data tainting to optimize inference costs and prevent privileged data leaks. - [01:42:43](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6163s) Databricks announced Genie ZeroOps for Machine Learning to act as an autonomous primary responder for production model operations, reversing traditional on-call workflows to automatically diagnose, retrain, and queue drifted models for deployment. - [02:00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7200s) Addressing domain-specific agentic workflows, Databricks unveiled CustomerLake for 1-to-1 Infinity Campaigns and Lakewatch, an open agentic security lakehouse accelerated by the acquisition of Panther Labs.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=0s) Enterprise AI bottlenecks have shifted from raw model intelligence to the challenge of infusing proprietary enterprise context into agent reasoning loops. - [00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=0s) Inference and token consumption are projected to become cost-prohibitive for enterprises within 6 to 12 months without architectural interventions like semantic ontologies and advisor routing patterns. - [11:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=671s) The proliferation of custom, fragmented agent software has created an 'agent harness problem,' where incompatible execution environments cannot share state, collaborate, or enforce unified enterprise controls. - [20:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1200s) Omnigent provides an open-source meta-harness architecture featuring an execution runner and a centralized server to unify CLI, desktop, and web agent interactions across diverse models and tools. - [21:20](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1280s) Omnigent replaces static tool-call permissions with dynamic contextual policies that restrict sensitive external actions based on an agent's prior data access history. - [36:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2171s) Agent Bricks abstracts the 99% of infrastructure work required beyond the core agent loop by providing multi-model choice, Genie Ontology grounding, Agent Memory Services, and ephemeral VM Sandboxes. - [46:51](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2811s) Unity AI Gateway shifts organizations from 'tokenmaxxing' to 'valuemaxxing' through intelligent model routing, centralized MCP skill registries, and runtime data tainting to restrict data egress. - [01:09:16](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4156s) Databricks Apps solves the prototype-to-production gap of enterprise vibe coding by bundling managed authentication, AppSpaces administrative boundaries, and serverless micro-VM runtimes. - [01:37:07](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5827s) AI Runtime delivers serverless, on-demand GPU capacity without upfront commitments, enabling distributed deep learning, fine-tuning, and LLM pre-training natively in the lakehouse. - [01:42:43](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6163s) Genie Code for Machine Learning integrates with team-specific MLflow experiments, feature stores, and validation suites to act as an automated senior ML engineer. - [01:42:43](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6163s) Genie ZeroOps for ML addresses the reality that teams spend 60% to 80% of bandwidth maintaining models, placing an autonomous agent on primary on-call triage while human engineers serve as approvers. - [01:52:48](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6768s) Buyer behavior is transitioning to delegate shopping agents acting over APIs in seconds, rendering traditional waterfall marketing campaigns obsolete. - [02:00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7200s) CustomerLake establishes an agentic CDP natively inside the lakehouse, leveraging Profile Agents for LLM-driven identity resolution and Campaign Agents to execute continuous 1-to-1 Infinity Campaigns. - [02:20:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8400s) Legacy SIEM platforms cannot withstand automated AI cyberattacks due to 30-day retention cutoffs, high licensing fees, and an inability to ingest rich contextual telemetry like code commits and Slack chats. - [02:22:29](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8549s) Lakewatch and the acquisition of Panther Labs merge detection engineering with the open lakehouse, deploying defensive agents that automate OCSF log normalization, detection authoring, and alert triage. - [02:43:40](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9820s) Databricks' overarching strategy anchors the agentic era in open standards across data formats, models, and harnesses, unified under Unity Catalog governance without cloud lock-in.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=0s) Infusing Enterprise Context into AI
Databricks co-founder and CEO Ali Ghodsi takes the stage to kick off Day 2 of the Data + AI Summit, joking with the crowd about the early start: “All right, super excited. Wow, everybody made it 8:00 AM. It was hard for me, actually.” Recapping the core message from Day 1, Ghodsi argues that the industry's obsession with chasing raw model capabilities misses the real enterprise bottleneck. As Ali Ghodsi states: “We don't need more intelligence for the AI, we need more context.” The real objective for Databricks and its customers is infusing the proprietary enterprise context trapped across disparate organizational systems directly into models to unlock autonomous, production-ready AI agents.
To ground this approach, Ghodsi outlines four non-negotiable pillars required to solve the enterprise context equation: 1. **Choice and Openness:** Organizations must retain the freedom to run any model, leverage any cloud or infrastructure, and store data in open file formats. Ghodsi warns that architectural lock-in has been the primary factor handicapping enterprise velocity over the last two to three decades as teams layered proprietary software into bloated stacks. As Ali Ghodsi puts it: “Avoiding lock-in, I think, is one of the most important things that Databricks can help with.” 2. **Control and Governance:** Enterprises require granular security policies, strict access controls, and transparent auditability spanning agents, data, and underlying compute infrastructure. 3. **Cost Controls:** Unchecked model consumption poses an existential risk to widespread deployment. While companies are already feeling the pinch of inference and token usage, Ghodsi cautions that the industry is only scratching the surface: “I think in the next 6 to 12 months, most organizations are going to see that the costs are going up so much that it's actually prohibitive to their companies.” 4. **Context via Genie Ontology:** By continuously computing in the background across an organization’s entire data footprint—reaching beyond the Lakehouse—the Genie ontology builds an ambient institutional index. Ghodsi likens this background synthesis to “a PageRank that Google did, but for the enterprise,” which drastically accelerates agent execution, slashes unnecessary token burn, and elevates response quality.
[03:21](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=201s) Keynote Roadmap: Agent Bricks, Genie Code, Apps, Lakewatch, and CustomerLake
Ghodsi pivots to previewing the major architectural launches and technical deep dives scheduled across the keynote, starting with Agent Bricks presented by Casey and Ankit. Designed for developers building custom agents, Agent Bricks provides isolated virtual machine sandboxes for secure code execution, persistent memory services, and native integration with Omnigent, the open-source meta-harness. A standout architectural pattern demonstrated during this segment pairs Agent Bricks and Omnigent with the Unity AI Gateway to tame token costs using an advisor pattern. Under this setup, a fast, cost-effective open-source model handles primary task execution and invokes a frontier, high-cost model strictly as a fallback tool when it encounters an obstacle.
```mermaid flowchart TD accTitle: Advisor Pattern for Cost-Optimized Agent Execution accDescr: Small open-source model executes tasks directly and calls an expensive frontier model as a tool advisor only when stuck.
subgraph Execution["Agent Runtime"] Task["Task or Code Request"] --> SmallLLM["Cost-Effective Open Model (Primary Agent)"] SmallLLM --> Check{"Agent Stuck?"} end
subgraph Escalation["Advisory Tool"] Check -- "Yes (Escalate)" --> Gateway["Unity AI Gateway"] Gateway --> SmartLLM["Frontier Large Model (Advisor Tool)"] SmartLLM --> Guidance["Structured Hint / Resolution"] Guidance --> SmallLLM end
Check -- "No (Proceed)" --> Output["Task Completed"] ```
The keynote agenda also expands into specialized agentic workloads across data science, application delivery, and domain platforms: - **Genie Code for Machine Learning and ZeroOps:** Building upon its data engineering automation capabilities, Genie Code is expanded into an autonomous data science and machine learning agent. It writes end-to-end ML pipelines, automatically troubleshoots runtime failures, and establishes a self-healing loop that democratizes advanced model development. - **Databricks Apps and the Agent System of Record:** As organizations build custom vibe-coded apps for hundreds of thousands of employees, Databricks Apps introduces governed authentication, data authorization, and enterprise runtime guarantees to bridge the prototype-to-production gap. - **Lakewatch and Panther Integration:** A walkthrough of Lakewatch—the agentic Security Information and Event Management (SIEM) system launched two months prior—covering its expansion and integration with Panther following Databricks’ agreement to acquire the security operations platform. - **CustomerLake:** An educational dive into the Customer Data Platform (CDP) reimagined as an agentic system of record, designed to autonomously orchestrate customer touchpoints at massive scale. - **Omnigent Open Source:** Chief Technologist Matei Zaharia will introduce Omnigent, an open-source meta-harness open-sourced over the weekend that serves as the architectural foundation for Databricks’ own products, including Genie. Teasing Zaharia’s entrance, Ghodsi jokes with the event production team: “And I want a request here for the camera crew: please zoom in on his shoes, okay? And make sure you capture that, because that's going to be a really important part of his talk.”
[06:33](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=393s) Mukesh Ambani on Democratizing AI for 1.5 Billion Indians
Before welcoming Matei Zaharia to unveil Omnigent, Ghodsi introduces a video address from Mukesh Ambani, Chairman and Managing Director of Reliance Industries, underscoring their joint mission to deliver digital sovereignty and affordable AI across the developing world. Ambani explains that high inference costs often price emerging economies out of advanced artificial intelligence, necessitating an architecture centered on high-efficiency open models and scalable data infrastructure.
Addressing the summit delegates, Ambani frames the Reliance-Databricks collaboration as a national modernization initiative: “Today, with Reliance Intelligence, our mission is to democratize intelligence for 1.5 billion Indians. Our motto is simple: AI everywhere for everyone.” Ambani outlines a vision where accessible AI elevates farmers, local neighborhood shopkeepers, healthcare outcomes, and public education across India. Furthermore, he emphasizes that technical architectures engineered to operate under India's demanding cost, scale, and language constraints provide an exportable blueprint for the entire Global South, where four-fifths of humanity lives.
Ambani highlights the technical magnitude of the migration, noting that Reliance has consolidated petabytes of operational data across Jio telecommunications, retail networks, green energy, materials, and media onto a unified Databricks intelligence platform to serve over 800 million customer relationships. Ambani describes this transformation as a fundamental operational shift: “For many years, enterprises used data mainly to look back. The next generation will use data to make better decisions, act faster, and continuously improve. This is the shift from dashboards to decisions.”
- **Reliance Intelligence Scale:** Migrated multi-petabyte datasets across diverse verticals (Jio, retail, energy, materials, and media). - **Consumer Footprint:** Powers real-time decision-making across more than 800 million active customer relationships. - **Global Reach:** Targets population-scale accessibility for 1.5 billion Indian citizens while generating open architectures applicable to the Global South.
[10:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=600s) Reliance and Databricks: Powering Scale in India
Speaking about the deployment of Databricks across Reliance, the presenter highlights that decision-makers are already leveraging Databricks Genie to query enterprise data in natural language and receive real-time, actionable intelligence. This capability is now scaling to thousands of business leaders across the entire organization. India is described as one of the most demanding and rigorous testing environments for artificial intelligence globally, driven by massive scale, diverse languages, demanding affordability constraints, and rapid adoption speed.
As the Reliance speaker reflects on the broader potential of this enterprise deployment: “If an AI solution works for India, it can serve the world.” He expresses gratitude to Databricks CEO and Co-founder Ali Ghodsi for his partnership, friendship, and shared vision, noting that the two companies are establishing a global blueprint for harnessing data and AI responsibly, openly, and at enterprise scale.
[11:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=671s) The Explosion of Coding Agents and the "Harness" Problem
Databricks Co-founder and Chief Technologist Matei Zaharia takes the stage to open the second day of keynotes. He announces a brand-new open-source project released by the Databricks engineering team over the preceding weekend. The initiative addresses recurring architectural roadblocks encountered across the industry as developers build, run, and scale AI agents. While coding agents have achieved widespread developer adoption and agents like Genie prove that specialized agentic workflows succeed, their very ease of creation has sparked a new problem: when custom software is this easy to create, developers naturally attempt to "vibe code" their own solutions rather than relying on standard infrastructure.
Zaharia explains that creating custom software around models leads directly to what he terms the "agent harness" problem. A language model in isolation simply predicts and generates text tokens. The software wrapped around that model—providing the interface to file systems, execution environments, toolsets, user interfaces, and permission models—is the harness. Products such as Claude Code, Codex, and countless custom agent SDKs each introduce their own proprietary harnesses. Because developers often co-design the harness alongside the underlying language model, the ecosystem has exploded with dozens of incompatible harnesses. Zaharia compares shopping for agent frameworks to browsing a pet store: you can pick an agent harness in any color, size, or shape, but none of them talk to one another.
```mermaid flowchart TD accTitle: The AI Agent Harness Architecture accDescr: Diagram contrasting raw language models, proprietary agent harnesses, and external system interfaces.
subgraph Models["Underlying Intelligence"] LLM["Foundation Language Model<br/>(Predicts / Generates Tokens)"] end
subgraph Harness["Agent Harness Layer"] H["Harness Software<br/>(Claude Code, Codex, Custom SDKs)"] UI["User Interface"] SEC["Security & Permissions"] TOOL["Tool & File Connectors"] end
subgraph World["External Environment"] FS["File Systems & Codebases"] DB["Databases & APIs"] USR["Human Developers"] end
LLM <--> H H --- UI H --- SEC H --- TOOL UI <--> USR TOOL <--> FS TOOL <--> DB ```
[13:48](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=828s) Three Challenges of Interoperability: Composition, Collaboration, and Control
Because fast coding cannot resolve systemic architectural divergence, Zaharia outlines three core interoperability challenges that siloed harnesses cannot fix on their own:
1. **Agent Composition:** Engineers rarely stick to a single agent. When troubleshooting complex bugs or distributing tasks, developers frequently query multiple coding agents in parallel, feeding them different prompts and comparing outputs. While running loops or multi-agent workflows inside a single harness is straightforward, coordinating across different harnesses (for instance, bridging Claude Code and Codex) is brittle and fragmented. Zaharia notes emerging architectural patterns such as multi-agent loops and "advisor" frameworks, where lightweight, cost-effective models handle routine volume while calling larger frontier models when blocked. Research from Harvey AI, Anthropic, Databricks Genie, and Zaharia’s own lab at UC Berkeley confirms that mixing multi-size models improves both operating costs and answer quality. 2. **Human-Agent and Agent-Agent Collaboration:** A typical developer desktop has devolved into a chaotic mosaic of terminal windows, Claude Code sessions, Slack channels, and Google Docs, leaving users manually copy-pasting code snippets across isolated harness interfaces. Zaharia questions why developers cannot seamlessly transfer an active coding session—complete with full state and context—directly to another team member when stepping away from their desk. 3. **Enterprise Control and Security:** Agents are vulnerable to prompt injection, cost overruns, and catastrophic operational errors such as accidental database deletions. While transactional lakes like Lakebase can revert accidental table drops, security controls remain fundamentally weak. Zaharia points out the absurdity of current human-in-the-loop authorization dialogs, where models interrupt developers with unreadable prompts asking: “Do you want to run this 100-line Python script? Yes or no.” Expecting human users to scrutinize every underlying agent script provides neither real security nor acceptable developer velocity.
Summarizing the coordination hurdle facing engineering organizations, Matei Zaharia observes: “If everyone is just vibe coding their own thing, they will still be incoherent.”
[18:56](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1136s) Introducing Omnigent: The Open Meta-Harness
To resolve the fragmentation across agent implementations, Databricks recognized the need for an open abstraction layer positioned directly above individual agent harnesses: a "meta-harness." Instead of replacing existing frameworks, the meta-harness creates a unified interface that standardizes agent composition, team collaboration, and policy-driven control.
Zaharia announces the open-source release of **Omnigent**. Designed to accommodate any underlying harness, Omnigent natively integrates with leading coding agents (including recent additions such as Cursor and Antigravity, added right after launch in response to developer demand) as well as custom multi-agent architectures built on tools like the OpenAI Agents SDK. By sitting across varied frameworks, Omnigent enables centralized auditing, seamless session handoffs, cross-agent loops, and robust policy enforcement without locking organizations into a single vendor's agent runtime.
[20:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1200s) Omnigent Architecture: The Runner, the Server, and Uniform Interfaces
Omnigent is designed to solve the fragmentation of agent harnesses by providing an open meta-harness architecture structured around a clean, minimal interface: sending messages in, receiving event streams out, and invoking tools. Under the hood, Omnigent consists of two primary architectural components: a local runner and an optional centralized server. The runner acts as a lightweight wrapper around any underlying agent harness. It sandboxes execution to enforce system-level security constraints, continuously monitors agent processes, and ensures runtime stability without requiring proprietary infrastructure.
```mermaid flowchart TD accTitle: Omnigent Architecture Overview accDescr: Diagram illustrating how agents run inside an Omnigent Runner and connect to an optional Omnigent Server with multi-modal interfaces.
subgraph Execution ["Agent Execution Layer"] A["Agent (Claude / Codex / Custom)"] R["Omnigent Runner\n(Sandboxing & Monitoring)"] A <--> R end
subgraph Server ["Omnigent Server (Postgres, Docker, OpenTelemetry)"] H["Central Session History"] C["Agent Catalog & Sharing"] M["Central MCPs & Skills"] P["Security & Cost Policies"] end
subgraph Interfaces ["Uniform Access Interfaces"] T["Terminal CLI"] W["Web UI"] N["Native App / Mobile"] API["REST API"] end
R <-->|"State, Events & Policies"| Server R --> T Server --> W Server --> N Server --> API ```
Connecting the runner to an Omnigent server unlocks enterprise-grade centralization and governance. The server maintains a unified session history across all agent interactions, provides a catalog for publishing and sharing agents across teams, standardizes Model Context Protocol (MCP) integrations and agent skills, and distributes centralized artifacts and security policies. Crucially, regardless of which underlying harness or runtime is used, Omnigent abstracts the presentation tier into a uniform interface: developers retain their native terminal CLI experience, while simultaneously gaining access through a web browser, a native desktop application, mobile clients, and a programmable REST API. The entire system is fully open source, requires no dependencies on Databricks, and is built on standard commodity infrastructure including Docker, PostgreSQL, and OpenTelemetry to support deployments ranging from individual developer laptops to large-scale enterprise clusters.
- The core Omnigent interface standardizes message inputs, event stream outputs, and tool-calling primitives across disparate agent runtimes. - The server stack relies on open-source, non-proprietary standards (Docker containers, PostgreSQL database, OpenTelemetry tracing). - Developers do not sacrifice their terminal workflows; Omnigent layers Web, native desktop, mobile, and REST interfaces on top of existing command-line interfaces.
---
[21:20](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1280s) Composition, Real-Time Collaboration, and Contextual Governance
Standardizing agent runtimes into this uniform meta-harness layer unlocks three fundamental capabilities: composition, collaboration, and control. For composition, engineers can combine heterogeneous agents across different harnesses and reinforcement-learning-tuned models (RLMs) using simple YAML declarative definitions. Developers can seamlessly swap the underlying harness of an active, running agent session mid-flight without losing execution state or history. For collaboration, Omnigent introduces live multi-user sessions, enabling teammates to jump directly into an ongoing agent workspace, converse with the agent, inspect diffs, and comment on modified files simultaneously in real time.
Addressing the third pillar—control—the speaker highlights how conventional agent tooling relies on crude, static tool-level permissions that merely approve, deny, or prompt the user per invocation. Omnigent replaces this with state-aware, contextual policy enforcement where runtime permissions dynamically adjust based on the agent's prior actions and data access patterns:
> “Today in most agent tools, the control is just for each type of tool call. Do I allow it or deny it or ask the user? They're very static. We support contextual policies where the permissions vary based on what the agent did before.”
Under this model, if an agent inspects confidential internal files, the policy engine automatically revokes its ability to call external communication tools, such as sending external emails; conversely, if the agent has accessed only public marketing collateral, outbound communications remain authorized. Omnigent also introduces strict financial guardrails—a capability missing from existing agent frameworks—allowing operators to set hard expenditure limits (such as capping a session's API consumption at $5.00), paired with an OS-level sandbox to prevent malicious or accidental host system damage.
---
[23:20](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1400s) Demo: Wrapping Claude with Omnigent and Bridging Terminal with Web UI
Moving to a live demonstration, the presenter showcases Omnigent running between a remote development machine inside Databricks and a local laptop. Wrapping an existing agent requires nothing more than prepending the CLI command with `omni`, demonstrated via `omni claude`. The runner initializes, loading Databricks' internal configuration of Claude Code, pre-wired with internal Databricks MCP servers and custom prompt behaviors—including insider Databricks terminology like “archiving.” When prompted in the terminal to explain the current workspace repository (`What is this project?`), Claude parses the repository and responds directly within the terminal interface.
```mermaid flowchart LR accTitle: Bi-directional UI Synchronization in Omnigent accDescr: Diagram showing bi-directional synchronization of inputs and outputs between the terminal CLI and the Omnigent Web UI.
U1["Developer (Terminal CLI)"] <-->|"omni runner"| Session["Omnigent Active Session State"] U2["Developer (Web Browser / Native App)"] <-->|"WebSockets / API"| Session Session <-->|"Tool Calls / File Edits"| Project["Project Workspace (README.md)"] ```
While running an agent in the terminal is familiar, Omnigent simultaneously exposes that exact session inside a web UI. The browser view renders a rich chat interface, an embedded live mirror of the terminal output, and a visual file explorer showing modified assets and diffs. Demonstrating cross-surface synchronization, the presenter submits a prompt from the web interface instructing the agent to add a joke to the top of the project's `README.md`. The input instantly pipes into the running terminal process, which executes the edit.
Inspecting the changes through the web UI's markdown visualizer, the presenter highlights how developers can review diffs, leave line-by-line comments (e.g., requesting the removal of previous jokes and keeping just one), and instruct the agent to address all visual review feedback in a single batch. To circumvent the friction of losing agent sessions inside cluttered browser tabs, the presenter also demonstrates Omnigent's dedicated native desktop application, which provides persistent access, desktop notifications, and focused window management.
---
[26:35](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1595s) Switching Harnesses Mid-Session and Orchestrating Multi-Agent Workflows with Poly
A central capability of Omnigent is the ability to fork an ongoing session and switch harnesses without sacrificing context. Demonstrating this, the presenter forks the active Claude session directly from the interface and switches the runner over to OpenAI Codex. Codex spins up in the terminal, instantly inheriting the complete conversational history and file context established by Claude. The presenter prompts Codex to add a second joke to the `README.md`, showing how a developer who gets stuck on a complex coding problem with one model or harness can switch to another without restarting from scratch.
```mermaid flowchart TD accTitle: Poly Multi-Agent Supervisor Workflow accDescr: Flowchart illustrating the Poly multi-agent supervisor orchestrating parallel worker agents to evaluate project tasks.
Supervisor["Poly Supervisor Agent"] Task["Evaluate & Rate README Jokes"] Supervisor --> Task
subgraph ParallelWorkers ["Parallel Sub-Agent Execution"] Worker1["Claude Code Worker\n(Runs evaluation)"] Worker2["Codex Worker\n(Runs evaluation)"] end
Task --> Worker1 Task --> Worker2
Worker1 -->|"Rating & Critiques"| Aggregator["Supervisor Aggregation"] Worker2 -->|"Rating & Critiques"| Aggregator Aggregator --> Output["Consolidated Verdict:\nJoke 1 vs. Joke 2"] ```
Beyond point-to-point harness switching, Omnigent natively facilitates multi-agent hierarchical composition. The presenter launches Poly, an included open-source multi-agent coding supervisor that orchestrates multiple autonomous agents across distinct harnesses. Executing Poly in the same project directory, the presenter tasks it with evaluating the jokes added to the documentation:
> “So we actually include an example here, Poly, which is a multi-agent coding supervisor that can use both Claude and Codex to do subtests.”
Poly automatically inspects the environment to verify whether Claude and Codex are installed, spins up two dedicated worker processes in parallel—one running Claude Code and the other running Codex—and assigns each worker the task of reviewing and grading the jokes. The developer can inspect each worker's discrete terminal stream in real time as they perform their analysis. Poly's supervisor layer synthesizes their independent evaluations, reporting back a comparative judgment (rating the first joke as solid, and the second as inferior). The presenter notes that this pattern extends far beyond simple reviews—such as parallelizing the triage of open GitHub issues and verifying in codebase history whether bugs have been patched—before pivoting to demonstrate real-time engineer collaboration.
- Poly automatically detects available local CLI agents (such as Claude Code and Codex) and spawns isolated sub-agents as background workers. - Developers can monitor both high-level supervisory summaries and raw, per-worker terminal execution trees simultaneously. - Composition enables practical parallelization of developer tasks, including bug investigation, PR reviews, and multi-model consensus evaluations.
[30:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=1800s) Live Collaboration, Stateful Policies, and Open-Sourcing Omnigent
Continuing his live demonstration of Omnigent, Matei Zaharia simulates a collaborative debugging session by inviting Ankit, an engineer on the Databricks AI team, into his workspace. Zaharia rings Ankit on a live call, catching him off guard. Ankit answers with mock exasperation: “Oh man, Matei’s calling me on vacation?” Matei asks him to review an update to a repository's `README` file: “I know the agent said that joke one is better, but I kind of like joke two.” With session-sharing enabled, Ankit immediately views the active session, leaves comments, and issues direct prompts to the running agent, demonstrating how teams can seamlessly hand off complex agent workflows. Zaharia emphasizes that because every session and full interaction history is retained on the user's server, teams can retroactively inspect and analyze all agent activity.
Turning to governance, Zaharia highlights Omnigent’s built-in stateful policy controls, tracking per-agent token spending in real time. Because these policies are contextual and evaluate current runtime state, developers and centralized administrators can establish programmatic constraints, such as restricting an agent from spending more than a specified budget threshold without human approval. To demonstrate, Zaharia inputs a mock condition prompting for confirmation if spending exceeds $0.20 after an agent has already consumed $0.25. When he commands it to execute a task requiring extensive file reading, the agent pauses mid-execution and asks for user permission. Zaharia approves it, showing that the system will execute until reaching an absolute $5 ceiling, where it shuts off completely:
> “So this is a really cool feature, again, on top of any agent, Claude, Codex, whatever it is, and it's just scratching the surface.”
```mermaid flowchart TD accTitle: Omnigent Stateful Policy and Collaboration Flow accDescr: Flow diagram showing user collaboration, agent execution with stateful budget checkpoints, and centralized policy enforcement. subgraph Collaboration ["Live Multi-User Workspace"] U1["Primary Developer (Matei)"] <-->|"Shares session"| U2["Reviewer (Ankit)"] U1 & U2 -->|"Prompt & instruct"| AG["Running Agent"] end subgraph RuntimeGovernance ["Stateful Policy Engine"] AG -->|"Execute tool calls"| POL{"Cost > Budget Policy?"} POL -->|"Exceeds threshold"| HIL["Pause & Request Approval"] HIL -->|"Approved"| EXEC["Resume Execution"] POL -->|"Hard limit reached"| STOP["Halt Agent Run"] end ```
Matei Zaharia explains that Databricks chose to open source Omnigent because agent development must move above the silo of individual software development kits (SDKs) and harnesses:
> “We think this meta harness layer is going to be needed, and it will benefit from being open the same way that data formats and sharing protocols and stuff like that benefit from being open, because it's an ecosystem thing where people sort of have to agree on a way to do it, to make it work well.”
The project has already drawn significant community contributions since its launch. Beyond programmatic workflows, Omnigent provides a complete open-source collaborative user interface—including mobile responsiveness—eliminating the need for teams to build bespoke frontends for internal agent tooling. Zaharia hints at upcoming ecosystem extensions and automated agent optimization capabilities, wrapping up by welcoming attendees to adopt Omnigent.
- **Collaborative features**: Full session-state sharing, inline commentary, and live handoffs to team members. - **Stateful policy engine**: Enforces dynamic spending limits (e.g., pause at $0.20 for human-in-the-loop signoff; hard kill at $5.00). - **Backend persistence**: Complete session and trace histories stored on-premises or on private enterprise servers. - **Cross-model and cross-harness support**: Works agnostically across Claude, OpenAI Codex, and other backends.
---
[34:44](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2084s) Databricks Free Edition: Bringing Enterprise Tooling to Learners
A recorded transition video outlines the adoption and educational scope of the Databricks Free Edition. The platform serves hundreds of thousands of users seeking hands-on mastery of production data and AI skills, ranging from agent design to biomedical literature synthesis and fragmented environmental data unification.
Educators and industry professionals spotlight how providing university students access to tools like Lakeflow Designer on the Free Edition bridges the traditional divide between academic instruction and enterprise deployment. When incoming hires possess hands-on familiarity with the platform, they bypass the onboarding learning curve and address core business challenges from their first week on the job. The program offers learners free, permanent access to an authentic production environment.
- **Target audience**: Students, researchers, and transitioning professionals developing enterprise data and AI skills. - **Core capabilities**: Hands-on access to production Databricks tooling, including Lakeflow Designer. - **Pricing tier**: Free indefinitely for registered learners.
---
[36:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2171s) Agent Bricks: Overcoming the Infrastructure Hurdle in Enterprise AI
Casey Uhlenhut, Director of Product Management at Databricks, takes the stage to shift the keynote's focus toward enterprise builders. While the summit's opening day explored consuming AI through interfaces like Databricks Genie, day two is aimed directly at developers constructing mission-critical autonomous agents.
Uhlenhut reviews the growth of Agent Bricks, Databricks' agent developer platform launched one year prior. Since debut, developers have built over 100,000 custom agents on the platform, MLflow GenAI experimentation and tracing has exceeded one billion downloads, and the Databricks AI Gateway has processed more than one quadrillion tokens. Enterprise adoption includes: - **7-Eleven**: Accelerated maintenance and repair workflows. - **MasterCard**: High-throughput fraud detection across billions of financial transactions. - **AstraZeneca**: Accelerated extraction of commercial insights for drug discovery teams.
Despite these milestones, Uhlenhut cautions that standard industry demonstrations misrepresent the difficulty of production agent engineering:
> “When you see agent demos and agent pilots, you're usually just seeing the agent loop, but the agent loop is only 1% of the work that you have to do. It's the other 99% that's gonna get you.”
```mermaid flowchart LR accTitle: Production Agent Development Breakdown accDescr: Diagram contrasting the 1 percent agent loop against the 99 percent infrastructure iceberg. subgraph Pilot ["1% Visible Work"] LP["Agent Loop / Prompting"] end subgraph Production ["99% Enterprise Infrastructure"] direction TB AUTH["Authentication & Identity"] MCP["Model Context Protocol (MCP) Integrations"] EXEC["Long-Running Reliability & Timeout Handling"] GOV["Context Retrieval & Budget Guardrails"] end Pilot -.->|"Deceptively simple"| Production ```
Developers tasked with building hundreds or thousands of enterprise agents inevitably become bogged down building foundational infrastructure rather than optimizing applications. Uhlenhut identifies the three primary bottlenecks facing these engineering teams: 1. **The Rapidly Shifting Frontier**: Models and agent harnesses evolve weekly, rendering six-month-old architectures obsolete and forcing teams to replatform their entire software stack to stay competitive. 2. **Context Fragmentation**: Competitive differentiation relies on corporate data, yet that context is scattered across disparate formats and silos—from unstructured PDF documents to external REST APIs—making accurate context selection difficult. 3. **Privileged Access Risks and Runaway Costs**: Because enterprise agents can execute mutations and access private records, they represent the most privileged actors in an organization. Developers face risks of accidental data leakage or catastrophic production database deletions, while unchecked execution loops risk exhausting cloud budgets before an application ever reaches deployment.
[40:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2400s) Agent Bricks: Unlocking Developer Choice Across Frontier and Custom Models
To keep engineers from getting trapped in the weeds of spinning up and managing bespoke infrastructure, Databricks is introducing Agent Bricks—a dedicated agent developer platform anchored on three core pillars: choice, context, and control. As the speaker explains, “Agent Bricks is our developer platform where we provide answers to all of these so that developers can focus on building agents, rather than on building infrastructure.” The first pillar, choice, ensures that developers are never locked into a single ecosystem. Agent Bricks provides out-of-the-box, native access to every leading frontier model, including Claude Opus, OpenAI's GPT series, Google Gemini, Alibaba's Qwen, Moonshot AI's Kimi, and, announced in partnership with SpaceX and xAI, the Grok model family.
Frontier models alone, however, cannot solve every enterprise challenge. Certain specialized or domain-critical tasks require tailored reinforcement learning (RL). Agent Bricks includes a complete, built-in model platform enabling organizations to train, serve, and deploy custom RL models optimized for their unique workflows right alongside frontier endpoints. Highlighting this capability, the speaker points out that Databricks’ internal Genie research team used this exact RL infrastructure to build custom models that actually outperformed Opus on their specific benchmarks.
Beyond model endpoints, choice extends directly to agent authoring frameworks and meta-harnesses. Developers can build and deploy agents using Databricks' newly open-sourced Omnigent harness, or stick with popular external frameworks like LangGraph and the OpenAI Agent SDK. To prevent painful engineering rewrites every time a new benchmark leader emerges, the platform exposes a single, unified interface across all models and agents. In the words of the speaker: “So we believe choice is about giving you the flexibility to stay at the cutting edge without having to re-platform your solution every time, and we're able to do this by giving you a unified interface across all of your models so that you can easily switch amongst them.”
- Native model access spanning Claude Opus, GPT, Gemini, Qwen, Kimi, and the newly announced Grok family. - Built-in reinforcement learning platform for training task-specific models that can run head-to-head with or beat top commercial models. - Native support for Omnigent, LangGraph, OpenAI Agent SDK, and custom agent harnesses. - A unified multi-model API eliminating the need to re-platform when switching underlying LLMs.
---
[42:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2531s) Solving the Context Problem: Unity Catalog, Document Intelligence, and Agent Memory Services
The second pillar of Agent Bricks is context—the information that grounds an agent in an organization's business, assets, and users. The speaker frames this plainly: “The best model in the world, though, is only as good as the information you give it, which takes us to context.” In Agent Bricks, all enterprise context is centrally cataloged and governed within Unity Catalog. Everything from tabular datasets and dashboards to external APIs becomes discoverable and callable as tools by agents. To solve the discovery and relevance challenge, the Genie Ontology acts as a rich semantic metadata layer sitting across all agentic assets, allowing agents to accurately retrieve and select exactly the right data without hallucinating irrelevant context.
```mermaid flowchart TD accTitle: Agent Bricks Context Layer accDescr: Diagram showing how enterprise context flows through Unity Catalog and specialized services to ground custom agents.
subgraph DataSources["Enterprise Context & Tools"] UC["Unity Catalog\n(Tables, Dashboards & APIs)"] DOCS["Unstructured Documents\n(PDF Contracts, Invoices)"] EXT["External APIs\n(Managed MCPs)"] end
subgraph Processing["Context Optimization Services"] ONT["Genie Ontology\n(Semantic Metadata Layer)"] DI["Document Intelligence\n(Parsing & Extraction)"] MEM["Agent Memory Services\n(Preferences, Sessions, History)"] end
subgraph AgentExec["Agent Runtime"] AGENT["Enterprise Agent\n(Omnigent, LangGraph, SDKs)"] end
UC --> ONT --> AGENT DOCS --> DI --> AGENT EXT --> AGENT MEM <--> AGENT ```
Handling unstructured enterprise data requires specialized tooling. Organizations frequently possess millions of complex invoices, contracts, and scanned PDFs that standard parsers mangle. Databricks' Document Intelligence provides purpose-built models for classification, parsing, and extraction engineered specifically for optimal placement on the quality-cost curve so teams do not go broke ingesting massive backlogs of files. Furthermore, for data residing outside Databricks, managed Model Context Protocol (MCP) integrations provide out-of-the-box authentication and secure connectivity to external APIs.
Finally, to support personalized user experiences without requiring developers to spawn independent, hard-coded agent instances for every single employee or customer, Databricks announced Agent Memory Services. These managed APIs store user preferences, conversational state, and long-term session histories. Addressing the sheer engineering lift required to build reliable memory, retrieval, and search, the speaker emphasizes: “Context is probably the most difficult problem that exists in the agent space today, which is why we have an entire research team dedicated to just solving the hardest problems here.” Innovations produced by this research team are continuously piped into both Agent Bricks and Genie.
- Context assets (structured tables, dashboards, external APIs) are registered, governed, and callable as tools through Unity Catalog. - Genie Ontology provides a semantic metadata layer so agents selectively retrieve the right context rather than drowning in irrelevant data. - Document Intelligence features custom parsing, extraction, and classification models tailored for complex PDF contracts and invoices. - Managed external MCP connectors handle authentication and data integration for systems outside Databricks. - Agent Memory Services provide managed APIs for storing long-term user preferences, multi-turn conversation histories, and persistent sessions.
---
[45:18](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2718s) Control via Databricks Sandbox: Safe, High-Speed Code Execution
The third pillar of Agent Bricks is control. As agents evolve from simple chat interfaces into autonomous systems that generate and run code, the risk of an unconstrained model running amok in enterprise systems becomes a major blocker. Developers often find themselves caught between wanting agents to write code to solve novel problems and fearing that an agent might accidentally wipe out an active database. Acknowledging this tension, the speaker jokes, “They can do like amazingly creative things if you just kind of let them cook, but you don't want them to take down production.”
To provide a secure, bounded execution environment, Databricks announced Databricks Sandbox. Databricks Sandbox provides ephemeral virtual machines where agents can safely execute generated code without any danger of contaminating or corrupting production environments. Unlike generic cloud virtual machines or container environments, Databricks Sandbox is natively wired into the data layer. It provides instant, downscoped access to enterprise datasets using OnBehalfOf permissions, ensuring that the running code operates strictly within the security clearance of the initiating user or identity. Crucially for autonomous workflows where agents iterate through trial-and-error loops, these sandboxes launch in mere milliseconds, allowing agents to spin environments up, execute analytical scripts, and tear them down on the fly without latency penalties.
- Instant-launch virtual machines designed for secure, isolated agentic code execution. - Native integration with enterprise data using downscoped OnBehalfOf permissions. - Millisecond startup and teardown times matched to rapid, multi-step agent reasoning loops. - Complete isolation from production infrastructure, eliminating risks of unintended system destruction or data corruption.
---
[46:51](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2811s) Unity AI Gateway: Unified Governance, Agent Registry, and Contextual Security Policies
Building toward what the speaker calls the “pièce de résistance” of the platform, Databricks introduced major capabilities in Unity AI Gateway. As the speaker explains: “Unity AI Gateway is our centralized governance, security, and cost control solution that allows you to unify governance across models, agents, MCPs, and skills.” It provides an operational control plane designed to tame sprawling, shadow AI development across large organizations through five key capabilities.
```mermaid flowchart LR accTitle: Contextual Security Policy Dynamic Tainting accDescr: Flowchart illustrating how accessing sensitive PII data dynamically taints an agent's session state and limits permitted actions.
subgraph State1["Public Data Access"] A1["Agent queries public web / docs"] --> B1["State: Untainted"] B1 --> C1["Action: Can post to public web & send email"] end
subgraph State2["Sensitive Data Access"] A2["Agent queries customer PII table"] --> B2["State: Tainted"] B2 --> C2["Action: Block public publishing"] B2 --> D2["Action: Allow internal email only"] end ```
The first capability addresses enterprise sprawl: the Agent Registry. Recognizing that hundreds or thousands of agents are being built across business units using mismatched tools, the Agent Registry acts as a single centralized system of record. Teams can register internal Databricks agents, external agents, MCP tools, and discrete agentic skills into one searchable inventory. The second capability introduces dynamic, contextual security policies based on data tainting. The speaker illustrates this with a concrete scenario: if an agent interacts solely with public data, it is allowed to publish its findings to the public web. However, the moment that same agent accesses sensitive customer data containing personally identifiable information (PII), its state is dynamically tagged as "tainted." From that point forward, the gateway alters its permitted actions—it may still be allowed to draft an internal email to a colleague, but any attempt to publish data to an external or public destination is strictly blocked.
The speaker highlights why this dynamic policy engine is distinct to the platform: “And this is only possible on Agent Bricks, because our agent platform is built and integrated into our data platform.” In addition to contextual tainting, Databricks has partnered with leading security vendors to embed turnkey security guardrails directly into the gateway. Finally, the speaker turns to the pervasive issue of runaway inference budgets—humorously labeled “tokenmaxxing”—where unconstrained iterative loops drain project budgets before applications ever reach production, setting up the gateway's advanced cost-control and rate-limiting features.
- Centralized Agent Registry functioning as the single system of record for all enterprise agents, external agents, MCP endpoints, and skills. - Stateful, contextual security policies that dynamically alter agent permissions (e.g., blocking public egress after touching sensitive PII). - Deep integration between data catalog governance and model runtime security. - Security partnerships offering pre-configured behavioral and content guardrails. - Centralized cost controls, monitoring, and budget boundaries to curb runaway token consumption.
[50:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3000s) Unity AI Gateway: Budgets, Intelligent Routing, and Agent Tracing
Casey expands on the governance and economic challenges enterprises face when managing autonomous agents. With Unity AI Gateway, organizations can move beyond simple post-hoc spend tracking to proactively enforce spend limits for individual users and entire teams through dedicated budgets. However, simply knowing how much has been spent is not enough; enterprise operators need automated, intelligent cost control so engineers and administrators do not have to micromanage every token.
```mermaid flowchart TD accTitle: Unity AI Gateway Intelligent Routing Architecture accDescr: Flow showing incoming prompts routed by complexity across model tiers, multi-cloud fallbacks, and logging into Unity Catalog.
subgraph Client["Developer / Agent Layer"] A["Developer or Agent Prompt"] end
subgraph Gateway["Unity AI Gateway"] B{"Intelligent Routing Engine"} C["Cheap / Fast Models\n(Trivial Tasks)"] D["Expensive / Powerful Models\n(Complex Tasks)"] E["Multi-Cloud & Multi-Provider\nFallback Protection"] end
subgraph Storage["Governance Layer"] F["MLflow Tracing Engine"] G[("Unity Catalog Trace Table")] H["Genie Analysis & Debugging"] I["Lakewatch SIEM / Alerting"] end
A --> B B -->|"Low Complexity"| C B -->|"High Complexity"| D B -.->|"Provider Outage"| E C --> F D --> F F --> G G --> H G --> I ```
To solve this, Unity AI Gateway introduces intelligent routing. The gateway evaluates the contextual complexity of incoming requests, automatically dispatching lightweight, straightforward questions to inexpensive, high-speed models while reserving costly, highly capable frontier models for deeply complex tasks. Because the architecture operates as a multi-cloud and multi-AI layer, smart routing also grants resilience: organizations can establish dynamic fallback chains spanning both distinct model providers and underlying cloud infrastructures to ensure high availability.
Finally, the gateway integrates full agent tracing powered by MLflow tracing. The system captures the full execution graph—every input, every output, and intermediate agent reasoning steps—and centrally records them into a customer-owned table managed within Unity Catalog. Because trace logs contain proprietary context and enterprise secrets, storing them in Unity Catalog allows enterprises to maintain complete data ownership. Once stored, teams can query these traces using Databricks Genie to diagnose failures or connect them directly to LakeWatch (Databricks' SIEM offering) to catch data leakage and trigger incident alerts in real time. Casey emphasizes:
> “So this is what AI governance should look like in the enterprise, where all of your assets can be centrally managed, you can have contextual policies to enforce security, you have an open ecosystem where you can connect to partners, and you can have the cost controls that you need.”
- **Centralized Asset Management**: Unifies tools, models, and agents under cohesive governance rules. - **Enterprise Observability**: Automatically persists granular MLflow traces into governed Unity Catalog Delta tables. - **Security & Incident Response**: Integrates directly with LakeWatch for automated real-time threat detection and leakage alerting.
---
[52:37](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3157s) Demo: Developer Coding Agents, Smart Budgets, and Token Economics
Ankit Mathur, a software engineer at Databricks, takes the stage after returning from vacation to demonstrate how Unity AI Gateway manages real-world developer workloads. Addressing the audience as an active practitioner, Mathur points out that engineering organizations are experiencing explosive cost growth from autonomous and interactive AI assistants. The top 1% of AI spenders now incur costs reaching $90,000 per user per year. Cost escalations are no longer confined to human software engineers drafting isolated scripts; employees across an entire company now trigger multiple agents running in parallel, which execute persistently as long as compute and API access remain unchecked.
To demonstrate how the gateway reclaims control, Mathur sets up his terminal environment using standard tools like Codex, simply pointing the tool’s endpoint directly at the Unity AI Gateway. As the session launches, inference requests and Model Context Protocol (MCP) tool calls are immediately funneled through a centralized governance choke point. MCP integrations authenticate once through a central registry rather than requiring fragmented credentials per machine.
```mermaid flowchart LR accTitle: Unified AI Gateway Developer Budgeting accDescr: Flow from local developer terminal to Unity AI Gateway showing real-time budget tracking and gradual model downgrade policies.
subgraph Terminal["Developer Environment"] A["Terminal / Codex CLI"] end
subgraph Gateway["Unity AI Gateway Layer"] B["Unified Central Choke Point\n(Single-Sign-On MCPs)"] C{"Real-Time Spend Tracker\n($50/day Budget Limit)"} D["Standard / Frontier Models\n(< Threshold)"] E["Token-Efficient Guidance\n(e.g., Databricks Kimi)"] F["Strict Policy Enforcement\n(At 100% Cap)"] end
A --> B B --> C C -->|"Normal Spend"| D C -->|"Approaching Limit"| E C -->|"Exhausted Limit"| F ```
Mathur walks through an initial developer query asking for a coffee shop near Moscone Center. The request executes transparently, but behind the scenes, the gateway aggregates inference into a cross-tool budget. In Mathur’s case, his profile is bound to a $50-per-day allowance. Instead of employing blunt, productivity-killing hard cutoffs, the gateway enforces nuanced, gradual policies. As a developer nears their ceiling, the gateway begins issuing contextual warnings and guiding them toward token-efficient alternatives, such as the newly released Databricks Kimi model, which operates at a fraction of standard frontier pricing. Active sessions are allowed to finish gracefully to prevent breaking developer workflows, but new sessions automatically default to high-efficiency alternatives based on real-time burn rates.
Mathur captures the scale of the developer spend dilemma when setting up the demonstration:
> “The top 1% of spenders are spending $90,000 per user per year, and that exponential isn’t showing signs of slowing down.”
- **Cross-Tool Budgeting**: Consolidates developer consumption across distinct command-line interfaces and editors under a unified policy. - **Graceful Soft-Throttling**: Prevents sudden workflow interruptions by advising model transitions during active work and applying restrictions to subsequent sessions. - **Centralized MCP Authentication**: Single-point authorization eliminates distributed API token leaks across local machines.
---
[55:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3300s) Omnigent Meta-Harness Routing and Unity Catalog Tracing Analytics
Mathur advances the demonstration by coupling the Unity AI Gateway with Omnigent, the open-source agent meta-harness. When an agentic system acts as a meta-harness, it operates at a layer above isolated individual tasks, allowing it to deconstruct complex user intents and make multi-tiered model routing decisions across sub-agents.
When Mathur issues a simple local recommendation request, Omnigent recognizes the query's low complexity, directing it through the gateway to an inexpensive model that returns an answer almost instantaneously with minimal latency. However, when he prompts it with a heavy software engineering task—instructing it to locate the top four bugs in the open-source Omnigent repository—the meta-harness partitions the objective and routes each sub-task to the most cost-effective tier: 1. **Distribution & Orchestration**: A balanced, medium-quality model splits the work. 2. **Repository Surface Scan**: A massive token-consumption task routed to Databricks Kimi to maintain low latency and rock-bottom token cost. 3. **Edge Case Review**: Dispatched to Gemini 3.5 Flash for high reasoning density at low price points. 4. **Core Logic & Deep Auditing**: Assigned to premier, expensive frontier models where code correctness is mission-critical.
Each delegated task runs in parallel inside isolated Databricks Sandboxes, preserving enterprise security boundaries while multiple sub-agents execute simultaneously.
Mathur points out that optimization requires measurement, which is enabled by routing all agent traces directly into a centralized Unity Catalog trace table. Because these logs can expose sensitive corporate data, access is protected using standard Databricks governance mechanisms, including Attribute-Based Access Control (ABAC) and governed tags. With all trace data centralized, engineering teams can use Genie to interrogate past sessions. Mathur reveals that an audit of his own team's developer traces before adopting the gateway showed that 90% of sessions had defaulted to top-tier models like Claude Opus for trivial prompts like "hi" or standard boilerplate tasks. Centralized tracing provides the exact data needed to pinpoint which developer workflows should be routed to cheaper models and which paved paths should be formalized into reusable, catalog-governed agent skills. Mathur observes:
> “Because without data, we can’t be data driven. And today, coding agent sessions aren’t centralized to a single place.”
- **Task-Specific Decomposition**: Dispatches repository-wide scans to high-throughput budget models while routing critical logic audits to frontier reasoning engines. - **Databricks Sandbox Execution**: Parallel sub-agents execute code in secure, isolated VMs to eliminate environmental security risks. - **Enterprise Trace Governance**: Protects sensitive agent interactions using Unity Catalog ABAC policies and governed classification tags. - **Data-Driven Cost Optimization**: Historical trace analysis through Genie revealed that 90% of internal developer sessions previously overpaid for frontier models on trivial interactions.
[01:00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3600s) Unity AI Gateway Wrap-Up and Agent Bricks Platform Overview
Ankit wraps up his demonstration of the Unity AI Gateway by emphasizing developer token efficiency and full operational observability. With features like configurable budget policies, the intelligent model router, and end-to-end tracing, organizations can finally alter their AI spending trajectory while continuing to deploy autonomous systems aggressively. Pointing out that raw intelligence remains one of the scarcest and most expensive resources in modern computing, Ankit highlights the core philosophical shift enabled by the gateway: transitioning enterprise development away from wasteful brute-force consumption toward deliberate value creation. As Ankit summarizes, “And so hopefully we can take the world from the old world of token maxing to something a little bit more like value maxing with the Unity AI gateway.”
Casey returns to the stage to contextualize where Unity AI Gateway sits within the broader Agent Bricks platform. She positions Agent Bricks as an end-to-end foundation built directly on top of Databricks’ core data engine, designed around three primary tenets: - **Choice**: Ensuring organizations retain architectural flexibility across underlying foundation models, distinct agents, and runtime harnesses without being locked into a rigid platform or forced to replatform. - **Context**: Supplying agents with rich metadata and lineage so they can retrieve the exact enterprise data required for complex workflows. - **Controls**: Enabling autonomous agent operation through the centralized governance of Unity AI Gateway and the isolation of Databricks Sandbox, which runs execution payloads safely against downscoped data.
```mermaid flowchart LR accTitle: Agent Bricks Architecture accDescr: Three core pillars of the Agent Bricks platform providing choice, context, and control. subgraph AgentBricks["Agent Bricks Platform"] direction TB Choice["<b>Choice</b><br/>Flexible Models, Agents & Harnesses"] Context["<b>Context</b><br/>Metadata & Enterprise Data Access"] Control["<b>Controls</b><br/>Unity AI Gateway & Databricks Sandbox"] end Choice --> AgentBricks Context --> AgentBricks Control --> AgentBricks ```
---
[01:02:44](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3764s) Satya Nadella on Systems of Context, Token Capital, and Enterprise IP
Databricks CEO Ali Ghodsi introduces Microsoft Chairman and CEO Satya Nadella to reflect on their nearly decade-long strategic partnership. Ali notes how quickly the landscape has transformed over the past year, prompting Satya to explain how customer mindsets have fundamentally evolved. Satya explains that enterprise leaders have moved past fixating solely on base foundation models and their standalone benchmark scores. Instead, organizations are focused on establishing their own proprietary frontier capabilities powered by their private business logic and intellectual property. Satya characterizes this industry-wide pivot: “So what I mean by that is, instead of just talking about here is the latest model and its capabilities, we are now going from frontier model worship to actually building a frontier ecosystem, building one company at a time.”
Ali connects this evolution to themes from Microsoft’s recent Build conference regarding the "future of the firm," where proprietary data and private IP form an organization's primary competitive moat. Satya expands on this by contrasting legacy enterprise software architectures with modern AI systems. While earlier enterprise eras centered on static systems of record (databases and ERPs) and systems of engagement (interfaces and communications), modern agentic AI demands dedicated *systems of context*. Without rich, curated contextual grounding, models engage in repetitive, costly token consumption.
Furthermore, enterprise data is not a static repository; it is an active, continuously evolving stream that includes execution histories and agent run traces. Satya shares a compelling example demonstrated at Microsoft's CEO Summit: a lightweight 5-billion-parameter model from Microsoft AI (MAI) successfully outperformed massive, frontier-class models on enterprise evaluations. This occurred not because of proprietary model magic, but because the smaller model was continuously tuned on high-quality enterprise data, reinforcement learning environments (RLE), and real execution traces. Satya warns that enterprises currently wasting tokens by running frontier models over raw data without retaining or learning from their traces are inadvertently leaking their most valuable asset—a dynamic he predicts will reverse quickly.
---
[01:05:40](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3940s) Deep Azure Integrations and Compounding Token Capital
Ali reinforces Satya's point, describing how poorly grounded agents enter inefficient loops—performing what amounts to a stochastic random walk over raw data, repeatedly burning tokens and driving up latency to extract insights that should have been captured structurally. Ali outlines the expanding native integrations between Databricks and the Microsoft ecosystem, including: - Embedding Databricks Genie directly within Microsoft 365 Copilot and Microsoft Teams. - Native integration across Microsoft Entra and Agent 365. - Exposing the Databricks Genie Model Context Protocol (MCP) server inside Microsoft Copilot Studio, allowing developers to build custom agents powered by Databricks context. - Bidirectional read/write interoperability across Delta Lake and Microsoft Fabric's OneLake.
Satya highlights how crucial these touchpoints are for enterprise economics, emphasizing that surfacing context right where users collaborate enables teams to accumulate compounding advantages in both workforce skills and computational assets. When Ali asks what the next decade of partnership and enterprise technology holds, Satya offers a foundational perspective on corporate longevity:
> “I mean, I look, I think the next 10 years for us will be a lot of what we talked about. That if a firm is about the tacit knowledge and learning, in fact, I always say this, which is you can always buy a tool, you can even sometimes outsource a job or a task, but you can never as a firm outsource your learning. So in this new world, what does it mean to build that tacit knowledge that is compounding both human capital and token capital?”
Satya concludes that while today’s enterprises rely on human capital navigating digital systems to build operational knowledge, the future belongs to organizations that compound human capital alongside "token capital"—autonomous systems trained on an organization’s own data and traces to generate sustained business value.
---
[01:09:16](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4156s) Databricks Apps: Breaking the Enterprise Data Bottleneck
Following Satya Nadella's appearance, Justin de Brabant, Director of Product Management at Databricks, takes the stage to address application development. Justin begins by reflecting on Databricks Apps, which launched the previous year with the goal of democratizing data and AI development across technical and non-technical teams alike.
He emphasizes how dramatically software creation economics have inverted over the past twelve months. With the explosion of autonomous coding agents and assisted development workflows, the historical friction and financial expense associated with writing application code has virtually collapsed. However, this shift has exposed a much harsher constraint for real-world enterprise deployments. Code generation may be trivial, but building production-grade enterprise software remains bottlenecked by governed, secure, and permissioned access to high-stakes internal data.
[01:10:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4200s) Databricks Apps: Governed Data Access and Massive Customer Momentum
Enterprise software applications are fundamentally different from standalone code repositories because their utility depends entirely on access to underlying enterprise data. As the speaker explains, the core hurdle teams face when attempting to move from an initial prototype into production is establishing secure, governed data access. To resolve that friction, Databricks Apps was engineered as an enterprise application platform with data security and governance built directly into its foundation. Over the past year, adoption has surged dramatically: more than 5,000 customers are actively deploying applications on the platform, and over 150,000 apps have been created—representing a sixfold increase compared to the previous year. Crucially, these systems are not sitting idle; hundreds of thousands of end users interact with them weekly, including many employees who previously had no direct touchpoints with enterprise data and AI tools in their daily workflows.
The speaker highlights that speaking with customers reveals the sheer variety of problems being solved across industries. Two aerospace leaders demonstrate this diversity: - **Thales:** The multinational aerospace and avionics company overhauled its in-flight intelligence platform for commercial airlines. The application aggregates system health telemetry, content engagement statistics, and passenger satisfaction signals across entire global airline fleets. This operational view allows the Thales operations team to anticipate cabin hardware and service issues before passengers ever notice them. The solution stitches together multiple Databricks components—real-time ingestion via Lakebase, lakehouse analytics, Genie conversational interfaces, and machine learning models forecasting operational needs—into a single interface. - **Joby Aviation:** The manufacturer of all-electric vertical takeoff and landing aircraft operates with extreme supply chain and material planning complexity behind every build. Joby created a unified application that maps demand, parts inventories, and procurement orders across all manufacturing programs in its fleet, allowing planning teams to detect parts shortages and bottlenecks well before they cascade into manufacturing delays.
As the speaker emphasizes when comparing these varied implementations, “These are all very, very different applications. Different teams, different applications. And this is what you see across all of our customers. Everybody is really innovating and doing different things across all their different industries.”
[01:12:44](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4364s) The Platform Foundation: Govern, Host, and Build
Databricks delivers these capabilities through three foundational pillars: Govern, Host, and Build. Each addresses a specific engineering hurdle that teams encounter when operationalizing internal web applications.
```mermaid flowchart TD accTitle: Databricks Apps Architecture Pillars accDescr: The three functional pillars of Databricks Apps comprising Govern, Host, and Build.
subgraph Govern["1. Govern: Identity & Data Controls"] direction TB G1["Databricks SSO Managed Auth"] G2["Unity Catalog Permissions (App or User Level)"] G3["Real-time Observability via ZeroBus into Managed Tables"] end
subgraph Host["2. Host: Managed Cloud Infrastructure"] direction TB H1["Configurable Instances & Auto-Scaling"] H2["Load Balancing & Sticky Sessions"] H3["Versioned Builds & Zero-Downtime Deployments"] end
subgraph Build["3. Build: Developer Experience & Ecosystem"] direction TB B1["Framework-Agnostic (Python / JavaScript)"] B2["AppKit TypeScript SDK (Lakebase, Genie, Jobs)"] B3["Coding Agent Plugins, Skills & MCPs"] end
Govern --> Host --> Build ```
- **Govern:** Identity is managed out of the box through Databricks SSO, eliminating the need to configure a separate identity provider or handle unique credentials for every application. Fine-grained authorization is enforced through Unity Catalog, giving administrators the flexibility to govern data access at either the overall application identity level or the individual end-user session level. Platform teams also get native, real-time observability: every user action and backend service invocation is streamed into ZeroBus in real time and persisted into Unity Catalog managed tables for audits and operational analysis. - **Host:** Databricks provides managed, cloud-agnostic container infrastructure. While the platform launched a year prior with only a single always-on instance type, it now supports customizable instance sizing and horizontal auto-scaling. The routing layer includes native load balancing and sticky session handling for multi-instance applications, complemented by versioned builds and zero-downtime rolling deployments to safeguard production releases. - **Build:** The runtime is framework-agnostic, supporting standard Python and JavaScript toolkits. Developers can accelerate delivery using AppKit, a native TypeScript SDK that includes pre-built integration plugins for Lakebase, Genie, autonomous agents, Databricks Jobs, and core Lakehouse tables. Furthermore, the platform integrates with AI coding agents through custom Model Context Protocol (MCP) servers, plugins, and skills created alongside developer ecosystem partners.
Reflecting on the platform's architectural philosophy, the speaker notes: “At the core of Databricks governance for apps is our managed auth offering. We have out-of-the-box identity through Databricks SSO. You don't have to have a separate IDP. You don't have to have a separate login for each individual app. It's all managed through Databricks.”
[01:16:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4571s) The Prototype-to-Production Gap in the Vibe Coding Era
While central data teams initially dominated internal app creation, the nature of enterprise software creation is experiencing a massive demographic shift. Industry forecasts predict that 1 billion new custom enterprise applications will be built by 2028, with the vast majority created by citizen developers rather than traditional software engineers. To illustrate the operational friction this creates for IT and platform teams, the speaker recounts an internal experience involving a Sales Operations manager at Databricks:
The sales ops manager sought to streamline her department's Quarterly Business Review (QBR) process. Her goal was to combine customer usage and consumption metrics, recent Salesforce meeting logs, and pipeline revenue forecast models into a single interactive dashboard. Using Replit, she vibe-coded a working prototype, demonstrated it to sales leadership to widespread praise, and believed the application was ready for immediate rollout.
However, when she presented the prototype to the internal platform engineering team to take it live, the project stalled. Platform engineers immediately pressed her on core operational realities: what specific production data tables were being queried, what sharing policies applied, what dependencies were bundled, who paid for compute costs, and how security was maintained. Because the sales ops manager did not possess administrative rights to production environments containing sensitive customer records, moving the prototype to production deteriorated into weeks of back-and-forth reviews.
The speaker highlights that this dynamic is no longer an isolated incident, but an organizational reality: “And this isn't about just happening once. This is happening thousands of times across your organization. And I guarantee some of you on the platform team have experienced this. Vibe coding is taking over. And unless we have a governed solution to it, it's going to overrun us.”
[01:18:24](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4704s) Scaling Citizen Development: Introducing AppSpaces and Serverless Micro Apps
To reconcile the speed of citizen vibe coding with enterprise security and fiscal discipline, Databricks interviewed enterprise customers and identified two fundamental product requirements: multi-application governance and cost-efficient execution for lightweight tools. In response, Databricks announced two major capabilities:
- **AppSpaces:** Standard governance in Databricks Apps historically operated on a per-app basis, which becomes unmanageable when hundreds of non-technical creators deploy thousands of distinct utilities. AppSpaces introduces a multi-application administrative boundary. Platform administrators can define governance, network, and security policies once at the space level, automatically inheriting those rules across all apps deployed inside it. Admins can restrict access boundaries—such as mandating that all applications deployed within a Sales Operations space can only be viewed by authenticated members of the sales organization. AppSpaces also provides centralized operational management, tracking aggregate compute spend, resource consumption, and user traffic across all child applications. - **Serverless Micro Apps:** While always-on, auto-scaling instances are essential for mission-critical, high-traffic tools, they are financially inefficient for lightweight departmental dashboards accessed only occasionally, such as an internal ops report referenced twice a month. Serverless Micro Apps provides a dedicated runtime featuring fractional CPU and memory allocations paired with dynamic scale-to-zero capabilities when idle. When an incoming request arrives, the application cold-starts in seconds. Despite utilizing shared underlying serverless infrastructure, each application operates inside a fully isolated execution sandbox to preserve multi-tenant security.
As the speaker summarizes the platform scaling challenge: “How do we unlock the next wave of builders without sacrificing security, governance, or budgets? And this is what you told us. First off, we need governance and ops that scale to thousands of apps.”
[01:20:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4800s) Announcing Genie App Builder: Context-Aware Vibe Coding
The presenter addresses the final major obstacle data teams encounter when trying to move vibe-coded applications into production: tools need to understand not just raw enterprise data, but the complete business context in which apps are deployed. This includes specific compute resources, operating environments, and sharing policies. In response, Databricks announced the release of Genie App Builder, designed specifically to bring business-level awareness into the vibe-coding experience. As the speaker explains: “Genie App Builder is vibe coding that understands your business.”
Rather than relying purely on generic code generation, Genie App Builder is context-aware and leverages the Genie Ontology to vibe code with deep semantic comprehension of an organization’s business logic and underlying data assets. Because it is 100% managed within Databricks, the platform delivers native enterprise security and governance guarantees. It also enforces production quality by default by integrating the AppKit SDK alongside custom skills, ensuring that enterprise development best practices are baked into every application generated out of the box.
```mermaid flowchart TD accTitle: Genie App Builder Architecture accDescr: Flow diagram showing how prototype code is brought into an App Space, governed, connected to enterprise systems, and deployed via Serverless Micro VMs.
subgraph Ingestion ["1. Prototype Ingestion"] A["GitHub Repo / Prototype"] --> B["Genie App Builder UI"] C["Docs & Wireframes"] -.-> B end
subgraph Governance ["2. Governed App Space"] B --> D["Supply Chain Ops App Space"] D --- E["On-Behalf-Of (OBO) Credentials"] D --- F["Approved Sharing Policies"] D --- G["Allowed Connectors"] end
subgraph Resources ["3. Databricks Resources"] D --> H["SQL Warehouse"] D --> I["Model Serving Endpoint"] D --> J["Lakebase Real-time Instance"] D --> K["Embedded Genie Agent"] end
subgraph Deployment ["4. Execution & Preview"] H & I & J & K --> L["Serverless Micro VMs (Side-by-Side Preview)"] L --> M["One-Click Production Deployment"] end ```
---
[01:21:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=4871s) Demo: Building and Deploying an App with Genie App Builder
Stepping into the role of a Supply Ops Manager for the “Brickster Cup,” the presenter showcases an inventory command center app designed to track site-level inventory, monitor demand forecasts as tournament teams advance, and execute direct operational actions like triggering production runs. The workflow begins not from scratch, but from an existing prototype vibe-coded using one of Databricks’ vibe-coding partners and stored in a GitHub repository.
Navigating to the Genie App Builder homepage, the presenter walks through configured app spaces—designated project workspaces curated by system administrators that bundle allocated computes, data assets, on-behalf-of (OBO) credential scopes, sharing policies, and external connectors:
* **Resource Linking:** Upon importing the GitHub prototype into the `Supply Chain Ops` app space, the Genie App Builder immediately inspects the available space inventory and auto-wires necessary backend components: a SQL warehouse, a model-serving endpoint for inventory forecasts, a Lakebase instance for real-time inventory tracking, and default questions for an embedded Genie conversational agent. Real data promptly begins streaming into the interactive preview. * **Governance and Permission Boundaries:** When the app attempts to load production manufacturing data, the builder flags that the current app space lacks permission. Rather than bypassing corporate rules, the user is offered an admin access request flow; for the V1 release, the presenter simply asks the agent to strip out the manufacturing data and its corresponding chart. * **Side-by-Side Micro VM Previews:** Changes can be evaluated visually through a proposed-versus-current preview window. Under the hood, serverless micro VMs spin up multiple isolated instances of the application concurrently, enabling developers to review code and styling changes (such as turning a chart green) before approving them. * **Boundary Enforcement:** To verify governance guardrails, the presenter attempts to configure a custom agent to send daily summaries through a Slack connector, which fails gracefully because Slack has not been whitelisted for this space. Similarly, attempting to update sharing permissions to “everyone at my company” is blocked because the space strictly restricts distribution to the designated supply operations group. * **Deployment and Developer Hub:** With resources confirmed, hitting deploy transitions the app into production within seconds. The presenter highlights that all showcased capabilities are available immediately, and directs users to the new developer portal at `developers.databricks.com` for prompts, architectural blueprints, sample apps, and practical implementation guides.
---
[01:28:01](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5281s) Customer Story: Virtue Foundation Powers Clinical Care with Genie
A video presentation highlights the humanitarian work of the Virtue Foundation, a non-governmental organization focused on delivering surgical and clinical healthcare to underserved populations across low- and middle-income countries. Using Genie, the organization developed a custom agent trained on decades of proprietary healthcare operational data, transforming historic travel logs and clinical field records into an authoritative, actionable source of truth.
A representative from the Virtue Foundation describes the dramatic efficiency leap unlocked by their custom agent, noting: “Using Genie has allowed us to take what would have taken a research assistant months. We’re now able to get answers to questions in seconds.” By asking plain-language questions—such as identifying total hospital counts in Ghana and filtering specifically for those equipped for cardiology—the team surfaces mission-critical medical intelligence in real time. Ultimately, the speaker emphasizes the profound human impact of agentic data systems: “The fact that we’re solving real problems for real people’s lives is just a humbling experience.”
---
[01:29:22](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5362s) Introducing Agentic ML Engineering with Mike Del Balso
Mike Del Balso, Director of Product Management at Databricks, steps onto the stage, opening with a lighthearted acknowledgment of the physical distance from the wings: “Justin was right, this is a deceptively long walk.” Del Balso establishes his roots in foundational enterprise machine learning, recounting his early career on the Google Ads machine learning infrastructure teams that helped catalyze the modern MLOps movement. He then details his subsequent work helping start the machine learning team at Uber, where he led the creation and expansion of Michelangelo, Uber’s pioneering internal ML platform, setting the stage to explore the evolution from traditional MLOps to agent-driven machine learning systems.
[01:30:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5400s) The Technical Debt of Production Machine Learning and Why GenAI Accelerates Demand
Reflecting on their career, the speaker shares how they previously ran a production machine learning startup that was recently acquired by Databricks, where their team continues to tackle the industry’s defining challenge. Across the entire machine learning field, the north star has always been “to make machine learning as easy and automated as possible, so we can bring intelligent decisions to all of the important places, all the important decisions across our business.” Despite countless attempts across the tech landscape, most efforts have struggled because production ML remains fundamentally difficult.
To illustrate the point, the speaker presents a well-known diagram from a foundational paper published by their Google team roughly twelve years ago. In the visual, the actual machine learning code occupies just a tiny red box in the center, surrounded by massive blocks representing all the supporting systems required to build, deploy, monitor, and maintain production models. Under the hood, this operational friction is fundamentally a data problem. While some industry observers predicted that generative AI would dampen interest in traditional machine learning, the speaker explains that GenAI has actually accelerated it: every new piece of software and AI-driven user experience creates more products to recommend, prices to optimize, fraud vectors to catch, and loan applications to evaluate.
Because production machine learning is notoriously hard to engineer and maintain, enterprises end up rationing it to only a tiny fraction of potential use cases. Most operational decisions across an enterprise continue to rely on static logic, rigid heuristics, and rules of thumb, leaving enormous value on the table.
- In the past year alone, organizations have generated more operational software than ever, exponentially increasing the volume of automated decisions required. - Despite having thousands of business decisions ripe for intelligent modeling, most enterprises only run a handful of production models due to operational complexity. - Replacing intelligent ML with statically coded rules and heuristics results in companies “leaving money on the table” by making systematically dumber business decisions.
---
[01:33:04](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5584s) Why Agents Are the Only Way to Automate Machine Learning
Automating this machine learning loop requires more than raw pipeline code; it requires capturing the human intuition present at every stage. Currently, highly trained data scientists and PhDs are constantly in the loop, deciding which features to engineer, diagnosing model convergence, aligning training metrics with business objectives, and deciding when a candidate model is safe to promote into production. Accelerating this lifecycle demands an automation layer capable of evaluating tradeoffs and making decisions in the exact same manner as expert practitioners.
The speaker emphasizes that agents are uniquely positioned to solve this problem, remarking: “And there's actually only one technology that can do this. And that's agents.” While agentic workflows have already reshaped software engineering, they have failed to transform machine learning for two structural reasons: 1. **Lack of Integrated Infrastructure:** Only a tiny handful of elite tech companies have built the unified operational infrastructure required to run the full machine learning loop automatically. 2. **Missing Operational Context:** Agents cannot make high-quality modeling decisions because they lack access to the rich institutional knowledge, historical performance metrics, and contextual data that human engineers rely on.
Databricks aims to resolve both bottlenecks by pairing an agent-ready infrastructure stack with domain-specialized agents capable of driving the loop end-to-end.
---
[01:34:52](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5692s) Agent-Ready ML Infrastructure: Serverless, Single Data Model, and Unified Governance
Databricks has spent over a decade establishing its position as an industry leader in data science and machine learning, sitting at the top right of analyst evaluation matrices with more than 500 million models trained on the platform over the past year alone. However, that entire existing stack was architected specifically for human practitioners. Over the past 18 months, Databricks redesigned and optimized the platform to make it fully “agent-ready” across three foundational pillars.
```mermaid flowchart TD accTitle: Agent-Ready Machine Learning Infrastructure accDescr: Three pillars of Databricks agent-ready infrastructure feeding into a single source of truth for agents.
subgraph Pillars ["Agent-Ready Stack Pillars"] A["Serverless Compute<br/>(Auto spin-up/down, zero cluster state)"] B["Unified Data Model<br/>(Training, features, serving & monitoring)"] C["Governed Actions<br/>(Unity Catalog permissions for agent tools)"] end
subgraph Lifecycle ["ML Lifecycle Source of Truth"] D["Governed Delta Tables"] E["Autonomous Agents"] end
A --> D B --> D C --> D D <--> E ```
First, the entire environment is now serverless out of the box so that autonomous agents do not waste cycles configuring clusters, tuning driver nodes, or managing infrastructure state. Second, the entire machine learning lifecycle is instrumented with a unified data model: feature stores, training datasets, model serving endpoints, inference logs, and monitoring metrics all stream into the same governed tables. This shared foundation gives agents an immediate view of cause and effect whenever they alter features or retrain models. Third, Databricks expanded Unity Catalog beyond traditional data and asset governance (models, experiments, endpoints) to govern the specific actions and tools agents are permitted to invoke. As the speaker explains: “We now have governance on the actions that your agents can take so that you can trust any changes that your agents are making to automate this machine learning loop.”
---
[01:36:44](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5804s) Announcing AI Runtime: Serverless GPU Compute for Deep Learning and LLMs
To eliminate the operational hurdles of scaling advanced neural architectures and large language models, Databricks is introducing AI Runtime. This new compute tier brings serverless, on-demand GPU capacity to the platform with zero upfront commitments, allowing teams to instantly spin up deep learning and LLM training workloads at arbitrary scale.
Having run AI Runtime in private preview over the previous two months, hundreds of organizations have already migrated their deep learning workloads onto it. Highlighting real-world enterprise adoption, the speaker references Merck's deployment of the platform to accelerate their latest drug discovery models. Building on that momentum, Databricks is officially expanding AI Runtime to support multi-node distributed training, unlocking massive datasets and large-scale model architectures within the governed enterprise lakehouse environment.
- **On-Demand Elasticity:** Provides serverless GPU instances with no upfront reservation or capacity lock-in. - **Deep Learning & LLMs:** Built specifically to support neural network training, fine-tuning, and large language model pre-training. - **Multi-Node Expansion:** Now supports multi-node distributed GPU training to train on massive datasets at any scale. - **Enterprise Proven:** Hundreds of organizations, including pharmaceutical giant Merck, have already deployed production deep learning workloads on the runtime during preview.
---
[01:38:13](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=5893s) Announcing Genie Code for Machine Learning
With the underlying infrastructure optimized for automation, the speaker pivots to the agent layer designed to execute production ML tasks: Genie Code for Machine Learning. Available immediately, this tool upgrades Databricks’ core coding agent with specialized intelligence and best practices derived from seeing more real-world machine learning deployments than any other vendor in the ecosystem.
Genie Code for Machine Learning is natively plugged into the Databricks platform, giving it direct programmatic awareness of experiment tracking, feature stores, and model deployment pipelines. Crucially, its context engine has been overhauled to understand how real-world data science teams collaborate and evaluate tradeoffs. Instead of treating machine learning code in isolation, the agent draws upon the broader institutional context—the unspoken assumptions, baseline metrics, and validation goals that human practitioners use—so it can co-pilot feature selection, run automated evaluations, and work seamlessly alongside data scientists and ML engineers.
[01:40:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6000s) Genie Code for ML: Ontology-Powered Machine Learning Engineering
Machine learning decisions hinge on deep institutional context that is typically scattered across an organization's entire platform: historical experiments, specific business metric definitions, past failures and successes, and established evaluation scripts. Without direct access to this context, an AI agent runs the risk of referencing incorrect datasets, choosing improper evaluations, and generating outputs that appear superficially plausible while remaining fundamentally flawed. To address this gap, Databricks integrated Genie Code with Genie Ontology. By grounding the agent in the platform's ontology, Genie Code learns exactly how an internal machine learning team operates, how models are structured, and how performance is validated.
```mermaid flowchart TD accTitle: Genie Code Grounded in Genie Ontology accDescr: Diagram illustrating how Genie Ontology feeds institutional context into Genie Code to handle feature engineering, training pipelines, and production evaluations.
subgraph ContextSource["Genie Ontology"] A["Team Workflows & Data Lineage"] B["Historical MLflow Experiments"] C["Existing Evaluation Scripts"] end
subgraph AgentCore["Genie Code Agent"] D["Contextual Feature Engineering"] E["Pipeline & Script Reuse"] F["Standardized Model Evaluation"] end
subgraph Outcome["Production Delivery"] G["Senior MLE-Level Outputs"] H["Accelerated Deployment"] end
A --> D B --> E C --> F D --> G E --> G F --> G G --> H ```
This integration enables Genie Code to engineer features adhering strictly to the patterns already used by the engineering team, reuse verified training scripts, and seamlessly plug into existing pipelines. It also respects and runs established evaluation suites so engineers can verify that a newly trained model meets the organization's existing production bar. Highlighting this paradigm shift, the speaker notes:
> “And that allows Genie code to really act like a senior machine learning engineer on your team.”
The operational impact of this architecture was demonstrated with Danfoss, a multinational engineering firm, which went from having no machine learning in place to deploying a complete production machine learning system in just 90 minutes—a milestone that typically requires weeks, months, or quarters across enterprise environments.
- **Feature Engineering Alignment:** Builds new model features using the exact coding patterns and workflows already maintained by the user's team. - **Evaluation Reuse:** Automatically discovers and applies existing evaluation scripts rather than inventing synthetic metrics, ensuring benchmark integrity. - **Danfoss Case Study:** Transitioned from zero machine learning to a fully functioning production ML deployment in 90 minutes.
---
[01:42:43](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6163s) Announcing Genie ZeroOps for ML
While speeding up model creation represents a massive breakthrough, building models accounts for only a fraction of day-to-day work; real-world machine learning teams spend 60% to 80% of their operational time maintaining production applications. Accelerating model creation by 100x risks creating 100x the operational burden, which no human engineering organization can scale to meet. Models in production face endless vectors of failure, including data drift, inference server latency spikes, broken upstream pipelines, and a long tail of unpredictable edge cases.
To eliminate this bottleneck, Databricks announced Genie ZeroOps for ML, an autonomous operational agent described by the speaker as “the biggest launch in machine learning in the past 10 years.” ZeroOps for ML fundamentally rethinks traditional observability. Standard monitoring setups observe metrics, identify an anomaly, and fire an alert demanding human intervention. In contrast, Genie ZeroOps for ML detects the issue, explores hypotheses, diagnoses the root cause, and synthesizes a verified fix before contacting the engineer.
> “And the flip here is that for the whole history of machine learning, we've always had a team member on call for our models. And zero ops for ML inverts that and allows an agent to be on call and your team is just in a position to approve the recommendations from that agent.”
- **The Maintenance Bottleneck:** ML engineering teams routinely spend 60% to 80% of their bandwidth managing, debugging, and maintaining active production deployments. - **Problem-to-Solution Shift:** Replaces basic metric threshold alerts with autonomous root-cause analysis and executable candidate fixes. - **Inverted On-Call Paradigm:** An autonomous agent serves as the primary tier on-call responder, while human engineers remain in the loop solely to evaluate and approve production deployments.
---
[01:45:22](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6322s) Demo: Genie ZeroOps for ML in Action
Amber takes the stage to demonstrate Genie ZeroOps for ML on a live production workload at the fictitious organization Bricksports Global. The team maintains a critical production model forecasting ticket sales for every match in the Data and AI World Cup. Because ticket sales forecasts directly guide tens of millions of dollars in event staffing and venue operations spend, forecasting errors risk heavy financial losses or empty seats due to ticket overpricing.
Amber begins by prompting Genie Code to set up an alert on the production endpoint. The agent provisions and registers the rule automatically, configured to trigger whenever observed ticket sales deviate outside a ±5% margin of forecasted demand. Moving the timeline forward to a scenario at 5:30 p.m. on a Friday, an email notification shows the alert fired 30 minutes earlier due to a sudden drop in forecast accuracy.
```mermaid flowchart LR accTitle: Genie ZeroOps Incident Remediation Workflow accDescr: Flowchart tracing an incident from alert trigger through automated hypothesis testing, feature creation, retraining, and one-click deployment.
subgraph Detection["Detection & Triage"] A["±5% Forecast Alert Fires"] B["ZeroOps Inbox Thread Created"] end
subgraph Investigation["Autonomous Diagnosis"] C["Query Unity Catalog & Run SQL"] D["Inspect Feature Values & Lineage"] E["Root Cause: Missing Knockout Stage"] end
subgraph Remediation["Autonomous Fix"] F["Engineer Tournament Stage Features"] G["Retrain & MLflow Model Registry"] H["Human Approval & 1-Click Rollout"] end
A --> B B --> C C --> D D --> E E --> F F --> G G --> H ```
Rather than requiring manual troubleshooting during off-hours, the alert email includes a button linking directly to the investigation in the Genie ZeroOps inbox. Over the 30 minutes between the alert and the engineer checking the notification, the system worked through dozens of hypotheses completely in the background:
> “Work that would have taken me hours to do manually, Genie Code root caused it before I even realized there was an alert.”
Genie Code queried Unity Catalog tables, executed diagnostic SQL queries, checked feature distributions, and evaluated MLflow experiment runs. It isolated the exact failure: the model lacked a feature capturing the tournament stage, leaving it unable to anticipate the surge in fan interest and demand accompanying the knockout rounds.
After identifying the root cause, Genie Code located unutilized matchup and tournament stage metadata, constructed the missing features, retrained the model with hyperparameter tuning, and logged the run to the MLflow Model Registry using Databricks best practices. Leveraging end-to-end data lineage, it validated the candidate model against the original evaluation benchmark, verified that forecast accuracy returned to target thresholds, and presented the engineer with a single-click deployment interface to route a canary slice of live traffic to the patched model.
- **Alert Criteria:** Configured to fire if actual ticket sales diverge beyond ±5% from forecasted values. - **Autonomous Root-Cause Analysis:** Inspected training notebooks, queried Unity Catalog tables, monitored drift across feature distributions, and executed MLflow runs without human direction. - **Tournament Stage Drift:** Identified that the model failed to incorporate tournament phase features, causing predictions to falter as soon as the tournament reached knockout matches. - **Automated Remediation:** Feature engineered the tournament stage and matchup attributes, executed hyperparameter sweeps, logged artifacts into MLflow, verified accuracy against historical eval benchmarks, and queued the model for one-click deployment.
[01:50:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6600s) Genie ZeroOps Resolves Stale Models and Agentic ML Engineering Arrives
Wrapping up the live demonstration, the presenter points out the newly deployed Version 2 of the model. Under normal circumstances, diagnosing and fixing a stale production model is a major, cross-functional incident that drags on for hours or even days, racking up potential business losses in the tens of millions of dollars. Genie ZeroOps for ML triaged and resolved the failure swiftly with minimal disruption to the team. As machine learning enters what Mike dubbed the "100x era," organizations face an explosion in model volume—and with a hundred times more models comes a hundred times the operational complexity. The core mission is to shield engineering teams from that mounting complexity, automating operational toil so practitioners can return to building breakthrough AI.
```mermaid flowchart LR accTitle: Genie ZeroOps Incident Remediation accDescr: The automated cycle of model drift detection, remediation via Genie ZeroOps, and zero-downtime deployment. subgraph S1["Detection"] A["Stale Production Model"] --> B["Alert & Root-Cause Triage"] end subgraph S2["Remediation"] B --> C["Genie ZeroOps for ML"] C --> D["Candidate Model v2"] end subgraph S3["Production"] D --> E["Zero-Downtime Rollout"] end ```
The speaker recaps the three foundational pillars showcased during the ML portion of the keynote:
- **AI Runtime:** Serverless GPU compute for training and fine-tuning deep learning models and large language models (now in public preview). - **Genie Code for Machine Learning:** An embedded agentic assistant integrated directly into local developer environments (now generally available). - **Genie ZeroOps for Machine Learning:** An autonomous operations agent designed to monitor, maintain, and self-heal production ML systems (slated for release in Summer 2026).
Reflecting on the historical vision of the discipline, the presenter emphasizes that the industry has spent decades trying to automate machine learning. With serverless GPU infrastructure paired with agent loops, ML engineering can finally execute a hundred times faster across a hundred times more models, driving granular personalization into every product interaction and optimizing business decisions at scale. The speaker leaves the audience with a closing charge:
> “Let your agents do the work now, so your team can focus on what's next.”
---
[01:52:48](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6768s) The Agentic Shift in Buyer Behavior and the Collapse of Waterfall Marketing
The keynote shifts from internal machine learning infrastructure to customer-facing applications. The host introduces Tassos Argyros, General Manager for CustomerLake. Argyros shares a personal retrospective: having spent ten years building a database company and another decade building a marketing technology platform to bridge raw data with intuitive business interfaces, leading CustomerLake represents a natural synthesis of those two decades. He grounds his talk in a central reality of modern business: roughly 80% of companies report that they compete primarily on the basis of customer experience.
```mermaid flowchart TD accTitle: Waterfall Campaigns versus Agentic Buying Reality accDescr: Contrast between slow traditional marketing workflows and high-speed multi-channel agentic shopping. subgraph Traditional["Traditional Waterfall Campaign"] M1["Marketing Strategy"] --> D1["Data Eng (Customer 360)"] D1 --> O1["Marketing Ops (Audiences)"] O1 --> C1["Creative Content"] C1 --> A1["Activation & Review (Weeks)"] end subgraph Agentic["Agentic Era Reality"] B1["Consumer Delegate Agent"] -->|"APIs / Seconds"| CH["Multiple Headless Channels"] CH -->|"Instant Decisions"| BUY["Automated Checkout"] end ```
However, the mechanics of customer experience are facing structural disruption because AI is fundamentally altering how consumers buy:
- **Autonomous Agent Delegation:** Consumers no longer evaluate purchases in isolation. When Argyros asks the room how many people consult an LLM before buying goods, roughly 80% raise their hands. Chatbot advice is rapidly transforming into autonomous delegation, where personal agents conduct research, price discovery, shipping comparisons, and checkout. - **Compressed Decision Cycles:** Traditional buyer journeys spanning days or weeks are collapsing into transactions completed in seconds or minutes. - **Headless, Multi-Channel Commerce:** Because shopping has become API-driven, shopping agents execute concurrent inquiries across dozens of channels simultaneously.
Argyros contrasts this high-velocity reality with traditional marketing practices. Historically, marketing departments ran waterfall campaigns: marketing set strategy, data engineering aggregated static Customer 360 datasets, marketing ops carved out broad segments of millions of people, creative teams generated static copy, and marketers activated campaigns over weeks-long rollouts. While adequate for slow-moving human consumers, waterfall workflows are unsuited to agentic buyers operating across APIs in real time. As Argyros explains:
> “When agents buy on multiple channels at the same time, in seconds, you can't spend weeks designing a single campaign.”
---
[01:57:12](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7032s) Introducing Infinity Campaigns and the Architecture of the Agentic CDP
To match the speed of autonomous consumer agents, businesses must invert their operational model: instead of blasting campaigns to audiences of millions, organizations need an agent for every customer to serve audiences of one. Argyros introduces **Infinity Campaigns**, an architecture designed to deliver real-time, continuous personalization.
```mermaid flowchart LR accTitle: Infinity Campaign Closed Loop accDescr: Continuous signal processing loop between an individual customer and their dedicated campaign agent. subgraph Loop["Continuous Engagement Engine"] A["Individual Customer"] -->|"Behavioral Signals"| B["Dedicated Customer Agent"] B -->|"Determines Next Best Action"| C["Dynamic Content Generation"] C -->|"Personalized Interaction"| A end ```
The name "Infinity Campaigns" reflects three core design principles rather than a mere visual resemblance to the infinity symbol:
1. **Evergreen and Self-Adapting:** These campaigns eliminate fixed start and end dates. Because autonomous agents power the loop, the campaign continuously adapts to shifting customer signals and evolving business objectives without requiring manual teardowns and redesigns. 2. **Infinite Content Permutations:** Instead of choosing from a handful of pre-rendered creative assets, agents draw from an essentially limitless pool of tailored copy, recommendations, and pricing variations tailored to an individual customer's precise context. 3. **Sub-Second Response Latency:** Unbound by human processing bottlenecks, agents respond instantaneously across any channel the customer (or the customer's buying agent) queries.
Throughout this transformation, human oversight remains intact. The marketer functions as the “chief strategist, the chief orchestrator, and the chief approver,” defining guardrails, priorities, and rules while delegating execution to an army of agents.
To ground the concept in enterprise architecture, Argyros turns to Customer Data Platforms (CDPs). Historically, a CDP sits as a business application between raw enterprise storage and execution layers (such as email tools, push notification engines, and ad platforms). Its traditional mandate is two-fold: ingesting raw transactional logs to build organized Customer 360 tables, and providing business users with a graphical user interface to slice audiences and configure journey steps.
Argyros highlights the central tension that Infinity Campaigns aim to resolve:
> “The marketer is the chief strategist, the chief orchestrator, and the chief approver of what the agents are doing. But now they have an army of agents that can do all the one-to-one personalization work that humans were just too slow to do before.”
[02:00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7200s) The Rise of the Agentic CDP and CustomerLake
Traditional Customer Data Platforms (CDPs) were designed to send audience and customer records downstream into operational marketing tools such as email platforms and advertising networks—a process known as reverse ETL. However, legacy CDPs operated as middleware completely detached from the underlying data warehouse or data lake. In the agentic era, building effective autonomous workflows requires more than raw data pipelines; agents demand deep business context situated directly where the data and compute reside. To unlock true "infinity campaigns," organizations require an entirely new class of platform: the agentic CDP.
An agentic CDP must fulfill three fundamental architectural criteria: 1. **Embedded in the Lakehouse:** It must live directly inside the data foundation where data, enterprise context, and autonomous agents are co-located. 2. **LLM-Core Architecture:** Large language models cannot be treated as superficial copilot sidecars; agents must form the core engine driving data transformation, reasoning, and operational execution. 3. **Power Infinity Campaigns:** It must support evergreen, real-time, continuously adapting campaigns capable of deploying individual agents to deliver one-to-one personalization for every single customer.
To lead this new category, Databricks announced CustomerLake, an agentic CDP built natively on the Databricks Data Intelligence Platform. Because CustomerLake is embedded directly in the lakehouse, customer data never leaves the security boundary, and all data and agent governance established in Unity Catalog carries over automatically. Designed with an autonomous engine yet tailored for business users through a democratized, business-first user interface, CustomerLake is built to power personalized customer experiences "a billion times a day." The platform is split into two primary operational modules: Profile Agents, which automate the raw-data-to-Customer-360 pipeline, and Campaign Agents, which empower marketing teams to execute Infinity Campaigns.
As the speaker highlighted regarding this shift: > “Now, in order to make agents work, you need data, but you also need context, and you need data and context to be where the agents live.”
```mermaid flowchart TD accTitle: CustomerLake Architecture accDescr: Architecture of CustomerLake showing Profile Agents transforming raw data into unified profiles, which Campaign Agents use to power Infinity Campaigns and reverse ETL.
subgraph DataPrep ["Data Ingestion & Context"] A["LakeFlow Connect & Lakehouse Data"] B["Databricks Marketplace Hydration"] C["Databricks Clean Rooms (3rd-Party)"] end
subgraph ProfileEngine ["Profile Agents"] D["Semantic Tagging & Cleaning"] E["Agentic Identity Resolution (Rules + LLM)"] F["Schema Mapping & Survivorship"] end
subgraph CoreLakehouse ["Unity Catalog & Lakehouse Storage"] G[("Governed Customer 360")] end
subgraph CampaignEngine ["Campaign Agents"] H["Genie-Powered Audiencing"] I["Infinity Campaigns (1:1 Agent Personalization)"] J["Native Managed Reverse ETL Connectors"] end
A --> D B --> D C --> E D --> E E --> F F --> G G --> H H --> I I --> J ```
---
[02:03:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7380s) Profile Agents and Agentic Identity Resolution
Profile Agents automate the intricate engineering journey from disparate raw datasets to a business-ready Customer 360. Triggered by natural language prompts outlining specific business use cases, these agents orchestrate an autonomous ingestion, transformation, enrichment, identity resolution, and evaluation loop. Data preparation begins using LakeFlow Connect to bring in third-party application sources, augment existing lakehouse tables, or hydrate profiles with third-party datasets from the Databricks Marketplace. Profile Agents automatically tag data semantically and resolve data hygiene flaws, including standardizing malformed email addresses, physical locations, and phone numbers.
At the center of Profile Agents is agentic identity resolution for first-party data—matching records belonging to the same individual across disparate datasets that lack common primary keys. Where traditional methods relied strictly on brittle static SQL rules or opaque, slow-to-train machine learning models, agentic identity resolution establishes a self-improving loop: - Agents first generate and test deterministic matching rules. - When ambiguous edge cases cause rule failure or inconclusive matches, reasoning LLMs step in to make probabilistic determinations. - If uncertainty remains, the system routes the record to human reviewers for feedback. - Every manual adjudication and confirmed match feeds back into the loop, allowing the agents to continually refine their underlying heuristics and LLM prompts.
For third-party data enrichment, CustomerLake introduces privacy-first external identity resolution powered by Databricks Clean Rooms, enabling cross-partner customer matching without exposing raw data outside the lakehouse. Finally, Profile Agents handle survivorship logic, schema mapping, and continuous data evaluation, verifying that final Customer 360 data products meet quality thresholds before downstream business consumption.
The speaker summarized the superiority of this hybrid resolution system: > “Historically, this has happened through static SQL rules or machine learning. Both approaches have significant limitations, and we think the agentic identity resolution wins on all fronts over these traditional approaches.”
---
[02:06:15](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7575s) Campaign Agents, Infinity Campaigns, and Managed Reverse ETL
Campaign Agents serve as the marketer-facing interface of CustomerLake, automating end-to-end campaign operations. Starting with a high-level natural language prompt, Campaign Agents review business goals and context to draft a complete campaign brief. Once human marketers inspect and approve the strategic brief, autonomous agents are deployed to handle hyper-personalized content generation and execution.
Campaign Agents also reinvent foundational CDP tasks like audiencing. Marketers can explore, segment, and refine audiences conversationally using conversational prompts powered by Genie, or work through a visual UI tailored for marketing operations teams. Crucially, any audience segment can be converted into an operational campaign with a single click.
At the center of this workflow are Infinity Campaigns. Because Infinity Campaigns run on LLMs rather than conventional predictive scoring models, teams do not have to wait months for data scientists to train, validate, and deploy complex machine learning pipelines or decision engines. Personalization starts out of the box on day one, complete with simulation environments, guardrails, and integrated measurement. To close the operational loop, Databricks announced built-in, native managed reverse ETL connectors within CustomerLake: - The platform launches with approximately 20 out-of-the-box managed connectors spanning leading advertising networks, marketing automation suites, and customer experience platforms. - Over a dozen additional native connectors are scheduled for release by the end of the year, with ongoing expansion planned thereafter.
As the speaker emphasized regarding the immediate time-to-value of this architecture: > “The beauty of Infinity Campaigns, because it's built around LLMs, is that it can start personalizing immediately.”
---
[02:08:53](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7733s) Demo: CustomerLake Campaign Agents in Action
The presentation transitioned to a live demonstration led by Databricks presenter Justin, who joked as he walked back onto the stage, “Encore demo. I'm flattered.” Justin outlined the scenario: assuming Profile Agents have already unified, cleansed, and resolved identities into a cohesive Customer 360 lakehouse table, his demonstration focused on showing how a non-technical marketing lead can use Campaign Agents and Genie to build segments and launch 1-to-1 Infinity Campaigns.
Assuming the persona of a marketing manager for an athletic apparel retailer named "Bricksport," Justin brought up the unified CustomerLake campaigns dashboard. The overview screen provided a high-level command center displaying active marketing campaigns, key monitored audience segments, campaign metrics, and the status of active reverse ETL sync pipelines pushing data downstream. Justin navigated into the Audiences tab, displaying existing customer segments alongside their current population counts and percentage growth over time, before initiating the creation workflow for a brand-new customer audience.
[02:10:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=7800s) Demo: Generating Audiences with Genie and Running Infinity Campaigns
Justin steps through a live demonstration of CustomerLake's campaign features, replacing rigid, manual filtering workflows with natural-language prompt execution inside Genie. He inputs the following target criteria:
> “Include our most valuable Team USA fans, who have spent more than 500 on merch this season. and exclude anyone who already bought the USA Fan Kit.”
Genie scans the existing catalog of attributes and audience segments, returning a qualified audience of 800,000 customers. The criteria remain entirely transparent and editable, allowing marketers to tweak thresholds directly. Justin inspects the underlying query generating the segment:
> “If you've ever supported marketing by pulling lists and writing custom campaigns, you can get really complex sequel to power some of these campaigns. And we got this from one prompt with Genie.”
```mermaid flowchart LR accTitle: Genie Audience to Infinity Campaign Workflow accDescr: Natural language prompt transforms into SQL, builds an audience segment, and powers 1-to-1 Infinity Campaign agent decisions.
subgraph Audience["Audience Generation"] A["Genie Natural Language Prompt"] --> B["Automated Complex SQL"] B --> C["800,000 Customer Audience"] C --> D["Dynamic Insights Dashboard"] end
subgraph Campaign["Infinity Campaign Execution"] D --> E["Infinity Campaign Engine"] E --> F{"Agent Reasoning per User"} F -->|"Marcus"| G["Defer: Frequency cap / timing"] F -->|"High-intent fan"| H["Matchday Offer: 15% off seat upgrade"] F -->|"General attendee"| I["Matchday Guide: Kickoff & parking"] end ```
After accepting the audience, CustomerLake produces an automated, dynamic insights dashboard dedicated to that specific cohort. Moving from audience building to continuous activation, Justin showcases an active **Infinity Campaign** set up for an "in-venue fan experience." The configuration interface sets the operational bounds for the autonomous system: - **Campaign Guardrails & Objectives**: Explicit conversion goals, metric trackers, frequency capping rules, and messaging catalogs (e.g., a matchday guide vs. a matchday promo offer). - **One-to-One Agent Reasoning**: In the Agents tab, the platform reveals the agent's individualized decisions profile by profile. For a fan named Marcus, the agent decides to "defer" sending anything. Drilling into the decision trail exposes the complete chain of reasoning: timing, real-time incoming signals, and frequency cap guardrails that prevented spamming. - **Dynamic Content Personalization**: For another fan, Genie triggers a personalized offer featuring 15% off a seat upgrade. For an attendee needing logistical support, it dispatches a matchday guide containing kickoff times and parking information instead. - **Aggregate Reporting**: Justin inspects total opens, clicks, and conversion rates directly within the campaign console before handing the stage back.
---
[02:14:14](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8054s) The Agentic CDP Era and CustomerLake Launch
Returning to the stage, the presenter highlights that true, continuous 1-to-1 personalization at scale is no longer theoretical:
> “The area of true one-to-one personalization It's here. It's very, very, very exciting.”
CustomerLake was built alongside forward-leaning enterprise design partners and an ecosystem spanning data providers, content engines, ad-tech platforms, activation pipelines, and systems integrators. The presenter formally outlines the rollout details: - **Availability**: CustomerLake is officially launched in **Private Preview**, accessible by reaching out directly to Databricks account teams. - **Summit Sessions**: A dedicated **CDP breakout track** and a special **Marketing Forum at 1:00 PM** are opened across the event schedule, featuring presentations by the Databricks engineering and product leads behind CustomerLake, alongside initial enterprise adopters. - **Strategic Vision**: Summarizing the milestone, the speaker notes how special it is to unite business context, data architecture, and autonomous agents under one roof “to build something that provides the perfect customer experience a billion times a day.”
---
[02:16:25](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8185s) AI-Driven Cyber Threats and the Launch of Lakewatch
Andrew Tu, General Manager of Lakewatch, takes the stage to introduce Databricks' security lakehouse strategy. Originally previewed two months prior at the RSA Conference, the initiative was significantly accelerated by Databricks' acquisition of Panther Labs, announced the previous day by CEO Ali Ghodsi.
Tu warns that modern LLMs have fundamentally shifted the economics and capabilities of offensive cyber operations: - **Instant Vulnerability Discovery**: Capabilities previously restricted to nation-state threat actors—requiring dedicated teams of reverse engineers hunting zero-days over months or years—can now be generated immediately by anyone utilizing advanced LLMs across open-source and proprietary software alike. - **Automated End-to-End Exploitation**: As detailed in recent research from Anthropic regarding attacks directed at the Mexican government, malicious actors used Claude to run automated kill-chains. The attackers prompted an agent to discover open server vulnerabilities, instruct the agent to exploit them, use vulnerabilities inside a crontab configuration to escalate to root access, and exfiltrate sensitive files—ultimately stealing 200 million personnel records and gigabytes of classified data.
Tu argues that the only viable defensive posture is deep architectural alignment between security operations and enterprise data engineering:
> “And now just about anyone with access to an LLM can put on an attack at a scale and speed that was previously unimaginable. That's the world we live in. And the only way that we're going to be successful in defending against this is if security and data teams work together.”
Tu breaks down the traditional Security Operations Center (SOC) model into three distinct operational pillars: 1. **Telemetry Ingestion**: Aggregating massive streams of machine data, endpoint metrics, server traces, and identity/access management logs from laptops and cloud infrastructure into central servers. 2. **Detection Engineering**: Crafting detection rules and SQL queries to identify suspicious behavior—such as impossible travel anomalies where an account logs in from San Francisco and then minutes later from the opposite side of the world. 3. **Incident Response (IR)**: Pushing flagged anomalies downstream to analysts, who must triage alerts and determine whether an event represents routine developer debugging or a catastrophic compromise.
[02:20:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8400s) The Failure of Legacy SIEMs Against Agentic Threats
The speaker explains how traditional security operations center (SOC) teams work: analysts investigate incoming alerts one by one, determining whether an anomaly is a genuine breach or merely a false positive. When a threat is verified, responders take targeted mitigation actions such as blocking an offending IP address or triggering a mandatory password reset. However, the legacy tooling underpinning this workflow—specifically traditional Security Information and Event Management (SIEM) systems—is fundamentally broken for the modern threat landscape. As the speaker emphasizes:
> “The challenge is the tools they're using are outdated. They typically use a system called a SIM, a security information event management tool, and these systems don't -- they're not designed for the scale of threat that we're facing today.”
Traditional SIEMs suffer from severe ingestion constraints driven by an unforgiving combination of technical bottlenecks and cost structures. To cope with steep licensing fees and storage limits, security teams are forced into operational compromises: - **Aggressive Data Pruning:** Organizations discard event data older than 30 days because retaining historical telemetry for months or years is cost-prohibitive. - **Upstream Data Filtering:** Pipelines drop high-volume telemetry deemed lower priority, risking critical blind spots in post-incident forensics. - **Exclusion of Contextual Data:** Modern rich telemetry—such as large language model (LLM) execution logs, source code changes, Jira tickets, and Slack messages—is omitted entirely, even though it provides crucial context (for example, proving whether an engineer was legitimately running a VPN to troubleshoot a service or experiencing an active machine takeover).
Compounding the data challenge is an unsustainable manual burden. A typical large enterprise managed roughly 7,000 alerts per week in 2020; today, that figure exceeds 30,000 alerts weekly. With a single investigation taking an analyst an average of 30 minutes—tracing pre-incident behavior, historical login origins, and downstream actions—an organization would require an impractical team of over 400 analysts just to keep pace. As automated, agent-driven attacks accelerate in volume and velocity, human-only defense models fail completely.
> “So the reality is that with the tools we have today, we don't stand a chance in meeting these agentic threats. We're dealing with incomplete data because we can't afford and we can't scale to bring it in. We're dealing with humans fighting agent attackers. And we're dealing with proprietary data silos where you can't analyze all the data because it's locked up in different locations.”
---
[02:22:29](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8549s) Lakewatch: The Agentic Security Lakehouse Architecture
To solve these architectural bottlenecks, Databricks built **Lakewatch**, an agentic security lakehouse designed natively on the Databricks platform. Built around open table formats, Lakewatch eliminates proprietary data lock-in and dramatically reduces storage costs, allowing enterprises to ingest and retain all structured, semi-structured, and unstructured telemetry without arbitrary retention cutoffs. Unity Catalog provides enterprise-wide data governance and auditability, while Agent Bricks and Genie inject autonomous AI agents across the entire security operational lifecycle:
- **Ingestion & Normalization:** Autonomous agents apply Extract, Transform, Load (ETL) pipelines and normalize diverse schemas into standard formats. If an upstream log format changes, the agents automatically adapt the pipeline to maintain uninterrupted data flow. - **Rule Authoring & Detection:** Agents assist security engineers in researching emerging attack vectors and authoring complex SQL-based detection logic. - **Investigation & Remediation:** Incoming alerts trigger agents that construct complete forensic case dossiers, allowing human analysts to simply review findings and approve proposed remediation actions.
```mermaid flowchart TD accTitle: Lakewatch Agentic Security Lakehouse Architecture accDescr: Telemetry flows through ingestion into Unity Catalog and Open Formats, where Genie and Agent Bricks orchestrate automated ETL, detection, and investigation.
subgraph Ingest["1. Multimodal Ingest"] A["Security Telemetry<br/>(Cloud, Endpoint, Network)"] B["Contextual Data<br/>(LLM Logs, Code, Jira, Slack)"] end
subgraph Platform["2. Databricks Platform Core"] C["Open Data Formats (Delta Lake)"] D["Unity Catalog<br/>(Central Governance & Access)"] C --- D end
subgraph Agents["3. Agentic Layer (Genie & Agent Bricks)"] E["Auto-ETL & OCSF Normalization"] F["Automated Detection Engineering"] G["Background Case Investigation"] end
subgraph Operations["4. Security Analyst Workflow"] H["Dedicated Security UI & Pre-built Incident Cases"] end
A --> C B --> C C --> E E --> F F --> G G --> H ```
---
[02:24:43](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=8683s) Demo: Automated OCSF Normalization and Genie Rule Authoring
Michael Anderson, Tech Lead for Lakewatch, takes the stage to demonstrate the live product, which integrates directly into the Databricks unified app switcher:
> “And Lakewatch is our agentic security lake house, and it's designed for the entire Sim journey. And that begins with data.”
Lakewatch ships with dozens of security-specific connectors supporting multimodal and heterogeneous data streams. Following its initial preview at the RSA Conference, Lakewatch introduced an automated normalization pipeline powered by Genie that maps incoming logs into the Open Cybersecurity Schema Framework (OCSF). By landing normalized data directly into curated gold tables, SOC teams can write a detection once and run it across disparate logs from multiple cloud providers and third-party vendors.
Michael shows how Lakewatch scales detection management across tens of thousands of simultaneous rules with native MITRE ATT&CK tagging and coverage heatmaps. Rather than spending days or weeks researching emerging threat actor tactics, security engineers can prompt Genie to generate production-grade detection rules in minutes.
Entering the prompt *“Please research and write me a detection for Storm-2460”* (noting with a laugh as the prompt processes, “This is how you know it's live”), Michael highlights the coordinated multi-agent workflow: - **Background Threat Intelligence Research:** An autonomous research agent crawls web databases, tracking CVE disclosures, threat blogs, and attack path reports for the complex multistage malware Storm-2460. - **Pipeline Introspection & Sampling:** Concurrently, Genie inspects the environment's telemetry pipelines, discovering connected CrowdStrike sensor feeds transformed into the OCSF `process_activity` gold table. Genie samples active records to confirm prerequisite fields exist (identifying certutil/search utility command patterns without active malicious indicators). - **SQL Synthesis and False-Positive Validation:** Combining web research with internal telemetry schema, Genie drafts a detection SQL query covering all seven stages of the malware kill chain. It validates the query against the past 24 hours of data and runs historical backtests over the previous 30 days to ensure zero false positives. - **Turnkey Rule Generation:** Genie packages the validated SQL into a production rule pre-tagged with relevant MITRE ATT&CK tactic/technique metadata, compressing a week-long engineering cycle into roughly two minutes. - **Agentic Case Summarization:** Turning to the Lakewatch incident cases console, Michael demonstrates that cases marked with a star icon have been automatically investigated in the background by Genie (such as analyzing an anomalous rate-limiting surge alert generated by an enterprise Web Application Firewall), delivering pre-compiled forensic summaries to triage analysts.
[02:30:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9000s) Demo: Autonomous Threat Investigation with Genie and Lakewatch
Continuing his live demonstration of Lakewatch, Michael reviews an auto-generated incident report. Genie does not simply summarize an event; it actively hunts down related cases, matching attack patterns against the web application firewall (WAF) and tracking activities originating from the same IP address. It cross-references external threat intelligence feeds to check for known malicious reputations and inspects internal perimeter logs to determine whether the suspicious traffic breached defenses beyond the perimeter. Furthermore, Genie continuously learns from how human analysts handled prior investigations, automatically running updated analysis routines so that human staff can bypass repetitive runbooks and focus exclusively on novel aspects and threat mitigation.
To illustrate how this works on signals that usually get ignored, Michael navigates to an uninvestigated informational alert labeled "suspicious API request." In a typical security operations center (SOC) flooded with tens of thousands of alerts weekly, low-fidelity statistical deviations from baseline traffic are routinely discarded due to lack of bandwidth. Michael triggers Genie to investigate the case live. Genie immediately inspects the raw event details and identifies an incoming POST request where a parameter named `readContext` deviates sharply from established traffic baselines.
```mermaid flowchart TD accTitle: Genie Autonomous Threat Investigation Workflow accDescr: Sequence of analytical steps taken by Genie to triage a low-fidelity alert into a critical security finding. subgraph Triage["Initial Triage"] A["Informational Alert: POST request"] --> B["Detect anomaly in readContext parameter"] B --> C["Query external threat intelligence feed"] end subgraph DeepDive["Code & Context Analysis"] B --> D["Inspect endpoint source code"] D --> E["Identify hardcoded backdoor logic"] E --> F["Check commit history & author details"] end subgraph Resolution["Report & Escalation"] C --> G["IP enriched: Google Cloud origin"] F --> H["Scan Jira tickets: zero authorization"] G --> I["Escalate to high severity true positive"] H --> I end ```
Working concurrently, Genie queries threat feeds to check whether the origin IP is part of a known botnet (finding no external hits), identifies the microservice hosting the endpoint, and scans the application source code. Within seconds, Genie spots a backdoor: when the `readContext` parameter matches a specific string, the code completely bypasses all access control checks and directly returns the decryption key. Genie then broadens its probe autonomously, pinpointing the exact commit date (which matches the date of the anomalous request), locating a duplicate instance of the backdoor elsewhere in the repository, checking the committer's manager, and querying Jira for associated tickets. Finding no Jira tickets or legitimate work authorization for the change, Genie assembles a comprehensive incident brief noting that the request originated from Google Cloud infrastructure and recommending an immediate escalation to high severity. Reflecting on the speed and thoroughness of the automated triage, Michael emphasizes the paradigm shift:
> “This is the first time in history that my capacity to investigate cases and my capacity to be thorough in investigating every single thing in my infrastructure is no longer bounded or limited by the size of my SOC team.”
- Genie analyzes low-fidelity signals that human SOC teams lack the bandwidth to investigate manually. - The automated code inspection identified a hardcoded backdoor that bypasses access control and exposes decryption keys. - Investigation history, commit records, and project management tickets (Jira) were correlated without manual analyst intervention.
---
[02:33:15](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9195s) Open Security Lakehouse: Breaking Silos and Fighting Agents with Agents
Taking the stage after the demo, Andrew underscores that Lakewatch allows organizations to analyze all their enterprise telemetry without being constrained by traditional ingestion limits or runaway licensing costs. Because modern cyberattacks increasingly deploy autonomous agents to probe attack surfaces, defensive postures must counter with equally sophisticated defensive agents. Lakewatch achieves this by breaking out of proprietary vendor silos, storing security data in open formats within an open ecosystem that accepts feeds from any source.
Although Lakewatch launched only a few months prior, enterprise adoption has ramped quickly, as showcased at the previous day's Cyber Forum featuring organizations such as Atlassian, National Australia Bank (NAB), and Adobe. Andrew highlights Adobe's deployment, where the security team faced petabytes of telemetry that overwhelmed their legacy SIEM infrastructure. By ingesting those petabytes into an open security lakehouse on Databricks, Adobe applied machine learning and agentic workflows to uncover sophisticated threats that were previously impossible to detect. Andrew urges technical leaders across disciplines to align their efforts:
> “To sum it up, the only way that we can meet today's security threats is by having security and data teams work together.”
- Lakewatch eliminates data silos by keeping enterprise telemetry in open lakehouse storage formats. - Enterprises deploy AI defense agents to counter automated, agent-driven attacks. - Adobe migrated petabytes of telemetry from legacy systems to Databricks to run advanced AI/ML threat detection.
---
[02:35:45](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9345s) Databricks x Panther: Rebuilding the AI SOC from First Principles
Reynold Xin takes the stage to discuss Databricks' agreement to acquire cybersecurity company Panther, welcoming Panther's founder and CEO, Jack Naglieri. Recognizing that many in the audience come from data engineering and analytics backgrounds rather than cybersecurity, Reynold invites Jack to outline his history. Jack explains that he spent roughly eight years in cyber operations, eventually leading detection and response engineering at Airbnb. At Airbnb, standard commercial SIEM products proved too slow, expensive, and operationally burdensome to handle modern cloud-scale workloads. His team built and open-sourced an internal detection platform, which ultimately served as the founding inspiration for Panther. Today, technology companies including Coinbase, Docker, and Block rely on Panther to outpace attackers.
Reynold highlights an unusual decision in Jack's founder journey: stepping down from day-to-day operational CEO responsibilities a few years earlier to spend 100% of his time writing code and hacking on early LLMs, before later returning to the CEO seat. Jack explains that as large language models advanced, he realized AI could not simply be bolted onto manual legacy triage workflows; the SOC needed to be redesigned from first principles around autonomous agents. Starting with LLM-generated detection rules, his research quickly advanced to full end-to-end alert triage, removing the human bandwidth bottleneck that has constrained security teams for decades. Addressing the broader conference theme of whether security teams will eventually operate like data teams, Jack offers a nuanced perspective: security personnel are incident response specialists who must understand attack mechanics, not data infrastructure plumbing. The goal of combining Panther's detection application layer with the Databricks data lakehouse is to deliver cloud-scale data processing and AI agents to security operators without burdening them with pipeline maintenance.
> “And during that time, we found that the typical off-the-shelf SIEMs just didn't really work for our use cases.”
> “So agents really allowed us to break that fundamental scaling problem in the SOC.”
- Jack Naglieri founded Panther after developing custom, open-source security data tooling to replace legacy SIEMs at Airbnb. - Key enterprise customers running on Panther include Coinbase, Docker, and Block. - Jack temporarily stepped away from CEO duties to prototype first-principles agentic workflows for end-to-end alert triage. - The Databricks and Panther combination pairs enterprise data infrastructure with specialized security detection, response, and compliance applications.
[02:40:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9600s) Transition and Recap: Laying the Agentic Data Foundation
Wrapping up the discussion on security and Panther, Jack Naglieri highlights the immense value of giving security agents access to broader organizational datasets so they can enrich context, triage alerts, and empower security teams with greater response accuracy. With warm applause, Databricks CEO Ali Ghodsi retakes the stage to close out the two-day keynote and weave together all the announcements. He reiterates the foundational thesis introduced on day one: while modern AI models are extraordinarily capable, they are ineffective without enterprise context. As Ali Ghodsi summarizes: “We started the keynote by saying that we have all this AI that's amazing. But we have to give it all this context that we have at the bottom.” Over the two days, Databricks demonstrated how its integrated platform feeds clean, controlled, governed context directly into AI agents.
Ali recaps the base data layer required to make enterprise AI reliable and real-time: * **Lakeflow:** Serves as the unified ingestion layer, expanded with hundreds of native connectors to pull data from diverse enterprise sources into the lakehouse in any open format. * **Lakehouse RT:** Powered by the Raiden engine—which Ali jokingly describes as “Reynold's dream engine”—bringing blazing-fast execution speeds that allow executive dashboards to render instantaneously and AI agents to query underlying data with minimal latency. * **Lakebase:** Serverless Postgres built directly into Databricks to significantly drive down total cost of ownership (TCO) for data-backed applications. * **LakeTap:** Unlocks unified hybrid querying by seamlessly combining the Lakehouse and Lakebase, serving as the prime operational bridge for transactional and analytical workloads.
---
[02:41:20](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9680s) Governance, Genie Ontology, and the Fleet of Specialized Agents
Moving up the stack from raw data to governance and context, Ali highlights how Unity Catalog and Unity AI Gateway ensure safe and cost-effective execution. He reflects fondly on the live demo presented by Ankit Mathur, which showcased the advisor pattern: using a lightweight, open-source model to handle the majority of tasks and calling an intelligent advisor model only when expensive or complex reasoning is required. On top of governance sits the Genie Ontology, an automatically constructed enterprise knowledge graph that updates continuously in the background to supply business context to agents.
To make this accessible across the entire enterprise, Databricks introduced Genie 1 as a single pane of glass that eliminates the complexity of navigating individual workspaces or multiple data warehouses. Business users can query enterprise data directly without being overwhelmed by technical data tooling. Ali urges data teams to deploy this broadly: “Please help your organization get Genie 1 in everyone's hands.” He explains that administrators can achieve this simply by enabling Automatic Identity Management (AIM) in Databricks and granting non-technical colleagues a lightweight consumer role.
Beyond Genie 1, specialized Genies and agentic tools handle dedicated lifecycle workflows: * **Genie Agents:** Intelligent digital coworkers that handle recurring operational tasks and scheduled data automations. * **Genie Code for ML:** Highlighted in Mike Del Balso’s demo, enabling data scientists and engineers to build sophisticated machine learning prediction models, troubleshoot code paths, and maintain automated real-time performance monitoring. * **Genie ZeroOps for ML and Data Engineering:** Ali’s personal favorite agent, designed to autonomously maintain, diagnose, and repair data engineering pipelines whenever upstream or downstream jobs fail. * **Genie App Builder:** Demonstrated by Justin as part of Databricks Apps, enabling users to vibe-code custom internal applications and micro-apps directly on governed lakehouse infrastructure.
---
[02:43:40](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9820s) Agent Bricks, Lakewatch, CustomerLake, and the Open Vision
Ali recaps the developer and operational layers unveiled during the keynote. For developers building custom logic, Matei Zaharia introduced Omnigent, an open-source meta-harness powering Agent Bricks alongside managed developer services such as Agent Memory Services, secure VM Sandboxes, and dedicated storage. Through Databricks Apps, organizations can deploy agile, serverless micro-apps that scale to zero to maintain minimal run costs. For domain-specific workflows, the keynote showcased Lakewatch—an open agentic security lakehouse integrated with Panther to automate triage, alert resolution, and continuous threat hunting—and CustomerLake, an agentic Customer Data Platform (CDP) designed for real-time Customer 360 insights and automated Infinity Campaigns.
```mermaid flowchart TD accTitle: The Databricks Agentic Enterprise Architecture accDescr: An architecture diagram showing data foundation, governance, context, and specialized agent applications.
subgraph AppsAgents["Agentic Apps & Experiences"] G1["Genie 1 & Consumer Role"] GTools["Genie Code & Genie ZeroOps"] Apps["Databricks Apps & Genie App Builder"] SecMkt["Lakewatch (Security) & CustomerLake (CDP)"] end
subgraph DevContext["Context & Agent Infrastructure"] Ontology["Genie Ontology (Enterprise Knowledge Graph)"] AgentBricks["Agent Bricks & Omnigent Meta-Harness"] Memory["Agent Memory Services & Sandbox Execution"] end
subgraph GovLayer["Unified Governance & Cost Control"] Unity["Unity Catalog & Unity AI Gateway"] Advisor["Advisor Pattern (Cost & Model Routing)"] end
subgraph DataLayer["Agentic Data Foundation"] Lakeflow["Lakeflow (Ingestion & Connectors)"] RT["Lakehouse RT (Raiden Engine)"] Lakebase["Lakebase Serverless Postgres & LakeTap"] end
DataLayer --> GovLayer GovLayer --> DevContext DevContext --> AppsAgents ```
In closing, Ali abstracts the entire architectural vision into Databricks' core commitments: open-source freedom, rigorous cost controls, unified governance, and continuous enterprise context. Summarizing the ultimate platform goal, Ali emphasizes: “In short, we want you to have any data, any model, and any harness, no cloud lock-in, for your whole infrastructure.” He encourages attendees to spend the rest of the conference exploring technical breakout sessions, engaging with ecosystem partners in the expo hall, and joining the evening celebration at Oracle Park featuring The Chainsmokers.
- **Universal Interoperability:** Freedom of choice across models, data formats, and agent harnesses with zero cloud lock-in. - **Unified Governance:** A central control plane in Unity Catalog and Unity AI Gateway to track permissions, enforce data security, and audit agent spend. - **Enterprise Context:** Automated, background graph generation via Genie Ontology that grounds every agent action in enterprise reality.
## Notable quotes > [00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=0s) We don't need more intelligence for the AI, we need more context. — Ali Ghodsi
> [36:11](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=2171s) When you see agent demos and agent pilots, you're usually just seeing the agent loop, but the agent loop is only 1% of the work that you have to do. — Casey Uhlenhut
> [52:37](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=3157s) Instead of just talking about here is the latest model and its capabilities, we are now going from frontier model worship to actually building a frontier ecosystem, building one company at a time. — Satya Nadella
> [01:52:48](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=6768s) When agents buy on multiple channels at the same time, in seconds, you can't spend weeks designing a single campaign. — Tassos Argyros
> [00:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=0s) I think in the next 6 to 12 months, most organizations are going to see that the costs are going up so much that it's actually prohibitive to their companies. — Ali Ghodsi
> [02:30:00](https://www.youtube.com/watch?v=sn9My5Pj0mE&t=9000s) This is the first time in history that my capacity to investigate cases and my capacity to be thorough in investigating every single thing in my infrastructure is no longer bounded or limited by the size of my SOC team. — Michael Anderson
## Mentioned 7-Eleven, Adobe, Agent 365, Agent Bricks, Agent Memory Services, AI Runtime, Airbnb, Ali Ghodsi, Alibaba Qwen, Amber, Andrew Tu, Ankit Mathur, Anthropic, Antigravity, AppKit, AppSpaces, AstraZeneca, Atlassian, Block, Bricksport, Bricksports Global, Casey Uhlenhut, Claude, Claude Code, Claude Opus, Coinbase, CrowdStrike, Cursor, CustomerLake, Danfoss, Data and AI World Cup, Databricks, Databricks Apps, Databricks Clean Rooms, Databricks Free Edition, Databricks Jobs, Databricks Marketplace, Databricks Sandbox, Databricks SSO, Delta Lake, Docker, Genie, Genie 1, Genie App Builder, Genie Code, Genie Code for Machine Learning, Genie Ontology, Genie ZeroOps, Genie ZeroOps for Machine Learning, GitHub, Google, Google Cloud, Google Gemini, Harvey AI, Jack Naglieri, Jio, Jira, Joby Aviation, Justin de Brabant, Lakebase, Lakeflow, LakeFlow Connect, Lakeflow Designer, Lakehouse RT, LakeTap, Lakewatch, LangGraph, MasterCard, Matei Zaharia, Merck, Michael Anderson, Michelangelo, Microsoft, Microsoft 365 Copilot, Microsoft Copilot Studio, Microsoft Entra, Microsoft Fabric, Microsoft Teams, Mike Del Balso, MITRE ATT&CK, Model Context Protocol (MCP), Moonshot AI Kimi, Mukesh Ambani, National Australia Bank, Omnigent, OneLake, Open Cybersecurity Schema Framework (OCSF), OpenAI, OpenAI Agents SDK, OpenAI Codex, OpenAI GPT, OpenTelemetry, Oracle Park, Panther Labs, Poly, PostgreSQL, Raiden, Reliance Industries, Reliance Intelligence, Replit, Reynold Xin, RSA Conference, Satya Nadella, Serverless Micro Apps, Slack, SpaceX, Tassos Argyros, Thales, The Chainsmokers, UC Berkeley, Uber, Unity AI Gateway, Unity Catalog, Virtue Foundation, xAI Grok, ZeroBus
## Video description Watch the full keynote from day 2 of Data + AI Summit, which brought together 30,000+ data professionals across data engineering, scientists, analysts, architects, and business leaders.
The speakers covered new open source innovations, major updates to the Databricks Platform, and marketing and security solutions built for the agentic era.
0:00 Databricks CEO and Co-founder Ali Ghodsi on how context + AI unlocks enterprise agents that work 3:21 Product updates and innovations you’ll explore in this keynote 11:11 The AI harness problem: anyone can build their own agent harness, making interoperability hard 18:56 Why we built Omnigent, an open meta-harness, and how it works 23:20 Demo: Omnigent OSS 33:05 Why we open sourced Omnigent 34:44 What’s new in Databricks Free Edition 36:11 Engineers get stuck building infrastructure, and Agent Bricks solves this problem 40:28 Agent Bricks provides choice across models, agents, and harnesses 42:11 Agent Bricks solves the context problem 44:23 Announcing Agent Memory Services, powered by Lakebase 45:18 Announcing Databricks Sandbox: VMs for secure code execution 46:51 New capabilities in Unity AI Gateway: centralized governance, security, and cost controls over agents, models, and tools 52:37 Demo: from toxenmaxxing to valuemaxxing with Unity AI Gateway 1:09:16 Governed access data is the biggest roadblock to production for AI apps 1:13:04 How Databricks Apps brings app vibe coding in the enterprise 1:16:35 The prototype to product gap with vibe coded apps 1:17:53 What’s new in Databricks Apps 1:21:11 Demo: Building an app with Genie App Builder 1:28:01 Virtue Foundation powers healthcare with Genie 1:29:22 The only way to automate ML is with agents 1:37:07 Announcing AI Runtime: Serverless GPU compute for enterprise deep learning 1:38:13 Announcing Genie Code for ML: an agent for production ML engineering 1:42:43 Announcing Genie ZeroOps for ML: an agent that runs your ML systems 1:45:22 Demo: Genie ZeroOps for ML 1:51:01 Agentic ML engineering is here 1:52:48 AI is changing how people buy 1:57:12 Infinity Campaigns, a new approach to marketing campaigns, powered by the agentic CDP 2:01:33 Announcing CustomerLake: an agentic CDP to power the perfect customer experience – a billion times a day 2:08:53 Demo: CustomerLake Campaign Agents for always-on engagement 2:16:25 Agents are fundamentally changing security 2:22:29 Lakewatch: fight every security threat with agents 2:24:43 Demo: Lakewatch, an open agentic security lakehouse 2:35:45 Databricks x Panther 2:40:36 The Agentic Data Foundation
[Source](https://www.youtube.com/watch?v=sn9My5Pj0mE)