Data + AI Summit Keynote 2026 | Day 1

Data + AI Summit Keynote 2026 | Day 1 — Databricks

Databricks

AI-assisted summary of Data + AI Summit Keynote 2026 | Day 1 — original source. Check the original for full context.

Source published:

Key ideas

  • Databricks introduced Lake Transactional/Analytical Processing (LTAP), which uses idle CPU cycles on Lakebase storage nodes to transcode row-based operational Postgres data inline into open Parquet, Delta Lake, and Iceberg formats without CDC pipelines. (165:18)
  • The company announced Lakebase, a serverless, decoupled PostgreSQL operational engine delivering sub-10ms transactional latencies past 600,000 ops/sec, featuring sub-500ms zero-copy database branching and cross-cloud disaster recovery. (134:27)
  • Databricks launched Reyden and Lakehouse//RT, an ML-optimized query engine capable of 7ms query latencies and sustaining 12,000 QPS under sub-second P90 tail latencies on Delta Lake and Iceberg tables. (119:44)
  • Genie One has reached General Availability alongside Genie Ontology and the OntoRank context graph, designed to overcome enterprise agent context bottlenecks and cut agent runtimes in half. (79:22)
  • Databricks agreed to acquire Panther Labs to launch Lakewatch, an agentic SIEM built directly on security lakehouses to counter machine-speed cyber threats using Python-native detections. (55:58)

Detailed notes

## Background Published by Databricks on YouTube on June 16, 2026, this video captures the Day 1 Keynote of the Data + AI Summit. According to the page description, Databricks co-founders, product leaders, and global enterprise executives unveil major product launches and real-world deployment stories centered on enterprise data and AI systems.

## TL;DR The keynote argues that enterprise AI adoption is held back by a lack of operational business context and governance rather than raw model intelligence. To address this, Databricks presents an agentic data foundation that includes the real-time Reyden query engine, Lakehouse//RT, Serverless Postgres with Lakebase, and an LTAP architecture that unifies transactional and analytical data on open formats without pipeline replication. Above the data layer, the company introduces Genie Ontology for context indexing, the Genie One AI coworker, and Genie ZeroOps for autonomous data engineering remediation, alongside centralized model governance via Unity AI Gateway. The presentation also highlights strategic enterprise expansions, including the planned acquisition of Panther Labs for the Lakewatch agentic SIEM, production use cases from PepsiCo and Mastercard, and an on-stage conversation with OpenAI co-founder Greg Brockman.

## Atmosphere & style The presentation carries the polished, high-voltage momentum of a major enterprise developer keynote, alternating between visionary founder addresses and fast-paced technical deep dives. Speakers punctuate their architectural arguments with self-deprecating engineering humor and real-world failure modes, from runaway agent costs to fabricated executive briefing numbers. The atmosphere remains tense and pragmatic during high-concurrency live benchmarks and split-screen pipeline failovers, creating a narrative arc that moves from theoretical AI bottlenecks to practical, hardened infrastructure.

## Bulletin - [02:45:18](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9918s) Databricks introduced Lake Transactional/Analytical Processing (LTAP), which uses idle CPU cycles on Lakebase storage nodes to transcode row-based operational Postgres data inline into open Parquet, Delta Lake, and Iceberg formats without CDC pipelines. - [02:14:27](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8067s) The company announced Lakebase, a serverless, decoupled PostgreSQL operational engine delivering sub-10ms transactional latencies past 600,000 ops/sec, featuring sub-500ms zero-copy database branching and cross-cloud disaster recovery. - [01:59:44](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7184s) Databricks launched Reyden and Lakehouse//RT, an ML-optimized query engine capable of 7ms query latencies and sustaining 12,000 QPS under sub-second P90 tail latencies on Delta Lake and Iceberg tables. - [01:19:22](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4762s) Genie One has reached General Availability alongside Genie Ontology and the OntoRank context graph, designed to overcome enterprise agent context bottlenecks and cut agent runtimes in half. - [55:58](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3358s) Databricks agreed to acquire Panther Labs to launch Lakewatch, an agentic SIEM built directly on security lakehouses to counter machine-speed cyber threats using Python-native detections.

## Key takeaways - [07:47](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=467s) Enterprise AI is constrained by an organizational context bottleneck rather than model reasoning capability, preventing autonomous agents from acting safely on private data. - [19:23](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1163s) Iceberg v3 and Delta Lake 5 are actively converging, establishing unified physical and metadata layers that eliminate format fragmentation across cloud object storage. - [33:50](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2030s) Unity AI Gateway provides a free centralized governance plane to manage multi-model API spend, set department-level rate limits, and secure Model Context Protocol (MCP) integrations. - [38:50](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2330s) Genie Ontology indexes internal knowledge into a live context graph using the OntoRank algorithm, boosting enterprise agent accuracy by over 30 percentage points. - [46:32](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2792s) Genie ZeroOps deploys autonomous agents into the data plane to automatically detect schema or data anomalies, write patches, and safely verify them inside isolated shallow clones. - [49:20](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2960s) Databricks positioned its unified lakehouse platform as the new agentic system of record, replacing disconnected SaaS application silos. - [55:58](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3358s) Panther Labs was acquired to power Lakewatch, enabling security teams to retain raw telemetry cost-effectively and automate threat triage using Python-native detection logic. - [01:34:23](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5663s) ZeroBus and Lakeflow deliver serverless, millisecond-latency ingestion wire-compatible with Apache Kafka at up to 12 GB/s without managing broker infrastructure. - [01:37:43](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5863s) Traditional data engineering infrastructure faces severe operational collapse unless maintenance workflows are automated, as agents dramatically accelerate pipeline sprawl. - [01:53:32](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6812s) Traditional data warehouses and lakehouses hit a latency floor around 1 second, historically forcing engineering teams to construct fragile, redundant serving tiers. - [01:57:54](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7074s) The Reyden analytical engine was developed by training ML telemetry models on zettabytes of execution traces to dynamically optimize runtime query execution paths down to tens of milliseconds. - [02:12:01](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7921s) Lakebase decouples PostgreSQL compute from elastic lake storage using SafeKeepers with Paxos consensus and dedicated Page Servers to achieve sub-second branching and scale-to-zero suspension. - [02:25:14](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8714s) Frontier model development operates on an accelerated monthly release cadence, shifting engineering focus from model pre-training to grounding agents in enterprise workflows. - [02:38:40](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9520s) Current frontier models exhibit jagged intelligence, demonstrating world-class performance on complex academic benchmarks while remaining brittle on basic enterprise operations. - [02:45:18](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9918s) LTAP resolves the historic failure of HTAP systems by using idle storage node CPU capacity for inline conversion to open columnar Parquet, avoiding compute interference on OLTP workloads. - [02:57:08](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=10628s) Databricks framed its overarching architecture as unifying isolated enterprise data silos into a single 'Pangea' spanning operational, analytical, and agentic workloads.

## Detailed walkthrough ### [00:08](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8s) Opening Video: The Isolated Brain and Enterprise Context

The keynote opens with an atmospheric cinematic video exploring the current state of artificial intelligence. An evocative voiceover describes how humanity succeeded in building an extraordinary artificial brain, only to trap it in isolation—capable of answering virtually any theoretical inquiry, yet fundamentally divorced from real-world business operations.

To bridge this gap, the narration emphasizes that organizations must ground this brain in what is tangible and genuine across enterprise life: every decision, transaction, operational signal, discovery, and commitment made to customers. The core message frames context as the missing ingredient, while insisting that enterprise autonomy and security must remain paramount. Organizations must hold total governance over what the artificial brain learns, the systems it can reach, who is authorized to interface with it, and what actions it can execute across diverse infrastructures. The real technological frontier is not merely possessing an advanced model, but transforming operational systems, pioneering discoveries, and driving human impact with it.

[01:42](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=102s) Welcome to the Largest Data + AI Summit

Databricks co-founder and CEO Ali Ghodsi steps onto the stage to enthusiastic applause, noting that the opening montage featured authentic customer footage and use cases from across the Databricks ecosystem. Reflecting on the scale of the gathering, Ghodsi highlights that this event marks the largest Data + AI Summit in the company's history. Just days prior to the conference, registration stood at approximately 97,000 attendees, but momentum pushed total registrations beyond the 100,000 milestone.

Ghodsi expresses particular pride in the international breadth of the audience, pointing out that participants represent 174 countries and a wide tapestry of languages. In-person attendance reached an exact count of 31,309 people on site, cementing its standing as the preeminent data and AI gathering worldwide.

- Registrations surpassed 100,000 individuals (up from 97,000 just days earlier). - 174 countries represented by global attendees. - 31,309 in-person attendees gathered at the conference.

[03:06](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=186s) Open Source Heritage and the Databricks Ecosystem

Ghodsi traces the summit's identity back to its origins as a grassroots open-source community gathering, recalling its launch as Spark Summit in 2013 centered on Apache Spark. Today, Spark has grown to exceed 3 billion downloads annually. Over the intervening years, Databricks has systematically introduced and supported foundational open-source technologies, including Delta Lake, Apache Iceberg compatibility, MLflow, and the open-sourcing of Unity Catalog.

In particular, Ghodsi highlights Databricks' deep institutional commitment to the PostgreSQL ecosystem. Emphasizing that Databricks sees Postgres as the architectural foundation for the future of databases, he notes that one-seventh of the global PostgreSQL governance board and over 6% of the project's active committers are now employed at Databricks. He also reveals that over the preceding weekend, Databricks co-founder Matei Zaharia open-sourced Omnigent, a new meta-harness designed for orchestrating AI agents. Beyond internal projects, the conference features leaders from across the AI landscape—including OpenAI co-founder Greg Brockman, Microsoft CEO Satya Nadella, and executives from Anthropic, Cognition, Glean, Decagon, Agno, LlamaIndex, and CrewAI—alongside systems integrators, builders, and data-sharing partners exhibiting in the conference expo hall.

```mermaid flowchart TD accTitle: Evolution of Databricks Open Source Ecosystem accDescr: Diagram illustrating Databricks open-source milestones from Spark in 2013 to Postgres and Omnigent.

subgraph Heritage["Core Data Foundations"] SPK["Apache Spark (2013)<br>3B+ annual downloads"] DL["Delta Lake & Iceberg"] MLF["MLflow & Unity Catalog"] end

subgraph ModernStack["Next-Gen Database & Agentic AI"] PG["PostgreSQL Embracement<br>1/7 of governors & 6% of committers"] OMNI["Omnigent Meta-Harness<br>(Open-sourced by Matei Zaharia)"] end

Heritage --> ModernStack ```

[05:07](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=307s) Inspiring Customer Transformations: Insulet, Tonal, and Merck

Ghodsi shifts focus to the real-world achievements of enterprise customers, underscoring that practical innovation on the Databricks platform is propelled by practitioners solving complex challenges. To illustrate the tangible impact among tens of thousands of deployed workloads, he highlights three standout organizations:

First, Ghodsi highlights Insulet and its Omnipod system, a life-changing medical device for individuals managing diabetes. The device pairs a continuous glucose monitor with real-time data and predictive AI on Databricks to monitor blood sugar dynamics and automate personalized insulin delivery.

Second, he shares an internal favorite among early Databricks employees: the smart home-fitness company Tonal. The wall-mounted digital resistance machine captures 100 billion seconds of workout telemetry annually. Using Databricks, Tonal processes this vast stream of biometric and strength data to deliver individualized personal training, dynamic load adjustments, and targeted dietary guidance.

Finally, Ghodsi showcases Merck's biomedical breakthrough known as TEDD (Transformer-Enabled Drug Discovery). Built and trained in close collaboration with the Databricks research team across more than 100 million biological cells, TEDD applies transformer-based language model architecture—ordinarily used for next-token text prediction—to gene regulatory networks instead of human prose. By modeling cellular dynamics, TEDD accurately differentiates causal disease drivers from mere reactive cell states, dramatically accelerating the drug discovery pipeline.

[07:47](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=467s) Has AGI Arrived? Intelligence Versus Enterprise Context

Turning to the central question occupying the modern technology landscape, Ghodsi pauses the presentation to conduct an informal poll with the audience on whether Artificial General Intelligence (AGI) has already arrived. Scanning the auditorium, he observes that only about 5% of attendees raise their hands, while 90% to 95% indicate that AGI remains unachieved.

To challenge that prevailing skepticism, Ghodsi displays an ultra-specialized mathematical problem on screen:

> "Compute the reduced 12-dimensional spin bordism of the classifying space of the Lie group G2."

When he invites anyone in the hall who knows the solution to shout it out or take the stage, only a solitary hand goes up. Ghodsi points out that for state-of-the-art frontier AI systems and reasoning agents, answering this problem is essentially trivial. It forms part of "Humanity's Last Exam," a rigorous multi-subject benchmark comprising 2,500 expert-level questions spanning complex domains like advanced mathematics and specialized visual recognition. Current leading frontier models already answer roughly half of that benchmark correctly.

Recalling his arrival in the United States in 2009 to conduct research at UC Berkeley's AMPLab (where the "A" stood for Algorithms and AI), Ghodsi notes that by the academic criteria researchers established at the time, today's models have surpassed early definitions of AGI by leaps and bounds. Former AMPLab colleagues readily agree that the benchmark of that era has been cleared, even as industry goalposts continue to shift. However, raw reasoning capacity is not what is holding businesses back. Explaining why raw intelligence alone is not transforming corporate workflows, Ali Ghodsi asserts: “AI does not have an intelligence problem right now.” The true bottleneck is that advanced agentic intelligence has not yet permeated the enterprise, because models remain disconnected from operational systems and corporate context.

[10:37](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=637s) The Vision vs. Reality of AI and the Context Problem

Ali Ghodsi observes that while the industry constantly hypes an autonomous future—where AI agents collaborate, draft proposals, negotiate with one another, and humans simply manage them—everyday reality is far more mundane. Most professionals today simply type prompts into chatbots or use coding assistants to generate snippets. The grand vision has not yet been realized across organizations, which explains why so few hands go up when people ask if AGI has arrived. The central theme of the summit is bridging that divide and enabling practitioners to lead an enterprise AI transformation: “It's basically our jobs here to figure out, how can we actually make this AI work?”

The obstacle is not model capability. With both frontier models and open-source models rapidly evolving, the core challenge is feeding those systems the rich operational context trapped across corporate systems and employee heads. Enterprise tasks rarely involve extreme theoretical complexity; instead, they center on everyday workflows like extracting records from Salesforce, summarizing them, shifting them across systems, and preparing reports. When provided with deep business context, existing models are already capable of solving these problems. Yet supplying that context reliably remains an elusive, profoundly difficult undertaking that Databricks has made its primary focus.

[12:24](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=744s) Four Pillars of Enterprise AI: Context, Control, Cost, and Choice

Unlocking context requires systematically addressing four intertwined operational challenges. Ghodsi frames enterprise context as far broader than isolated text prompts: organizations must capture everything from meeting transcripts to disparate operational silos. Databricks even dispatches forward deployed engineers to help organizations restructure internal workflows to make their operational data AI-ready. However, once access to enterprise data is established, governance becomes paramount. Ghodsi points out that when tools like OpenClaw emerged, over 10% of their community-contributed skills were found to be malicious. Unleashing unvetted agents across corporate data stores without strict security guardrails, auditing mechanisms, and policy enforcement creates unacceptable organizational risk.

```mermaid flowchart TD accTitle: Four Pillars for Enterprise AI accDescr: Sequential prerequisites required to successfully implement AI within an enterprise environment.

subgraph Foundation ["Enterprise Prerequisites"] A["1. Context: AI-Ready Data"] B["2. Control: Security & Policy Auditability"] C["3. Cost: Sustainable Inference Budgets"] D["4. Choice: Multi-Vendor Open Stack"] end

A --> B B --> C C --> D D --> E["Enterprise AI Impact"] ```

Beyond security, runaway compute expenses threaten project survival. When agents operate in loops and continuously query data lakes, inference and infrastructure costs rapidly spiral out of control. Ghodsi recalls how Uber's CEO publicly noted that their organization burned through an entire annual AI budget within a single quarter: “You cannot have your costs completely outrun your revenue. You will go bankrupt. And your companies will not allow that. So, they'll just shut it down or stop it.” Finally, Ghodsi highlights vendor lock-in as the most critical challenge. Established enterprises—many operating for 20, 50, or over 100 years—suffer from complex software sprawl accumulated across decades. Organizations cannot afford to tie their proprietary context and workflows to a single proprietary AI model; they require open systems that preserve long-term flexibility.

- **Malicious Agent Skills**: Over 10% of skills in OpenClaw were identified as malicious upon release, underscoring the necessity of enterprise policy and control frameworks. - **Budget Runaway**: Companies risk having leadership shut down initiatives entirely when iterative agent workflows exhaust annual budgets within months. - **Legacy Stacks**: Over 90% of the audience represents legacy companies with deep software stacks where ripping and replacing rigid proprietary systems is notoriously difficult.

[16:09](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=969s) Building the Open Data Foundation: Lakeflow, ZeroBus, and Spark RTM

To deliver choice and eliminate lock-in, organizations must start with open data infrastructure across any cloud environment. Databricks has significantly accelerated ingestion into the open lakehouse using Lakeflow. Whereas Databricks offered minimal turnkey connectors two years prior, Lakeflow now features over 100 prebuilt connectors spanning SaaS platforms (Salesforce, Workday, NetSuite), digital marketing engines (Meta, Google Analytics), and unstructured document corpora.

```mermaid flowchart LR accTitle: Lakeflow Ingestion Ecosystem accDescr: Architecture of data ingestion into the open lakehouse across multiple ingestion modalities.

subgraph Sources ["Upstream Sources"] S1["SaaS Apps: Workday, Salesforce"] S2["Streaming & Event Ingress"] S3["Documents & Analytics"] end

subgraph Modalities ["Lakeflow Engine"] Z["ZeroBus: Serverless High-Rate Ingestion"] R["Spark Real-Time Mode: Sub-10ms Streaming"] D["Lakeflow Designer: Visual AI Pipelines"] end

subgraph Storage ["Unified Foundation"] L["Open Lakehouse Storage"] end

S1 --> D S2 --> Z S2 --> R S3 --> D Z --> L R --> L D --> L ```

Ghodsi highlights three major ingestion capabilities that have reached General Availability (GA): - **ZeroBus**: Replaces legacy event brokers like Apache Kafka. Instead of maintaining dedicated streaming bus clusters, teams hit an API endpoint that handles bursty, high-frequency, narrow-row ingestion without generating micro-file fragmentation or requiring custom client-side buffering. - **Spark Real-Time Mode (RTM)**: An open-source streaming enhancement that pushes Apache Spark processing latencies down into the tens of milliseconds, eliminating the traditional ~1-second micro-batching barrier and offering a unified open-source alternative to dedicated streaming engines like Apache Flink. - **Lakeflow Designer**: A visual, drag-and-drop pipeline interface reminiscent of Alteryx, powered by Genie. Users specify transformations or joins through natural dialogue with the AI coworker, while the engine emits fully inspectable, version-controlled, production-ready open-source Apache Spark code.

[19:23](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1163s) Open Table Format Unification with Ryan Blue

Ali Ghodsi addresses one of the most contentious debates in the data industry: choosing between Delta Lake and Apache Iceberg. To resolve the debate, he invites Ryan Blue, the original creator of Apache Iceberg and co-founder and former CEO of Tabular (which Databricks acquired the previous year), onto the stage. Ali playfully teases Ryan by recalling a past video presentation titled "Why you shouldn't care about Apache Iceberg." Ali plays a clip where Ryan candidly declares:

> “Um and I I sincerely mean that. Um you shouldn't care about Apache Iceberg.”

Ali agrees completely, adding that users shouldn't have to stress over Delta Lake either. Ryan elaborates on the philosophy behind that statement, explaining that practitioners should be free to run whichever computational engines they prefer without getting bogged down by raw storage representations:

> “No one should have to care about formats. You you should be able to use whatever tools you want with all of your data.”

Ali reflects on past customer meetings where conversations routinely degenerated into low-level architectural debates—such as differing deletion vector implementations—which he felt pulled the entire industry in the wrong direction. Ryan provides a status update on format convergence, noting that managed Apache Iceberg tables with Iceberg v3 support are officially generally available (GA) within Databricks Runtime (DBR). In Iceberg v3, the data layer itself is fully unified, meaning data files no longer need to be duplicated or rewritten when accessed across Delta Lake and Iceberg environments. The remaining step is converging the metadata layer, planned for Delta Lake 5 and Apache Iceberg v4, with specification completion targeted for around Q4 of the current year.

```mermaid flowchart TD accTitle: Unified Open Table Format Architecture accDescr: Convergence path uniting Iceberg and Delta Lake across storage, engines, and metadata layers.

subgraph Formats["Unified Storage Formats"] V3["Iceberg v3 / Delta<br/>Unified Data Layer (No Rewrites)"] V4["Iceberg v4 / Delta 5<br/>Unified Metadata Layer (Late Year)"] end

subgraph Engines["Unified Access"] Spark["Apache Spark & DBR Engine"] EnginesList["Multi-Engine Tools<br/>(SQL, ML, Agents)"] end

V3 --> EnginesList V4 --> Spark ```

- **Iceberg v3 GA in DBR:** Introduces a shared, unified data layer so files are identical on disk between Delta Lake and Iceberg, eliminating data file rewrites. - **Iceberg v4 & Delta Lake 5:** Will deliver a unified metadata layer to finish the full unification vision, targeted for late 2026 / Q4 spec finalization.

---

[22:14](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1334s) The Lakehouse Data Warehouse, SQL AI Functions, and Reyden

Ali emphasizes that Databricks users no longer have to worry about underlying table formats: everything is built on standard Apache Spark execution and unified storage. He pivots from data ingestion to analytical processing on the Lakehouse, which Databricks introduced four to five years earlier under the premise that “the best data warehouse is a lakehouse” because open architecture preserves data ownership while natively powering machine learning workloads. Over the prior year, Databricks added more than 110 enterprise features to ease migrations from legacy proprietary data warehouses.

The most widely adopted capability in the lakehouse warehouse has become SQL AI functions. Data teams can directly invoke native generative AI functions inside their standard SQL queries against relational tables, executing operations at massive scale without managing external pipelines: - Running automated sentiment analysis across millions of text cells. - Extracting named entities such as company names from unstructured columns. - Generating predictive values inline.

Ali notes that consumption of the Databricks Lakehouse data warehouse has more than doubled over the previous year, aided by LakeBridge—an automated migration converter that migrates on-premises and legacy cloud warehouses seamlessly. He then teases an upcoming deep dive by co-founder and Chief Architect Reynold Xin into a brand-new, cutting-edge query engine codenamed Reyden. Ali highlights Reyden as the fastest analytical engine the company has ever benchmarked, slashing query latency down to tens of milliseconds.

---

[24:26](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1466s) Lakebase: Serverless Postgres and Zero-Copy Branching for Agents

Turning to online transaction processing (OLTP), Ali explains why Databricks has embraced PostgreSQL. When modern AI systems and LLM coding assistants are asked to choose an operational datastore, they overwhelmingly select open-source Postgres. To bring transactional workloads into the lakehouse, Databricks developed Lakebase—an architecture that decouples Postgres compute from storage, persisting all transactional data directly onto low-cost object lakes in open formats.

Lakebase introduces sub-second serverless auto-scaling that scales up under load and scales down to zero when idle, drastically reducing the total cost of ownership (TCO) because organizations only pay for cheap object storage during periods of inactivity.

Beyond serverless scaling, Ali points out the defining capability engineered specifically for AI agents: instant database branching. Rather than requiring DBAs to provision and maintain thousands of distinct database silos across an enterprise for staging, R&D, user acceptance testing (UAT), and production, Lakebase enables sub-second zero-copy branching using copy-on-write technology:

```mermaid flowchart LR accTitle: Lakebase Zero-Copy Branching Flow accDescr: Single base lake storage generating instant isolated branches for production, testing, and AI agents.

LakeStorage[("Lakebase Base Storage<br/>(Petabyte-scale on Lake)")]

subgraph Branches["Sub-Second Branches (Copy-on-Write)"] Prod["Branch: Production"] UAT["Branch: UAT / Staging"] Agent["Branch: Autonomous Agent"] end

LakeStorage --> Prod LakeStorage --> UAT LakeStorage --> Agent ```

Ali explains that autonomous agents thrive on this architecture because they require rapid iteration without waiting ten minutes for a database instance to spin up: - Agents can branch a multi-petabyte database in under a second, test modifications or run sandboxed code, and cleanly discard the branch. - If an agent generates corrupt data or introduces errors, the workspace can immediately revert back to an earlier immutable snapshot.

Ali frames this breakthrough within the broader macro transformation occurring across software development:

> “In the last in the next 12 months, we're going to see more software be written than ever in the history of mankind.”

Because that unprecedented wave of AI-generated applications will all require a backing operational datastore, Ali urges teams to standardize on Lakebase Postgres. He concludes by previewing another major announcement slated for later in the keynote: Reynold Xin returning to present LTAP (lake transactional/analytical processing), unifying Lakebase and the analytical Lakehouse into a single cohesive platform.

[28:11](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1691s) Unifying the Agentic Data Layer: Lakeflow, Lakehouse, and Lakebase

Ali Ghodsi summarizes the foundational components required to support autonomous systems before turning to governance and control. A major industry breakthrough—addressing a challenge database engineers have tackled for forty years—will be detailed later in the keynote by co-founders, but the overarching architecture has taken shape: Lakeflow handles high-throughput ingestion to pull raw data in, Lakehouse provides the analytical data warehouse engine, and Lakebase powers operational transaction processing.

Together, these pillars establish an open, unified bottom tier that agentic software can interact with directly. For AI agents to work reliably, they need high-speed, seamless access to clean, fresh data so they can rapidly experiment and execute actions without hitting ingestion bottlenecks or siloed architectures. As Ghodsi emphasizes, “We got to get that data layer right.”

```mermaid flowchart TD accTitle: The Agentic Data Foundation accDescr: Flow showing data flowing from Lakeflow ingestion through Lakehouse analytics and Lakebase transactions to AI agents.

subgraph DataIngest["Data Ingestion"] LF["Lakeflow<br/>(Ingest Data)"] end

subgraph StorageEngines["Unified Storage & Processing Layer"] LH["Lakehouse<br/>(Analytics & Warehousing)"] LB["Lakebase<br/>(Transaction Processing)"] end

subgraph AgentLayer["Consumption"] AG["AI Agents & Coworkers<br/>(Fast Experimentation & Action)"] end

LF --> LH LF --> LB LH --> AG LB --> AG ```

---

[28:44](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1724s) Unity Catalog and Open Sharing: Expanding Beyond Structured Tables

Getting the data layer right is only half the battle; enterprises cannot simply unleash autonomous agents across their corporate repositories without stringent guardrails. Ghodsi reflects on the origins of the Unity Catalog project, contrasting its founding philosophy with legacy catalog architectures. Historically, data warehouse vendors built catalogs strictly around relational rows and columns, focusing narrowly on masking sensitive fields like an employee's salary. Databricks purposefully named their project Unity because governance must span far beyond tabular databases to encompass unstructured files, PDFs, machine learning models, and over fifty additional asset types introduced over the years.

Furthermore, governance is not merely about locking assets away. A complete catalog must enable self-service discovery, track end-to-end data lineage across graphical visualizers, manage infrastructure costs, and audit data quality across the entire enterprise. Underscoring Databricks’ open-ecosystem stance, Ghodsi highlights that Unity Catalog is completely free and open-sourced, with significant new external contributors joining in recent weeks.

Building on that open foundation, Ghodsi announces a major evolution in data sharing. While legacy platforms locked customers into proprietary walled gardens—requiring external partners to purchase the same software license just to exchange a table—Databricks previously introduced Delta Sharing. Now, Databricks is launching Open Sharing, a direct superset of Delta Sharing. Because Delta Lake and Apache Iceberg have converged into unified table formats via UniForm, Open Sharing permits frictionless sharing across both formats without vendor lock-in. Crucially, Open Sharing extends beyond data tables to govern and distribute AI assets—including agents, specialized skill files, and trained models—operating natively across multicloud environments and on-premises deployments.

“So, that’s what this was about. Access control on structured data like that. But, we called it Unity because we think you shouldn’t just do control on structured data, you should do it on all the data assets that you have in an organization, any data and AI assets.” — Ali Ghodsi

- **Unity Catalog Core Capabilities:** Cross-asset access control, data and model discovery, visual lineage graphs, cost monitoring, and automated data quality checks. - **Open Source Commitment:** Unity Catalog remains free, with key components continuously released as open source. - **Open Sharing Scope:** Extends sharing to both Delta Lake and Apache Iceberg formats, supporting on-premises infrastructure as well as cloud-native AI assets (models, agents, skills).

---

[31:18](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=1878s) The AI Governance Quagmire: Runaway Costs, Security Chaos, and Model Churn

Over the past year, enterprise AI governance has spiraled into chaos. Development teams are rapidly prototyping custom agents using frameworks like LangChain and LlamaIndex, while SaaS vendors bombard businesses with proprietary agent builders, bespoke agents, and Model Context Protocol (MCP) servers. Internal developers are compiling their own MCP tools and skills files, while software teams adopt vibe coding and agentic co-workers across every department. The result is an unmanaged operational mess marked by three critical pain points:

1. **Runaway Costs:** Uncontrolled “token maxing” and model invocation across disparate teams are driving infrastructure bills through the roof, with leadership having zero visibility into which agents or models are driving the spike. 2. **Missing Controls and Guardrails:** Security teams lack basic auditing mechanisms, identity tracking (determining on whose behalf an agent is acting), and robust input/output guardrails to verify that agents only access authorized corporate data. 3. **Hyper-Accelerated Model Churn:** Vendor lock-in is disastrous because state-of-the-art models become obsolete within weeks. Ghodsi walks through the dizzying turnover of frontier models: Google Gemini reigned supreme in November, Claude Opus overtook the benchmark lead in January and February, OpenAI countered with GPT-5.5, followed by brief excitement over Mythos and Fable.

“Because the state of the art models now have a lifespan of 1 month.” — Ali Ghodsi

---

[33:50](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2030s) Announcing Unity AI Gateway: Multi-AI Governance and Cost Control

To resolve this governance crisis, Ghodsi announces Unity AI Gateway, a centralized control plane integrated into Unity Catalog that provides a single pane of glass to administer all enterprise AI agents, tools, and token expenditures. Inheriting the open-source ethos of Unity Catalog and MLflow, Unity AI Gateway is free of software licensing fees.

A major highlight of the gateway is Databricks' flexible frontier capacity program. Enterprise customers who commit budget to Databricks—such as a $100,000 annual spend—are not limited to using those funds exclusively on Lakeflow pipelines or Lakehouse compute. Customers can draw down that exact same financial commitment to purchase API tokens directly for frontier models from OpenAI, Anthropic, or Google Gemini across AWS, Microsoft Azure, and Google Cloud Platform.

Beyond universal model procurement, Unity AI Gateway delivers comprehensive FinOps and safety controls. Administrators gain fine-grained cost observability with real-time dashboards to establish hard budgets, soft alerts, and rate limits at the department, team, or individual developer level. It acts as an enforcement layer for enterprise safety, compliance auditing, input/output guardrails, and agent identity management. Furthermore, the gateway integrates third-party tools by allowing organizations to register any external MCP server, skill file, or model. Rather than forcing users and agents to authenticate repeatedly across fragmented SaaS tools, Unity AI Gateway features single sign-on authentication across all registered MCP tools, unifying security, cost control, and model choice in a single operational hub.

“That’s why we’re super excited to announce as part of Unity Catalog uh we are breaking out one place where you can manage and control all of your agents, all of your AI spend. We call it Unity AI Gateway.” — Ali Ghodsi

- **Unified Billing & Frontier Capacity:** Draw down existing Databricks monetary commitments directly for OpenAI, Anthropic, and Gemini tokens across all major hyperscalers (AWS, Azure, GCP). - **FinOps & Observability:** Real-time visibility into AI token consumption with automated rate-limiting, daily usage emails, and hierarchical budget caps (organization, group, and user levels). - **Centralized MCP Registry:** Universal registration and usage tracking for any first-party or third-party MCP server, featuring single-point authentication across all external tools. - **Enterprise Safety & Identity:** End-to-end audit logging, input/output guardrail enforcement, and identity propagation to verify on whose authority an autonomous agent executes commands.

[36:44](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2204s) The Flaws of Frontier AI Agents: Latency, Cost, and Accuracy

Ali Ghodsi reiterates that the core issue facing enterprise AI is not reasoning capability, but access to relevant context: “AIs are plenty smart. AI does not have an intelligence problem, it has a context problem.”

Today’s leading coding and knowledge agents operate via live traversal loops across enterprise assets. When a user asks an analytical question—such as checking product revenue trends in Europe over the prior two weeks—an agent navigates an unstructured graph of corporate documents in real time. It opens an MCP (Model Context Protocol) server connection, reads a single file, discovers a link, initiates another call, and repeats the cycle in an iterative loop. Ghodsi outlines three fatal flaws in this operational model:

1. **Extreme Latency:** Conducting live traversals during a query execution causes even simple requests to stall for 10 to 15 minutes because the agent lacks an upfront understanding of where relevant information resides. 2. **Exorbitant Token Costs:** Continuous live exploration consumes massive token counts. Ghodsi highlights the absurdity with an industry quip: “The joke is you tell it, 'Please rename this file for me.' And it takes 10 minutes, and it charges you five bucks.” 3. **Degraded Output Quality:** Enterprise environments contain vast oceans of unstructured and structured data. An agent executing a live traversal can only execute a narrow, random walk through a tiny fraction of the organization's corpus, leading to low overall retrieval accuracy.

---

[38:50](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2330s) Announcing Genie Ontology and the OntoRank Algorithm

To overcome the latency and cost barriers of live document navigation, Databricks introduces **Genie Ontology**, a continuously updated context graph for enterprise knowledge. Rather than traversing files on the fly during user queries, Genie Ontology operates continuously in the background. It ingests data from Databricks lakehouses as well as enterprise workspace tools—including Google Drive, SharePoint, corporate email, and Google Calendar—and organizes internal knowledge into a structured semantic layer.

```mermaid flowchart TD accTitle: Genie Ontology Ingestion Architecture accDescr: Data sources feed into the OntoRank algorithm to build the Genie Ontology Context Graph, which powers AI agents.

subgraph Sources["Enterprise Assets & Systems"] LHouse["Databricks Lakehouse"] GDrive["Google Drive & Calendar"] SPoint["SharePoint & Email"] OrgChart["User Access & Org Charts"] end

subgraph Engine["Ontology Engine"] ORank["OntoRank Algorithm"] Graph["Genie Ontology Context Graph<br/>(Background Indexing)"] end

subgraph Agents["Downstream Consumers"] GOne["Genie One"] GAgents["Genie Agents / Genie Code"] MCPAgents["External Agents via MCP"] end

LHouse --> ORank GDrive --> ORank SPoint --> ORank OrgChart --> ORank ORank --> Graph Graph --> GOne Graph --> GAgents Graph --> MCPAgents ```

Underpinning the graph is **OntoRank**, an indexing algorithm developed by Databricks Research. Ghodsi compares it to Google's foundational web algorithm: “PageRank was how Google 25 years ago or 30 years ago figured out how to take the whole web and create an index for what are the most important sites.”

While early search engines relied on crude keyword frequencies that invited SEO keyword stuffing, PageRank leveraged link topology. However, OntoRank tackles a significantly harder problem than indexing uniform web pages: - It processes diverse asset types (such as source code versus formatted documents). - It factors in organizational hierarchies, access frequencies, and author context.

Organizations that have existing business glossaries or semantic layers can integrate them directly. Through **Unity Catalog Semantics**, companies can import metadata from partners like Atlan and AtScale, or leverage existing BI semantic definitions alongside the automated OntoRank graph. Third-party agents can also consume this context by connecting to Genie as an MCP server. Ghodsi notes that Databricks' own internal deployment spans more than 4.5 million curated ontology data snippets—an asset volume impossible for live agents to parse on the fly.

---

[43:16](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2596s) Introducing Genie One: The Universal Enterprise AI Coworker

Databricks introduces **Genie One**, expanding beyond domain-siloed spaces (such as marketing-only Genie rooms) into a company-wide interface for business analytics. Designed for broad corporate rollouts, administrators can enable automatic identity management across an entire enterprise—even for workforces of 100,000 employees. Non-technical users interact with a dedicated, lightweight workspace without needing direct access to manage or spin up costly data warehouses.

Genie One leverages the Genie Ontology context graph to query structured and unstructured corporate assets. It supports operational task automation, skills scheduling, and AI/BI dashboard integrations. Ghodsi highlights that the interface is fully optimized for mobile devices (iOS and Android), providing visual chart renderings directly within the workflow.

- **Enterprise Governance:** End-users query data directly without provisioning backend data warehouse infrastructure. - **Unified Context:** Queries resolve across lakehouse data, Google Drive, SharePoint, and external SaaS silos. - **Mobile AI/BI Integration:** Formatted visualizations and dashboards render natively on mobile and desktop clients.

---

[45:02](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2702s) Announcing Genie Agents: Turning Conversations into Reusable Tools

Databricks also unveils **Genie Agents**. This capability allows users to take any completed multi-turn analytical conversation within Genie One and convert it into a permanent, reusable agent that can be shared across the entire enterprise.

[45:16](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2716s) Deploying Autonomous Genie Agents Across the Enterprise

Organizations can take internal materials—such as HR documentation—and spin up dedicated conversational agents that answer employee queries accurately. Once verified, these agents can be deployed enterprise-wide across everyday communication channels like Slack or Microsoft Teams.

Beyond conversational Q&A, these agents perform autonomous, cross-system workflows without manual intervention. For instance, an agent can automatically extract operational records from Salesforce each day, prepare and transform the data, load it into Workday, deliver structured executive summaries, and collaborate directly with downstream specialized agents.

[45:48](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2748s) Genie Code: Data Engineering and Machine Learning Automation

Databricks introduces Genie Code, a programming agent built directly upon 13 years of data engineering expertise on the Databricks platform. Genie Code is designed around two specialized enterprise capabilities: data engineering pipeline creation and machine learning workflow automation.

For machine learning and data science, Genie Code handles the end-to-end lifecycle. It writes the modeling code, trains models directly on GPU infrastructure, evaluates accuracy metrics, and democratizes advanced data science across non-specialist teams.

- Automatically writes, refactors, and deploys complex data engineering pipelines. - Provisions GPU compute to train, validate, and benchmark enterprise ML models.

[46:32](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2792s) Genie ZeroOps: Autopilot for Pipeline Maintenance and Remediation

Data pipeline failures routinely disrupt operations during off-hours, requiring on-call engineers to spend hours diagnosing breakages under pressure. To eliminate this burden, Databricks is launching Genie ZeroOps, an autonomous operational agent that monitors production data pipelines and machine learning workloads in the background. Ali Ghodsi highlights its transformative potential:

> “I think this agent actually is going to be the big biggest transform transformative agent for most of your organizations. We call it Genie Zero Ops.”

When a pipeline fails—such as encountering an unmapped column at 2:00 a.m.—Genie ZeroOps investigates the root cause, inspects the schema delta, and synthesizes a patch. Crucially, it executes and tests the proposed fix inside an isolated sandbox environment rather than mutating production state directly. Once verified, it generates a mobile push notification, allowing the engineer on call to inspect the change and resolve the incident with a single click. While fully automated remediation is technically possible, keeping a human in the loop remains recommended.

```mermaid flowchart TD accTitle: Genie ZeroOps Incident Remediation Flow accDescr: An automated incident loop where Genie ZeroOps detects pipeline failure, tests fixes in a sandbox, and requests human approval.

subgraph Detection["Failure Detection"] A["Production Pipeline Fails (e.g. 2:00 AM)"] --> B["Genie ZeroOps Background Monitor"] B --> C["Root Cause Analysis: New/Unknown Schema Column"] end

subgraph Remediation["Isolated Patching"] C --> D["Synthesize Code & Schema Patch"] D --> E["Test & Validate in Isolated Sandbox"] end

subgraph Resolution["Human Verification"] E --> F["Mobile Push Notification to On-Call Engineer"] F --> G{"Engineer Review"} G -->|"Click Accept"| H["Deploy Fix to Production Pipeline"] G -->|"Reject / Modify"| I["Manual Intervention"] end ```

[47:51](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2871s) Genie App Builder, Agent Bricks, and the Omnigent Meta-Harness

To democratize application development, Databricks enables users to "vibe code" full business applications. Users can interface with external ecosystem partners like Lovable, Replit, or Vercel, or use the newly integrated Genie App Builder directly on top of the Genie Ontology. The platform connects not only to data housed within the lakehouse, but to all data distributed across the enterprise.

For professional software engineers seeking fine-grained control over coding agents and harnesses (such as Claude Code, Codex, or OpenCodeInterpreter), Databricks provides Agent Bricks. This developer-centric environment includes low-latency execution sandboxes for strong agent isolation, dedicated agent memory, and a new meta-harness named Omnigent. Unveiled by Databricks co-founder Matei Zaharia, Omnigent acts as an orchestrator of harnesses, pitting underlying coding agents against one another (e.g., orchestrating Claude Code against Codex) to produce results that surpass any single agent acting independently.

- **Genie App Builder:** Native platform tool to vibe-code business applications integrated with the context graph. - **Agent Bricks:** Low-level developer platform providing fast, isolated sandboxes and persistent agent memory. - **Omnigent:** A meta-harness capable of running multiple coding agents simultaneously to generate optimal code solutions.

[49:20](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2960s) The Agentic System of Record: Redefining Enterprise SaaS Architecture

The software-as-a-service (SaaS) architecture of the past two to three decades relied on transactional databases acting as systems of record for discrete departments (e.g., Salesforce for sales, Workday for HR, ERPs for finance). These systems relied on rigid, contractor-customized workflows and static user interfaces. While SaaS vendors now place proprietary agents on top of their systems of record, enterprise inquiries inherently span departmental silos. Because competing vendors do not readily share access, cross-system operations become brittle, complex, and uncoordinated.

```mermaid flowchart TD accTitle: Future Enterprise Agentic Stack Architecture accDescr: Diagram illustrating the modern enterprise software stack where agents and apps access an agentic system of record powered by open data, governance, and business context.

subgraph Consumption["Application & Interaction Layer"] Apps["Agentic Databricks Apps (Vibe-Coded or Custom)"] Agents["Autonomous Agents (Sales, Ops, Finance, HR)"] end

subgraph Platform["Agentic System of Record"] Context["Genie Ontology (Cross-Enterprise Context Graph)"] Governance["Unified Governance & Cost Controls (Unity Catalog / AI Gateway)"] Storage["Unified Open Table Storage (Lakehouse)"] end

Apps --> Platform Agents --> Platform Context --> Governance Governance --> Storage ```

Databricks asserts that the enterprise software stack must transition toward an "agentic system of record." Rather than querying operational databases directly—which introduces performance and availability risks—agents must run against a unified data and AI platform that exposes open storage formats, centralized cost and governance controls, and comprehensive business context via the Genie Ontology. On top of this foundation sit Databricks Apps, enabling companies to roll out bespoke, vibe-coded applications to hundreds of thousands of employees. Ali Ghodsi summarizes this architectural shift:

> “We think that the new system of record is actually the data and AI platform that we've been talking about all along.”

[53:24](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3204s) Databricks Marketplace and Lakewatch: An Agentic SIEM

Ali Ghodsi highlights the rapid proliferation of applications across the platform, noting that apps have quickly become Databricks’ fastest-growing product category. To support this momentum, Databricks is introducing an integrated marketplace where customers can discover, access, and directly transact to purchase applications built by third-party software vendors, in addition to building custom internal tools. However, Ghodsi emphasizes that there are specialized, highly data-intensive application domains where Databricks holds a distinct point of view and intends to build foundational solutions directly. Chief among these domains is cybersecurity operations.

Ghodsi revisits the recent launch of Lakewatch, an agentic security information and event management (SIEM) system designed from the ground up on the open security lakehouse architecture. He outlines why conventional SIEM architectures are fundamentally broken in the modern threat landscape: legacy vendors charge exorbitant ingestion fees, forcing security teams to pre-filter and discard raw telemetry data just to keep costs manageable. Ghodsi argues that this filtering strategy is disastrous when organizations face automated, agentic threat actors capable of discovering zero-day vulnerabilities at machine speed. He points out that within the past six months, an autonomous agentic system named Expo reached the top tier of the HackerOne leaderboard, while tools like Mythos are actively surfacing novel exploits. Security teams need to retain all raw log and telemetry data indefinitely at low cost in an open lakehouse. On top of this data foundation, Lakewatch deploys autonomous agents that automatically engineer detections, triage the deluge of daily security alerts for security operations center (SOC) analysts, and conduct continuous threat hunting against emerging zero-day vulnerabilities.

```mermaid flowchart TD accTitle: Lakewatch Security Lakehouse Architecture accDescr: Raw telemetry flows into an open security lakehouse where Lakewatch AI agents perform automated triage and hunting.

subgraph Ingestion["Open Data Layer"] A["All Raw Telemetry & Logs"] --> B["Open Security Lakehouse"] end

subgraph Agents["Lakewatch Autonomous Agents"] B --> C["Automated Detection Engineering"] B --> D["SOC Alert Triage"] B --> E["Zero-Day Threat Hunting"] end

subgraph Operations["SOC Output"] C --> F["Continuous Pipeline Updates"] D --> G["Triaged Analyst Queue"] E --> H["Proactive Threat Remediation"] end ```

- Traditional SIEM models break down because companies cannot afford to ingest exploding data volumes into legacy, proprietary repositories. - Modern threats require complete, unfiltered data retention: “So, we need all of the data. It needs to be very cheap. It can't be skyrocketing cost and we should just store it all in any means we have in an open secure lakehouse.” - Autonomous hacker systems such as Expo on HackerOne and tools like Mythos demonstrate that defense must also be automated at machine speed. - Lakewatch acts as a virtual SOC assistant, handling alert triage, proactive threat hunting, and automated pipeline configuration for newly detected attack patterns.

---

[55:58](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3358s) Databricks to Acquire Panther Labs

To accelerate its vision for the modern security operations center, Ghodsi announces that Databricks has reached an agreement to acquire Panther Labs. Built by founder and former Airbnb security analyst Jack Naglieri, Panther pioneered a modern, code-first SIEM architecture centered entirely on Python rather than rigid, proprietary query languages or complex visual drag-and-drop interfaces. Ghodsi recalls meeting Naglieri five years earlier and hearing his contrarian thesis that SOC engineering must become purely Pythonic—a design choice that has proven remarkably ahead of its time as generative AI systems natively express logic and execute tasks in Python.

Panther brings deep production-proven capabilities to Databricks, including hundreds of pre-built telemetry connectors and an extensive repository of Python-based detection logic. The platform is already trusted by leading tech and security-conscious enterprises, including Anthropic, Coinbase, and Plaid. Naglieri is slated to present on stage the following day to showcase the deep integration between Panther and Databricks.

- Ghodsi reflects on Panther’s architectural foresight: “Which turns out to be really prescient because today um all the AIs want to speak Python.” - Jack Naglieri developed Panther based on firsthand frustrations as an Airbnb SOC analyst who rejected complex drag-and-drop security dashboards. - Key enterprise customers running production security workloads on Panther include Anthropic, Coinbase, and Plaid.

---

[57:05](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3425s) CustomerLake: The Agentic CDP and Keynote Wrap-up

Turning to the enterprise marketing stack, Ghodsi announces CustomerLake, an agentic customer data platform (CDP) and Customer 360 engine built natively on the lakehouse architecture. Traditional CDPs rely on rigid, heuristic rules to deduplicate records and group vast audiences into broad segments. CustomerLake introduces specialized autonomous agents to rethink these workflows: a Profile Agent that leverages large language models to execute nuanced identity resolution and deduplication, and a Campaign Agent that powers "infinity campaigns." Rather than splitting customer bases into dozens or hundreds of static demographic segments, infinity campaigns utilize distilled, cost-efficient LLMs to execute continuous, one-to-one personalized interactions at scale. CustomerLake was developed alongside several ecosystem integration partners and development design partners, with deeper technical sessions scheduled later in the summit.

Ghodsi concludes his opening keynote by synthesizing the core pillars of Databricks’ strategic vision. The central challenge of enterprise AI is not a raw intelligence ceiling, but rather bridging the context gap by connecting models to business data, enterprise knowledge graphs, and existing workflows. Delivering on this opportunity requires a platform that guarantees freedom of choice across any data, model, or cloud infrastructure without proprietary vendor lock-in, unifies multi-model governance and cost controls in a single pane of glass, and dynamically delivers enterprise context directly to autonomous agents.

- The Profile Agent inside CustomerLake performs high-accuracy entity resolution and identity deduplication using LLMs. - Infinity campaigns move marketing beyond static batch segmentation: “And infinity campaigns basically let you tell your personalize every interaction that you have.” - Ghodsi summarizes the ultimate architectural objective: “The first one is really about any data, any model, and any cloud without lock-in.”

---

[01:00:48](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3648s) Ken Wong Introduces Genie One: The Context Problem with Current AI Agents

Databricks Senior Director of Product Management Ken Wong takes the stage to introduce Genie One, an enterprise AI coworker engineered to interface with an organization's complete data footprint and software applications to surface analytical insights, automate operational tasks, and build autonomous sub-agents through natural conversation. Anticipating skepticism from an audience inundated with AI marketing, Wong addresses the obvious question: why does the industry need another agent in 2026? He argues that existing consumer and commercial assistants fail when confronted with enterprise data—they lack secure connectors, cannot reason across relational and semi-structured business data, and hallucinate under scrutiny.

To illustrate these shortcomings, Wong shares a real-world test from the day prior, when he prepared an executive briefing for the Databricks Product Advisory Board (PAB). Producing this customer profile required connecting across personal communication channels (Google Calendar, Slack, email) to determine meeting attendees, querying Salesforce for strategic account use cases, and extracting consumption telemetry from Databricks to understand specific Genie adoption patterns. While Genie One synthesized these sources into an accurate, highly visual dossier, Wong submitted the exact same prompt to five leading commercial assistants (two software-bundled assistants and three premier coding and coworker agents). The first bundled assistant produced an answer almost instantly, claiming that exactly 24 customers participated in the PAB. Suspecting an error, Wong asked the assistant where it sourced that number. The assistant candidly replied that it had completely fabricated the metric, underscoring why enterprise teams cannot trust generic AI assistants for high-stakes decision-making.

- Wong pinpoints the fundamental limitation of incumbent market solutions: “The ones that you have today aren't very good with enterprise data. They either can't connect and reason about data at all, or they're just not accurate enough for you to actually trust them to make real decisions, or let alone automating your work.” - Preparing for the Product Advisory Board required querying disparate systems: personal email, calendar, Slack, Salesforce customer accounts, and internal Databricks platform consumption telemetry. - When challenged on its fabricated figure of 24 advisory board accounts, the benchmarked commercial assistant offered no factual defense: “It confessed that it just completely made it up.”

[01:03:46](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=3826s) The Limits of Generic Coding and Co-work Agents

Comparing common AI tools reveals the severe shortcomings of existing approaches when answering enterprise data questions. Standard assistants fall short immediately: one tool simply pulled stale account briefs from an entire quarter earlier that were irrelevant to the current discussion, yielding fast but completely unusable output. On the other hand, generic coding and co-work agents attempted to connect directly to structured data sources and live systems, but quickly hit practical walls. These agents routinely burn through token budgets, halt to request user permission to keep running, enter repetitive loops, and ultimately return little more than partial reports and terminal ASCII art after 30 minutes of computation.

```mermaid flowchart TD accTitle: Generic Coding Agent Failure Loop accDescr: Diagram illustrating the costly and error-prone iterative walk of generic coding agents attempting to answer data questions.

subgraph SearchPhase["Agent Execution Loop"] A["User Asks Novel Data Question"] --> B["Connect to Live Sources"] B --> C["Agentic Random Walk / Reasoning"] C --> D{"Hits Token / Time Limit?"} end

subgraph FailureOutcome["Outcome"] D -- "Yes" --> E["Prompt User to Burn More Tokens"] E --> C D -- "Terminal State" --> F["ASCII Art & Partial Results (~50% Accuracy)"] end ```

This frustration is not an isolated edge case or a cherry-picked scenario; it matches findings from internal Databricks research benchmarks. When researchers tested leading coding agents against real, novel questions asked by Databricks employees—such as determining what percentage of Genie users also employ a third-party agent—the models failed to perform reliably. The speaker emphasizes how catastrophic this failure rate is in a production business context:

> "And remember this chart, we'll come back to it. But 50% of the time for for a real data question is basically unusable, right? You cannot make You literally can flip a coin."

The fundamental problem lies within the classic agentic loop. Agents perform a speculative walk across steps, hoping to chance upon the right reasoning sequence and context. This dynamic creates an inherent trade-off between runtime cost and accuracy. Manually curating context through "skills" or semantic layers fails to scale across an entire enterprise. Every marketing campaign interprets leads differently, and every engineering scrum team customizes its Jira fields; documenting every variation by hand is impossible. Attempting to use generative AI to pre-build these semantic models also backfires: as Anthropic detailed in research, automatically generated semantic models anchor the model to a single narrow use case, ultimately degrading overall accuracy.

- Generic coding agents achieved only ~50% accuracy on real-world employee enterprise data benchmarks. - Unassisted agent reasoning runs frequently exceeded 30 minutes before returning incomplete results. - Anthropic research demonstrated that automated semantic modeling causes over-anchoring, worsening net reasoning performance.

---

[01:07:54](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4074s) Genie Ontology: Automated Enterprise Context via OntoRank

The solution to enterprise grounding is the Genie Ontology, an automated context layer that overlays a dynamically learned tier on top of any existing modeled data in Unity Catalog or external semantic tools. Rather than requiring humans to manually configure brittle rules for every business scenario, Genie Ontology inspects existing enterprise assets—including data pipelines, queries, and business dashboards—to automatically discover operational logic, data expressions, table relationships, and subject-matter expertise.

```mermaid flowchart LR accTitle: Genie Ontology Architecture accDescr: Pipeline showing data assets ingested into OntoRank and surfaced securely to the Genie reasoning agent.

subgraph Ingestion["Asset Ingestion"] A["Pipelines & Queries"] B["Dashboards"] C["User Expertise & History"] end

subgraph Processing["Context Layer"] A & B & C --> D["Internal Knowledge Store"] D --> E["OntoRank Algorithm"] end

subgraph Serving["Runtime Query Execution"] E --> F["Permissions Filter\n(Unity Catalog)"] F --> G["Agentic Reasoning Loop\n(Genie)"] G --> H["Accurate, Verifiable Answer"] end ```

The system processes this accumulated operational data using an internal algorithm developed by Databricks engineers named **OntoRank**. Operating like PageRank for enterprise knowledge, OntoRank ranks disparate code snippets, queries, and calculations to identify which sources are genuinely authoritative. When a user queries Genie—such as asking for real-time registration metrics—the platform retrieves the authoritative calculation directly from the underlying dashboards, verifies the user's access permissions via Unity Catalog so no data leaks, and injects the precise context into the reasoning loop.

> "Well, in our internal testing, what we've seen is a consistent 30 percentage point plus improvement in in accuracy and roughly half the the run time of the leading agents, right?"

- In benchmarks against leading coding agents, Genie Ontology delivered a 30+ percentage point increase in query accuracy. - Execution runtimes were reduced by approximately 50% compared to standard agent architectures. - All extracted knowledge snippets retain underlying source permissions to prevent unauthorized access.

---

[01:11:20](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4280s) Live Demo: Genie One as an Enterprise Coworker

Product Manager Elise Georis joins the stage to demonstrate how the combination of Genie One and the Genie Ontology functions as a full-fledged data coworker for real-world workflows. Demonstrating her preparation for an upcoming executive review on Genie OKRs, Georis shows how the conversational workspace replaces manual cross-platform data gathering. The clean user interface centers on an open chat canvas, quick actions for drafting documents or defining skills, and a sidebar tracking chat histories, team agents, and connected data assets.

```mermaid flowchart TD accTitle: Genie One OKR Review Execution Flow accDescr: Sequence diagram of Genie One executing tasks across Unity Catalog, Jira via MCP, and running SQL queries.

A["User Prompt with Template Link"] --> B["Retrieve Unity Catalog Glossary & Ontology Snippets"] B --> C["Verify Unity Catalog Permissions"] C --> D["Locate Google Docs & Jira Issues"] D --> E["Execute Action: Post Jira Comments via MCP"] E --> F["Generate & Execute Live SQL Queries"] ```

Georis instructs Genie One to draft an executive review document using the standardized Databricks template (encompassing metrics, highlights, and lowlights) provided via a Google Docs link. Simultaneously, she directs Genie to inspect open Jira backlog tickets and nudge assignees to ensure work stays on schedule. Opening the real-time reasoning trace, Georis highlights how Genie executes these tasks in parallel:

> "Genie isn't forcing you to create a separate permission system. It's enforcing the permissions that you already have through Unity Catalog."

The system trace displays Genie retrieving relevant business definitions from the Unity Catalog Glossary, extracting authoritative calculations from ontology snippets, fetching the remote template document, identifying the relevant Jira issues, commenting directly on the tickets using the Model Context Protocol (MCP), and initiating live SQL queries against warehouse tables to supply the exact figures needed for the executive review.

[01:13:01](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4381s) Live Computation, Ontology Snippets, and OntoRank in Genie

Continuing the live demonstration, the presenter emphasizes a critical distinction between generic AI tools and Genie: instead of scraping outdated text, Genie accesses live operational data federated across both Databricks and BigQuery to establish the ground truth. As the presenter points out: “A lot of generic agents are just reciting what already exists in a document, but there’s no document for this week’s metrics.” Genie analyzes the operational data, breaks down the numbers by region, generates clean visualizations, and flags key highlights, lowlights, and anomalies. Crucially, the process finishes almost instantaneously: “Rather than just grinding through an agentic loop of trial and error and eating up a lot of tokens along the way, Genie just went straight to the ground truth.”

```mermaid flowchart TD accTitle: Ground Truth to Verified Analysis Workflow accDescr: Shows how Genie computes fresh results by querying operational stores and validating against ontology snippets. subgraph DataSources["Live Data Federation"] DB["Databricks Tables"] BQ["BigQuery Data"] end subgraph OntoEngine["Genie Ontology"] Snippets["Ontology Snippets & Definitions"] OntoRank["OntoRank Authority Scoring"] end subgraph OutputGen["Generated Review"] ExecDoc["Interactive OKR Review Doc"] ForecastViz["Live Forecast Visualizations"] end

DB --> OntoEngine BQ --> OntoEngine Snippets --> OntoRank OntoEngine --> ExecDoc ExecDoc --> ForecastViz ```

To demonstrate trust and explainability, the presenter drills down into the citations drawer, revealing every table, foreign catalog, Jira ticket, and document Genie referenced. A verified glossary definition confirms how key terms are established. Inspecting the underlying "ontology snippets"—the discrete business facts Genie learns and evaluates automatically—reveals definitions for engagement, what constitutes a Jira blocker, and the exact SQL used for regional breakdowns. Looking into the engagement metric snippet shows it tracks app sessions exceeding 30 seconds, sourced directly from a mobile KPI dashboard authored by a colleague named Chung. The snippet carries a high authority score calculated by the OntoRank algorithm, reflecting Chung’s active contributions and domain footprint across the company. The presenter schedules the analysis to run automatically every week, then prompts Genie to enrich the executive doc with a forward-looking forecast using a teammate's shared custom skill.

- **Data Sources:** Live computation executed across federated Databricks tables and BigQuery operational storage. - **Ontology Snippets:** Reusable facts that define key concepts, such as app engagement criteria (sessions lasting longer than 30 seconds) and Jira blocker logic. - **OntoRank Algorithm:** Scores the authority and reliability of snippets based on usage frequency, originating assets, and author engagement within specific business domains. - **Extensible Skills:** Incorporates a peer’s prebuilt forecasting skill utilizing Databricks' built-in forecast function with tailored review rules.

---

[01:16:07](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4567s) Vibe Coding Custom Agents and Proactive Mobile Insights

Because executive review prep is recurring rather than a one-off task, the presenter turns the conversational session into an autonomous, domain-specific agent dedicated to the Genie mobile app. With a simple prompt, Genie packages the entire context, guidelines, and logic into an expert coworker that stays permanently up to date and executes queries on demand without getting stale. The agent can be shared across product managers, certified for the company-wide registry so that the ontology prioritizes it, and deployed across organizational functions such as HR, IT, and finance.

Transitioning from the desktop interface to mobile, the presenter shows an alert arriving in real time on a smartphone. Without any alerting rules or monitoring scripts configured beforehand, Genie proactively detects an engagement spike on the mobile app and notifies the presenter directly. Explaining why this represents a paradigm shift for business intelligence, the presenter notes: “Genie’s no longer just a reactive Q&A tool. It’s now a proactive AI co-worker that can bring me things that matter before I even know to ask.” Within minutes, the workflow moves a strategic initiative forward, prepares an executive-ready deck, answers incoming team questions, and surfaces unexpected anomalies.

- **Agent Creation:** Generates a persistent, specialized AI coworker that retains product-specific domain context without repetitive prompt engineering. - **Enterprise Certification:** Certified agents appear on the organization's verified agent registry and receive priority ranking within the Genie Ontology. - **Proactive Notifications:** Genie monitors background trends aligned with user activity and issues autonomous push alerts on critical metrics without manual rule-setting. - **Multi-Channel Distribution:** Finished documents and dashboards can be scheduled and dispatched directly through Slack and email.

---

[01:19:24](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4764s) Ken Wong Announces Genie One General Availability and MCP Support

Ken Wong returns to the stage to formalize the launch of Genie One as a fully featured, data-smart AI coworker capable of performing net-new live analysis, running scheduled jobs while laptops are closed, and integrating with external systems via custom Model Context Protocol (MCP) tools governed by Unity AI Gateway. Highlighting broad enterprise adoption, Wong details how Warner Music Group relies on Genie across hybrid Databricks and non-Databricks data estates, General Motors has deployed hundreds of domain-specific Genie agents into production across organizational teams, and Foot Locker uses Genie to empower front-line employees with data-driven decision making.

```mermaid flowchart LR accTitle: Genie One Deployment and Ecosystem Architecture accDescr: Illustrates user touchpoints feeding through the Genie One platform governed by Unity AI Gateway. subgraph Channels["User Surfaces"] Slack["Slack & Teams"] Mobile["iOS & Android"] MCPClient["External AI Agents (MCP)"] end subgraph Core["Genie One Platform"] AgentEngine["Genie 1 Agent Core"] ContextOnto["Genie Ontology"] end subgraph GovernanceLayer["Governance & Data"] UnityGW["Unity AI Gateway"] UnifiedData["Federated Enterprise Data"] end

Channels --> AgentEngine AgentEngine --> ContextOnto ContextOnto --> UnityGW UnityGW --> UnifiedData ```

Wong announces that Genie One is available immediately within everyday workplace interfaces, featuring native integrations for Slack and Microsoft Teams alongside dedicated mobile apps for iOS and Android. For enterprises that already maintain external agents or custom copilots, Databricks is also launching the Genie MCP app today, enabling any third-party agent to call into Genie's underlying calculation engine and ontology for grounded mathematical accuracy. To accelerate adoption across the enterprise, Wong reveals an aggressive rollout incentive: “So in order to kind of encourage that to happen, we’re giving $10 worth of tokens for every single user inside your organization every single month.” With these capabilities in place, Wong officially announces that Genie One is generally available today.

- **Genie One General Availability:** GA launch encompassing full coworker workflows, scheduling, shared skills, and live computational abilities. - **Platform Ingestion & Collaboration:** Instant accessibility via Microsoft Teams, Slack, Android, and iOS. - **Genie MCP App:** Standardized Model Context Protocol interface allowing third-party and custom agents to leverage Genie’s ontology and live computing engine. - **Customer Success Proof Points:** - *Warner Music Group:* Employs Genie across both Databricks-native and federated non-Databricks platforms. - *General Motors:* Running hundreds of specialized Genie agents in production across internal departments. - *Foot Locker:* Equipping everyday retail and corporate staff to make operational choices backed by live data. - **Adoption Credit:** $10 in complimentary monthly token credits per user for every enterprise customer.

[01:22:53](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=4973s) Transitioning to PepsiCo's AI Transformation

Ali Ghodsi closes the Genie demonstration by urging the audience to integrate Genie One everywhere across their organizations and make it available to every employee. He then introduces PepsiCo to share the story of their ongoing collaboration. Databricks Co-founder and SVP of Field Engineering Arsalan Tavakoli-Shiraji welcomes PepsiCo's Global Chief Data and AI Officer, Magesh Bagavathi, to the stage to discuss how one of the world's largest consumer packaged goods companies tackles enterprise-scale AI transformation.

[01:24:11](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5051s) Scaling Data Architecture for 1.4 Billion Daily Occasions

Magesh Bagavathi describes PepsiCo's massive global footprint: an operation running continuously across roughly 200 countries with 320,000 employees and one of the largest corporate vehicle fleets in North America. To frame their operational scale, Bagavathi highlights their primary consumer engagement metric—"occasions"—which accounts for approximately 1.4 billion product consumption moments every single day across 6 million retail endpoints, direct-to-consumer (D2C) channels, and electronic B2B (eB2B) avenues.

To power this reach under PepsiCo's core mission of being faster for consumers, stronger for business and people, and better for the planet, the company embarked on an end-to-end digital and process transformation six years prior. Bagavathi explains that their foundational philosophy is straightforward: “For us what we truly believe is an AI transformation is a data transformation, and that's what we've undertaken.” They moved away from a fragmented legacy footprint of more than 60 disparate data lakes into a single Databricks Lakehouse, replacing rigid monolithic infrastructure with a modular, pluggable, "Lego-based" setup. This enterprise data foundation now houses 90% of PepsiCo's entire global data universe as an active, living platform.

```mermaid flowchart TD accTitle: PepsiCo Architecture Modernization accDescr: Migration from 60 plus fragmented data lakes to a unified Databricks Lakehouse enterprise data foundation.

subgraph Legacy["Legacy Architecture"] L1["60+ Disparate Data Lakes"] L2["Monolithic Systems"] end

subgraph Modern["Enterprise Data Foundation"] M1["Unified Databricks Lakehouse"] M2["Pluggable Lego-based Architecture"] M3["90% of PepsiCo Global Data"] end

L1 -->|"Consolidation"| M1 L2 -->|"Modularization"| M2 M1 --- M3 ```

- **Global headcount:** 320,000 employees across ~200 countries. - **Fleet scale:** One of the largest commercial fleets in North America. - **Consumer reach:** 1.4 billion consumption occasions daily across 6 million retailers and digital channels. - **Data consolidation:** Reduced 60+ legacy data lakes down to a single Databricks Lakehouse. - **Data footprint:** 90% of PepsiCo's enterprise data estate is unified in the lakehouse.

[01:26:41](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5201s) Moving from Fragile RAG to Genie

Arsalan Tavakoli-Shiraji notes that Bagavathi was among the earliest enterprise leaders to urge Databricks to build a system like Genie. Bagavathi recalls that when PepsiCo began exploring natural language "dialogue with data"—aiming to empower everyone from warehouse and frontline staff to knowledge workers—they originally tried building custom AI-for-BI systems using standard Retrieval-Augmented Generation (RAG) and vectorization pipelines.

However, building custom RAG architectures was slow and inaccurate. Deployments took anywhere from three to six months, and the resulting query accuracy consistently fell short of enterprise requirements. Once PepsiCo achieved 95% data cataloging accuracy within their unified Databricks foundation, they partnered with Databricks to deploy Genie directly against that governed data, bypassing fragile custom RAG stacks.

[01:28:11](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5291s) Real-World Impact: The Supply Chain AI Brain and SpendWise

Bagavathi emphasizes that technology proves its worth only when deployed to production. PepsiCo's overarching strategy centers on building a "Supply Chain AI Brain," which includes a specialized procurement brain sponsored directly by their Chief Procurement Officer. Managing direct and indirect procurement spending is critical for an enterprise generating $95 billion in annual revenue.

Within PepsiCo's procurement platform, known as "SpendWise," leaders historically relied on standard dashboards and static reporting suites. Introducing Genie fundamentally shifted user habits: - In the initial weeks post-launch, SpendWise processed nearly 30,000 conversational query interactions. - User traffic migrated away from legacy dashboards toward conversational exploration and Genie's autonomous "agent mode" for deep research tasks. - Following this success, PepsiCo is expanding Genie into international commercial reporting, starting with Europe and Latin America.

Looking ahead toward an "agentic PepsiCo," Bagavathi outlines an ambitious consolidation target: “So, our goal is now to go from these 30,000 plus reports down to 50 plus AI for BI or Genie consoles.” This consolidation from tens of thousands of static reports down to roughly 50 specialized Genie environments is designed to agentify workflows across all 320,000 associates.

[01:30:47](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5447s) The Data Engineer's Dilemma in the Agentic Era

Bilal Aslam, Senior Director of Product Management at Databricks, takes the stage wearing his signature purple jacket, jokingly noting that after wearing it two years ago and skipping it the previous year, the audience's primary feedback was to bring the jacket back.

Aslam turns the conversation to the reality faced by data engineers amid the industry-wide excitement around AI agents. While executives celebrate conversational agents and real-time outputs at the business end of the pipeline, data engineers recognize that every single agent depends entirely on reliable data, thinking: “This is great but all of this needs data... and you might be thinking, 'My job got a little bit harder.'”

```mermaid flowchart LR accTitle: Data Engineering Pipeline Complexity accDescr: Pipeline showing data flow from diverse raw sources through complex intermediate tools to agentic business outcomes.

subgraph Sources["Raw Data Sources"] S1["Kafka Streams"] S2["SaaS (Salesforce, NetSuite)"] S3["On-Prem (Oracle, SQL Server)"] end

subgraph Middle["The Messy Middle"] M1["Data Engineering"] M2["Multiple Tools & Glue Code"] end

subgraph Destination["Business Outcomes"] D1["AI Agents"] D2["Applications"] D3["Real-Time Operations"] end

Sources --> M1 M1 --> M2 M2 --> Destination ```

The desired end state is clean business value—applications, operational agents, and real-time execution—but the reality starts on the far left with fragmented raw data sources: Kafka message streams, SaaS platforms like Salesforce and NetSuite, and on-premises relational engines like Oracle and Microsoft SQL Server. Data engineering remains the "messy middle" that must reconcile these systems. Over decades, organizations have accepted massive architectural sprawl filled with dozens of disconnected point-solution logos and brittle pipeline connections. Aslam observes that teams have reluctantly made peace with this complexity simply because "at least it works," even while lacking critical capabilities like unified version control and end-to-end governance.

[01:32:41](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5561s) The Data Engineer's Dilemma in the Age of Coding Agents

Bilal Aslam begins by breaking down the painful contradictions of today's fragmented data engineering stack. Teams strive for consistent version control, yet they find themselves able to version control their Apache Spark code while remaining unable to manage their Apache Kafka infrastructure the same way. Universal governance remains an elusive pipe dream because data is read from and written to far too many disparate systems, leaving large portions of the stack completely locked down. On top of that, organizations want limitless scale and cost efficiency to drive business outcomes, but the underlying components simply do not scale equally.

```mermaid flowchart TD accTitle: The Fragile Data Engineering Stack vs The Agent Deluge accDescr: Diagram illustrating how the existing fragile stack of Spark, Kafka, and proprietary SaaS is overwhelmed by agent-driven code generation, creating an operational bottleneck.

subgraph Fragile ["Fragile Data Stack"] A["Spark Code<br/>(Version controlled)"] B["Kafka Infra<br/>(Unversioned / fragile)"] C["Proprietary SaaS / Formats<br/>(Locked down)"] end

subgraph Agents ["Agent Influx"] D["Genie Code<br/>(Coding agent)"] end

subgraph Outcome ["The Dilemma"] E["60% Pipelines Generated in 3 Mos"] F["Massive Pipeline Sprawl & Outages"] end

D --> E E --> F A -.-> F B -.-> F C -.-> F ```

Aslam delivers a sober warning to the room: “I'm here to tell you that your job, and my job as a data engineer, is about to get quite a bit harder.” The catalyst is Genie Code, Databricks’ purpose-built coding agent engineered specifically for data science, machine learning, and data engineering. While the technology is an undeniable technical triumph, Aslam expresses a candid tension. He loves agents, but the sheer volume of code they output threatens to overwhelm existing teams. In his view: “We're going to get swamped.” The only sustainable solution is to strip away the moving pieces, replacing proprietary complexity with open formats, open frameworks, and open tools. Aslam makes the bold claim that Lakeflow represents the unified, open foundation needed to anchor this AI-driven future while maintaining full interoperability with hundreds of partner ecosystem tools.

- Within just three months of release, 60% of Lakeflow pipelines are already being authored directly by Genie Code. - Data engineers face an explosion of net-new pipelines and data assets being created on top of fragile, unversioned legacy infrastructure.

---

[01:34:23](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5663s) Unifying Transformation and Ingestion with Open Declarative Pipelines

Moving through an architectural speed run, Aslam addresses data transformation (ETL)—the single area where enterprises sink the majority of their engineering hours and budgets. Teams are currently bogged down supporting legacy Spark installations or running proprietary Spark distributions on services like Amazon EMR. He proposes replacing that fragmentation with Apache Spark declarative pipelines on Lakeflow. These open pipelines can run anywhere—even locally on an engineer's laptop—and because they are declarative, AI coding agents excel at authoring them. They unify batch processing with streaming, support both SQL and Python interchangeably (ending the historical dichotomy between tools like dbt and PySpark), and incorporate a brand-new real-time streaming engine open-sourced the prior year to deliver millisecond-latency streaming without requiring Apache Flink.

```mermaid flowchart LR accTitle: Lakeflow Declarative Foundation accDescr: Flowchart showing how Lakeflow Designer, Connect, and Zero Bus Ingest all compile down to Apache Spark Declarative Pipelines.

subgraph Inputs ["Authoring & Ingestion"] UI["Lakeflow Designer<br/>(No-code / Genie)"] CONN["Lakeflow Connect<br/>(100+ connectors)"] BUS["Zero Bus Ingest<br/>(Kafka wire-compatible)"] end

subgraph Core ["Unified Core"] SDP["Spark Declarative Pipelines<br/>(Open / Batch + Streaming / Python + SQL)"] end

subgraph Engine ["Execution"] RT["Real-time Streaming Engine<br/>(Sub-second / Millisecond)"] LH["Lakehouse Open Storage"] end

UI --> SDP CONN --> SDP BUS --> SDP SDP --> RT RT --> LH ```

To curb the proliferation of "shadow IT"—where hundreds of business analysts construct unversioned, drag-and-drop pipelines on their laptops using proprietary tools that break as soon as an employee departs—Databricks announced the General Availability of Lakeflow Designer. Powered by Genie, this no-code interface allows users to prepare data visually while under the hood compiling standard, open Spark declarative pipelines.

Aslam extends this pattern to ingestion. Instead of relying on closed ingestion SaaS products that store data in proprietary silos inaccessible to autonomous agents, Lakeflow Connect now features more than 100 connectors backed by an open-source developer community. For streaming telemetry, Aslam introduces Zero Bus Ingest, a fully managed service that is 100% wire-compatible with Apache Kafka. Engineers can seamlessly retarget their existing Kafka producer applications to Zero Bus Ingest, landing telemetry straight into the lakehouse in open formats at massive scales of up to 12 GB per second without the operational headaches of managing brokers or debugging millions of tiny storage files.

- Lakeflow Designer is now Generally Available (GA), translating no-code data preparation into standard Spark declarative pipelines. - Lakeflow Connect supports over 100 data source connectors, alongside an open-source framework for building custom integrations. - Zero Bus Ingest provides 100% Kafka wire compatibility, eliminating Kafka cluster management while ingesting 12 GB/s without generating small-file bottlenecks.

---

[01:37:43](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5863s) Serverless Orchestration and the Reality of Data Operations

Next, Aslam targets pipeline orchestration, which historically relied on Apache Airflow. While developers appreciate authoring DAGs in native Python, Databricks survey data reveals that 80% of customer Airflow deployments run severely outdated distributions plagued by security vulnerabilities, high maintenance overhead, and missing features. Lakeflow Jobs replaces this burden with a fully serverless orchestration engine that supports pure Python workflows alongside visual DAG building. To eliminate the historical limitation where Jobs only effectively orchestrated Databricks tasks, Aslam reveals one of his favorite Summit announcements: over 50 open-source integrations (functioning like modern Airflow operators) that allow Lakeflow Jobs to orchestrate external ecosystems, including Snowflake.

```mermaid flowchart TD accTitle: The Reality of Modern Data Operations accDescr: Graph contrasting pipeline scale milestones with the stark operational realities and maintenance time sinks experienced by data teams.

subgraph Scale ["Lakeflow Production Scale"] S1["200 Trillion Rows / Day processed"] S2["1.7 Billion Job Runs / Month"] S3["50% Customers on Serverless Compute"] end

subgraph Ops ["The Operational Bottleneck"] O1[">50% Engineering Time Spent on Maintenance"] O2["Up to 60 Hours Monthly Downtime"] O3["Data Engineering!= Software Engineering<br/>(Code alone is not self-describing)"] end

Scale -.->|"Accelerates Pipeline Growth"| Ops ```

Proving that this open architecture is hardened for enterprise production, Aslam points to immense scale metrics. Yet he concedes that building pipelines is only half the battle. By making it vastly simpler for anyone—and any agent—to build pipelines, organizations unintentionally invite operational gridlock: “Great, now more people can build more pipelines, more outages, more problems.”

Partner survey data demonstrates that data engineering organizations already devote more than 50% of their bandwidth purely to firefighting and system maintenance, yet still endure up to 60 hours of pipeline downtime every single month. When addressing why simple API integrations and standard coding agents cannot magically resolve operational failures, Aslam draws a clear distinction between disciplines: “And the big idea here is that data engineering is not software engineering. It's something a bit more.” While software engineering relies on code that is completely self-describing, data pipelines are inherently tethered to unpredictable data states, schemas, and runtime conditions that code alone cannot predict.

- Spark declarative pipelines process over 200 trillion rows of data every single day. - Lakeflow Jobs executes approximately 1.7 billion job runs per month. - 50% of Databricks customers have fully transitioned to serverless compute for their data workloads, eliminating cluster and VPC management. - More than 50 open-source orchestrator integrations released to coordinate across third-party tools, including Snowflake. - Data teams routinely lose over 50% of their working hours to pipeline maintenance while averaging up to 60 hours of monthly downtime.

[01:40:51](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6051s) Why Traditional Coding Agents Fail in Data Engineering

Software engineering and data engineering operate on fundamentally distinct paradigms. In software engineering, tests are deterministic: code either passes or fails. In data engineering, code operates on dynamic, evolving data, meaning tests must account for statistical distributions rather than strict binary outcomes. Crucially, software engineering tolerates visible runtime failures:

> “When something goes wrong, you're going to see a trace. You're going to see an exception and then you can roll back your deployment. In data engineering, failures are silent and permanent.”

A silent failure often goes unnoticed until a CEO or executive stakeholder makes an angry phone call a week later asking why critical reporting metrics are broken.

Generic coding agents fall short in data engineering environments across every stage of the incident lifecycle: - **Detection:** Agents lack real-time telemetry and metrics. Providing raw logs (such as massive multi-megabyte Apache Spark execution traces) rapidly overwhelms LLM context windows. - **Root-Cause Analysis:** External agents lack visibility into lineage and require organizations to export complex graph structures outside the environment. - **Remediation & Verification:** Unlike software development, verifying a data pipeline fix requires running modified logic directly against production data.

Allowing a third-party, untrusted coding agent read access to production data already triggers severe enterprise compliance and security concerns. Granting that same agent write access to verify fixes creates untenable risk:

> “Who's here wants to give a random agent write access to their production data? There's got to be one person. Okay, there's no one who wants to do that, right? Because it can cause chaos. It can drop everything.”

Because agents require deep context, data lineage, and strict governance boundaries, Databricks concluded that an autonomous operations agent cannot reside externally. To safely resolve incidents, it must live natively inside the data plane.

```mermaid flowchart TD accTitle: Why External Coding Agents Fail in Data Operations accDescr: Diagram illustrating the failure points of external agents attempting to debug data engineering pipelines.

subgraph Problem["Data Plane Obstacles for External Agents"] A["Silent Data Quality Failures"] --> B["Multi-Megabyte Spark Traces"] B --> C["Missing Lineage Context"] C --> D["Risk: Unsafe Production Access"] end

subgraph Solution["Databricks Architectural Insight"] D --> E["Autonomous Agent Must Live Inside Data Plane"] E --> F["Native Lineage + Governed Shallow Clones"] end ```

---

[01:43:19](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6199s) Announcing Genie ZeroOps: Autopilot for Data and AI Operations

Databricks announces **Genie ZeroOps**, a background agent built directly into the data plane to automate the detection, assessment, diagnosis, remediation, and verification of data and ML pipeline failures.

Genie ZeroOps executes across four core phases: 1. **Autonomous Detection:** It builds per-table machine learning models that are continuously fine-tuned against native metrics, runtime events, and execution logs. Databricks already runs tens of thousands of these ML models in production to detect anomalies without manual threshold configuration. 2. **Lineage Assessment & Root Cause Analysis:** Using graph ranking over Unity Catalog data lineage, the system traverses upstream and downstream dependencies. To isolate the exact failure, a supervisor agent coordinates a fleet of specialized sub-agents that investigate hypotheses and establish consensus on the root cause. 3. **Cooperative Remediation:** Genie ZeroOps integrates with Genie Code. It reads version control repositories, tracks and updates ticketing systems, and drafts the necessary code fixes. 4. **Governed Verification via Shallow Clones:** Drawing on a decade of Lakehouse engineering, Genie ZeroOps verifies fixes by creating instantaneous, isolated shallow clones of production data (leveraging branching mechanisms similar to Lakebase). It executes the candidate code against production-scale data inside an isolated sandbox, ensuring pipeline logic produces correct results without endangering live tables.

---

[01:44:48](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6288s) Demo: Autonomous Incident Resolution and PII Governance

A live demonstration reveals that the Genie ZeroOps interface is modeled after a prioritized email inbox rather than a chaotic monitoring dashboard littered with red lights and unaggregated alerts.

Incidents are ranked strictly by business severity. ZeroOps automatically deduplicates and groups related issues; in one instance, it aggregates 16 distinct downstream job failures into a single root incident.

```mermaid flowchart LR accTitle: Genie ZeroOps Root-Cause and Verification Flow accDescr: Step-by-step resolution lifecycle executed autonomously by Genie ZeroOps during an incident.

subgraph Detection["1. Detection & Ranking"] I["Inbox Incident View"] --> S["Group 16 Failing Jobs"] end

subgraph Triage["2. Lineage Traversal"] S --> F["Forward: High-Impact Dashboard"] F --> B["Backward: Isolate fan_interactions"] end

subgraph Resolution["3. Isolated Verification"] B --> C["Generate Shallow Clone"] C --> V["Verify Row Counts"] V --> PR["Open Pull Request"] end ```

The walkthrough highlights a live data anomaly on a table named `top_fan_voters`: - **Anomaly Detection:** ZeroOps flags an unexpected drop in row count that occurred roughly 10 minutes prior. By the time an engineer opens the alert, background agents have already concluded the full investigation. - **Forward Impact Analysis:** ZeroOps traces the Unity Catalog lineage graph forward, determining that `top_fan_voters` feeds a critical executive fan engagement dashboard, marking the incident severity as critical. - **Backward Root Cause Discovery:** A fleet of sub-agents fans backward through upstream Directed Acyclic Graphs (DAGs), bypassing healthy intermediate steps to pinpoint an upstream source table: `fan_interactions`. - **Safe Branch Verification:** ZeroOps writes an automated code remediation. Instead of blindly modifying production state, it creates a shallow clone branch containing terabytes of data, executes the repair in network and code isolation, and confirms that the expected row counts are restored. - **Developer Review:** Rather than auto-applying changes, the agent presents a ready-to-merge pull request aligned with standard SDLC controls.

Beyond pipeline breakages, ZeroOps provides automated governance auditing. When an upstream application inadvertently exposes personally identifiable information (PII) belonging to approximately 1,000 users, ZeroOps scans the schema, quantifies table exposure, traces downstream blast radius through lineage, and generates an automated Unity Catalog governance policy update to mask the sensitive fields instantly.

---

[01:48:26](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6506s) Ali Ghodsi Introduces the "Known Data Realm"

Ali Ghodsi returns to the stage to contextualize the major architecture announcements scheduled throughout the summit, framing the fragmented enterprise data landscape as the "known data realm"—humorously comparing the historical sprawl of disparate systems to *Game of Thrones*:

> “So, but before I do that, I want to put it all in perspective, because it can get kind of confusing in the data world. There's so many different technologies, there's so many moving pieces. So, we wanted to simplify that. So, we call this what you see here on this slide the known data realm. Okay? It's kind of like the Game of Thrones, okay, of data infrastructure.”

Ghodsi charts the map of this infrastructure kingdom: - **The West (OLTP Databases):** Originating in the 1980s, the realm of Online Transaction Processing (OLTP) emerged out of the necessity to power transactional applications, placing an operational database behind every enterprise business application. - Over the subsequent decades, the divergence between operational transactional systems and downstream analytics engines generated fragmented silos, costly extract-transform-load (ETL) pipelines, and governance friction that modern lakehouse architectures aim to resolve.

[01:49:29](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6569s) The Fragmented Data Map and the Onslaught of Agents

Ali Ghodsi walks through the traditional database architecture landscape by framing it as a map of isolated territories. On the left side lies the island of OLTP (Online Transaction Processing). In critical software systems, such as an ATM where a user withdraws money, transactions must execute against the database with sub-second, millisecond latency, zero downtime, and bulletproof transactional integrity. Over decades, this side spawned specialized, niche engines, including key-value stores and, more recently, vector search systems for AI.

```mermaid flowchart LR accTitle: The Traditional Fragmented Data Architecture accDescr: Data flows from left-side OLTP transaction systems across brittle pipelines to analytical warehouses and real-time serving engines.

subgraph OLTP["OLTP Island"] ATM["Transactional DBs<br/>(e.g., ATMs, Core Apps)"] Niche["Niche Stores<br/>(Key-Value, Vector Search)"] end

subgraph Pipelines["Data Movement"] ETL["Data Engineering Pipelines<br/>(ETL, Shuffling, Copies)"] end

subgraph Analytics["Analytics & Real-Time"] DWH["Data Warehouses<br/>(Historical Aggregates)"] DS["Data Science & ML<br/>(Predictive Stats)"] RT["Real-Time Analytics<br/>(Dashboards, Low-Latency SLAs)"] end

OLTP -->|"continuous data shuffling"| ETL ETL --> DWH ETL --> DS ETL --> RT ```

On the right-hand side sits the world of data warehousing, which emerged after the 1980s to answer broader aggregate questions that would otherwise overwhelm operational databases. Ghodsi illustrates the divide: querying an OLTP database to calculate aggregate transaction volumes across thousands of ATMs risks knocking the operational database offline, causing ATM outages. Warehousing solved this by isolating analytical workloads. Over the past 15 years, organizations moved beyond basic aggregates to advanced predictive analytics and data science—asking questions like predicting future withdrawals rather than merely tallying past ones.

Recently, a separate need emerged for real-time analytics powering dashboards and responsive applications. However, bridging the OLTP island and analytical stacks required extensive data engineering pipelines to extract, shuffle, and copy data. With autonomous AI agents entering the enterprise, this architecture faces collapse.

> “It’s almost like everybody just hired double the number of employees. Okay, cuz the agents are submitting queries, they want more data stuff, more data movement is happening. So, we need to simplify this.” — Ali Ghodsi

- Modern transactional systems require millisecond latencies, continuous availability, and absolute transactional correctness. - The surge of autonomous agents running continuous queries risks overwhelming existing, brittle pipeline infrastructure that has already seen major industry outages over the prior year. - Databricks previously unified data engineering, data science (via Apache Spark and Spark ML), and data warehousing under the Lakehouse architecture introduced five years earlier, but low-latency real-time analytics still remained isolated.

---

[01:53:32](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6812s) Reynold Xin on Lakehouse Limits and the "One Size Fits All" Trap

Ali Ghodsi welcomes Databricks co-founder and Chief Architect Reynold Xin to the stage, teasing what he describes as their single largest architectural innovation since the original launch of the Lakehouse. Xin notes that while the Lakehouse successfully unified data science, data engineering, and data warehousing—earning leadership recognition from Gartner and Forrester, adoption by over 60% of the Fortune 500, and scanning exabytes of data every morning—existing lakehouses and data warehouses encounter a severe physical boundary when pushed toward sub-second response times.

```mermaid flowchart TD accTitle: The Latency Wall and Serving Stack Dilemma accDescr: Warehouses hit a wall at one second, forcing teams into duplicate serving stacks that create governance and pipeline overhead.

W["Lakehouse / Data Warehouse Engine"] -->|"Hits latency wall"| L["~1-Second SLA Boundary"]

L -->|"Cannot guarantee millisecond SLAs"| Split["Split Architecture"]

subgraph ServingProblems["Serving Stack Penalties"] P1["Custom ETL pipelines per dataset"] P2["Fast point queries, terrible general analytics"] P3["Security & governance synchronization drift"] end

Split --> ServingProblems ```

Xin clarifies that hitting a wall at around one second does not mean an analytical engine can never complete an occasional query in under a second; rather, it cannot reliably honor stringent millisecond-level service level agreements (SLAs) without suffering severe tail-latency spikes. To work around this, organizations build out separate point-solution serving stacks, copying slices of data out of the central warehouse. This architecture forces engineers to maintain redundant pipelines, introduces brittle systems that fail when queries diverge from simple point lookups, and creates critical governance vulnerabilities where access controls applied to the warehouse do not carry over to the downstream serving layer.

> “And as a result, I think for many organizations that should require workloads that have very stringent SLAs, they separating setting up a separate serving stack. They would copy their data instead of the data warehouse or the lakehouse and copy a portion of it that’s required for serving into a separate stack. And this caused a lot of problems.” — Reynold Xin

- Lakehouse adoption spans over 60% of the Fortune 500, routinely processing exabyte-scale workloads daily. - Standard warehouse engines struggle to maintain strict SLAs below 1,000 milliseconds due to execution scheduling and query planning overhead. - Dual-stack architectures create governance drift, where security policies implemented in the central warehouse are missing in the low-latency serving cache.

---

[01:55:45](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6945s) Flipping 50 Years of Database Engineering

Xin explains that unifying low-latency serving with deep analytical processing has historically been considered an intractable engineering challenge. He references Turing Award winner and Postgres creator Michael Stonebraker's landmark paper, *"One Size Fits All, an idea whose time has come and gone."* In that paper, Stonebraker demonstrated that building a single engine to handle disparate workload profiles inevitably fails because internal architectural complexity compounds uncontrollably.

Xin details the conventional development loop practiced by database teams—Databricks included—over the past four to five decades:

```mermaid flowchart TD accTitle: The Traditional Database Engineering Failure Loop accDescr: The cycle of implementing academic algorithms that accelerate specific benchmarks while inadvertently causing massive regressions across general workloads.

A["Identify target workload"] --> B["Study academic papers (past 40-50 years)"] B --> C["Implement specialized data structures & algorithms"] C --> D["Deploy to real-world multi-tenant workloads"] D --> E{"Performance Outcome"} E -->|"Subset A"| F["Massive speedup on tiny queries"] E -->|"Subset B"| G["Severe regressions on large datasets"] G --> H["Architectural dead end / fragmentation"] ```

Under this model, engineers study academic literature, write custom algorithms or data structures tailored to specific query patterns, and deploy them. Inevitably, optimizations crafted to produce sub-millisecond latencies on narrow lookups backfire catastrophically when applied to larger analytical datasets, while optimizations suited for scanning massive tables penalize point lookups. Two years prior, Databricks resolved to break this design cycle by heading back to the drawing board and fundamentally flipping the standard methodology of database engineering on its head.

> “In this paper, Stonebraker actually argued it’s very, very difficult to engineer a single database engine that’s capable of running a wide variety of workloads. And ultimately it’s because the increasing complexity makes it very, very difficult to actually engineer the system.” — Reynold Xin

[01:57:54](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7074s) Engineering Reyden: ML-Driven Optimization from Trillions of Queries

Databricks leveraged its massive scale to solve database optimization challenges. Over zettabytes of data scanned and quadrillions of execution traces from trillions of executed queries were gathered to train a high-fidelity machine learning model capable of predicting the practical impact of new techniques. Rather than using large language models—which introduce far too much latency—the lightweight model serves two distinct functions: it dynamically selects the right algorithm to dispatch at runtime, and it guides engine developers beforehand on which algorithms and data structures are actually worth implementing out of millions of possibilities. As the speaker explains:

> “Because there are millions of algorithms and data structures out there. Knowing what to implement is one of the biggest advantage one can get.”

The resulting engine is named Reyden (a deliberate nod to *Mortal Kombat*). Unlike existing specialized serving stacks that cap query execution times at 5 seconds, mandate engine-specific proprietary data copies, or fail outright on complex joins, Reyden is built to handle the full spectrum of analytical workloads within a new SQL warehouse architecture.

```mermaid flowchart TD accTitle: Reyden ML Optimization Flow accDescr: How Databricks uses telemetry from trillions of queries to train an ML model for engine optimization and runtime execution. subgraph Data["Telemetry Scale"] A["Zettabytes of Scanned Data"] B["Quadrillions of Query Traces"] end subgraph Model["ML Prediction Engine"] C["High-Fidelity ML Model<br/>(Low Latency, Non-LLM)"] end subgraph Decisions["Engine Optimization"] D["Pre-Production: Predict & Select Algorithms to Implement"] E["Runtime: Dispatch Optimal Algorithm Instantly"] end subgraph Target["Reyden Engine"] F["Reyden Engine / Lakehouse//RT"] end A --> C B --> C C --> D C --> E D --> F E --> F ```

[01:59:44](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7184s) Live Demos: Sub-10ms Queries and Concurrency Stress Testing

On stage, a side-by-side demonstration illustrates the performance leap using the New York City taxi dataset. While standard Lakehouse completes the query in roughly one second—respectable for typical interactive work—the Reyden-powered setup, branded as Lakehouse//RT, executes the exact same query in just 7 milliseconds (0.007 seconds).

To demonstrate resilience under pressure rather than relying solely on single-query runs, a simulator modeled after real dashboard applications hammers the Reyden engine: - Starting with a single active session running an 8-query dashboard, loading occurs instantaneously. - The load is then scaled up to 1,000 active concurrent agents hitting the engine simultaneously. - With each agent issuing multiple queries, the engine processes 8,000 queries at a throughput rate of 6,000 queries per second (QPS). - Reyden sustains a tail latency of only 37 milliseconds, handling the massive traffic spike without queuing delays or failure.

[02:01:46](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7306s) Benchmark Showdown: TPC-H Q6 and Complex Joins

The speaker contrasts Reyden against major competing architectures across latency versus throughput, plotting P90 latency against QPS using canonical data warehouse benchmarks:

| System / Architecture | TPC-H Q6 Performance & Scalability | Complex Joins & Full TPC-H Behavior | |:--- |:--- |:--- | | **Competitor Gen2 Warehouse** | Starts with low latency but degrades sharply and hits a wall at ~160 QPS. | Fails under larger scale factors and heavy joins. | | **Competitor Interactive Warehouse** | Handles ~300 QPS before latency spikes; requires duplicating lake data into an isolated, engine-specific store. | Crashes and cannot complete, as it enforces a hard 5-second query timeout limit. | | **Specialized Open Source Engine (Yellow)** | Excels at simple queries up to ~15,000 QPS, but crashes abruptly beyond that threshold. | Spikes in runtime at 100 GB due to poor complex join optimization; runs out of memory (OOM) and fails at scale. | | **Reyden (Lakehouse//RT)** | Sustains 12,000 QPS while maintaining P90 tail latency well below 1 second without crashing. | Successfully completes all 22 TPC-H queries across every scale factor without timeouts or OOMs. |

Addressing benchmark fatigue in the audience, the speaker emphasizes:

> “Who cares about synthetic benchmarks? Um so one of the thing that we're really excited about in the process of developing Reyden is the design partners we've been working with.”

[02:05:07](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7507s) Announcing Lakehouse//RT: Production Results and Launch

Real-world production results from design partners validate the synthetic tests: - **Enverus** (data and AI platform for the energy sector) evaluated Reyden across 11 representative production queries. Typical low-latency queries remained below 100 milliseconds, while complex, long-running queries saw dramatic reductions—achieving a 16x average overall speedup and shrinking the runtime of the slowest queries by nearly 100x. - **Meta** tested Reyden on internal workloads, achieving typical response times in tens of milliseconds while eliminating the overhead of managing a separate serving tier.

The speaker officially announces **Lakehouse//RT** (where "RT" stands for real time), the first commercial product offering powered by the Reyden engine. Designed initially for high-concurrency, read-only analytical workloads, Lakehouse//RT delivers sub-second and millisecond response times directly against underlying Delta Lake and Apache Iceberg tables governed unified under Unity Catalog, rendering siloed downstream point-serving databases obsolete.

[02:07:04](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7624s) Lakehouse//RT Beta and the Origin of "Reyden"

The session transitions from real-time analytics as Reynold Xin wraps up the introduction of the Reyden engine, which brings millisecond latency and massive concurrency directly against data lakes using open table formats. The engine enters beta immediately, with early access coordinated through account teams and two dedicated breakout sessions scheduled later in the day. As Reynold moves to introduce Nikita Shamgunov, Ali Ghodsi steps back onto the stage to share the inside joke behind the project's name. Reynold had steadfastly refused to explain the acronym on stage, leaving Ali to playfully out him to the audience.

Ali reveals that engineers within Reynold's data platform group named the engine "Reyden" as shorthand for "Reynold's dream engine." Beyond teasing their chief architect, the team had a calculated strategy: Databricks frequently pares down initiatives to keep resources focused, and engineers reasoned that management would hesitate to cancel a project explicitly named after a co-founder. Ali then recaps the broader architectural vision: real-time analytics, once stranded on a separate operational island, is now merged into the unified lakehouse through Lakehouse//RT. By consolidating streaming ingestion, low-latency queries, and massive batch analytics under shared open storage formats and Unity Catalog governance—at an aggressive, equivalent price point—Databricks unifies analytical workloads. Ali then pivots to the next major frontier on their map: the OLTP island on the far left. To tackle operational databases, he invites Nikita Shamgunov to the stage, highlighting his pedigree as the former CEO of MemSQL (later SingleStore) and Neon, which was acquired by Databricks to spearhead transactional databases.

- Lakehouse//RT and the Reyden real-time engine enter beta availability on day one of the summit. - Real-time and analytical processing are unified under identical underlying data formats and Unity Catalog access controls. - As Ali Ghodsi quipped about the development team's naming strategy: “So, his team actually called the engine Reyden because it stands for Reynold's dream engine.”

[02:09:23](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7763s) Why AI Agents and Applications Need a New Kind of Database

Nikita Shamgunov frames his keynote around a fundamental shift in software engineering: software is increasingly generated by AI agents rather than written manually by humans. Pointing to the steep acceleration across the first six months of 2026, Nikita argues that humanity is poised to create more code over the coming year than across all prior history combined. Yet regardless of how code is produced, every functional application and agent still requires an underlying database to track state and execute transactions.

To serve this explosion of autonomous software, modern databases must fulfill three foundational requirements: 1. **Familiarity**: AI models and coding agents excel when operating systems that are deeply documented across the public web. Open-source, widely adopted databases like PostgreSQL benefit from millions of Stack Overflow threads, Reddit discussions, and GitHub repositories that LLMs have already ingested. Broad extensibility is equally critical so teams can consolidate specialized and general-purpose workloads onto one engine. 2. **Nimbleness**: With agents continuously generating ephemeral environments for development, testing, staging, and individual pull requests, databases must be truly serverless and branchable in milliseconds while scaling compute down to zero to prevent runaway infrastructure costs. 3. **Mission-Critical Reliability**: Operational data stores power core business activities. They cannot afford downtime, bottlenecks, or data corruption, demanding infinite storage scalability, low latencies, and enterprise-grade fault tolerance.

Nikita explains that PostgreSQL was the natural foundation because it is the world's most advanced and extensible open-source database, but its legacy architecture presents a glaring barrier. PostgreSQL was conceived as a monolithic system with tightly coupled compute and local storage, making lightweight serverless branching and instant provisioning virtually impossible without a fundamental architectural overhaul.

- AI development trends in early 2026 signal an unprecedented surge in automated code generation exceeding all historical human output. - As Nikita Shamgunov asserted regarding the changing nature of development: “And now we can confidently say we're going to have more software generated in the next year than in the history of humanity.”

[02:12:01](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7921s) Lakebase Architecture: Decoupling Postgres for the Lake

To eliminate the monolithic constraints of traditional PostgreSQL, the engineering team re-architected the database from the ground up by separating compute from storage and moving the persistence layer directly into the data lake. While object storage in the lake is cheap, durable, and virtually bottomless, it is inherently slow and lacks the strict transactional consistency required for operational record-keeping. To bridge this divide, Databricks built two specialized distributed services between PostgreSQL compute and the lake storage layer.

For transactional consistency and low-latency writes, they created SafeKeepers, a dedicated service that coordinates updates via the Paxos consensus protocol. For read operations, they engineered Page Servers, which cache and serve individual data pages directly to stateless PostgreSQL compute instances with minimal latency. Combining these components yields **Lakebase**: a fully managed, serverless PostgreSQL service operating natively on the lakehouse.

```mermaid flowchart TD accTitle: Lakebase Decoupled Architecture accDescr: Stateless Postgres compute coordinates writes through SafeKeepers and reads through Page Servers on top of lake storage.

subgraph ComputeLayer["Compute Layer"] PG["Stateless Postgres Compute"] end

subgraph CoordinationLayer["Low-Latency Tier"] SK["SafeKeepers<br/>(Paxos Consensus / Fast Writes)"] PS["Page Servers<br/>(Cached Pages / Fast Reads)"] end

subgraph StorageLayer["Persistence Tier"] Lake["Lake Storage<br/>(Infinite, Cost-Effective Storage)"] end

PG -->|"Writes & Commits"| SK SK -->|"Asynchronous Persistence"| Lake PS -->|"Low-Latency Page Reads"| PG Lake -->|"Hydrate Cache"| PS ```

Lakebase provides dramatic agility improvements for agent-driven engineering workflows: - **Sub-500ms Provisioning and Branching**: New PostgreSQL instances spin up in less than half a second. Developers and autonomous agents can instantly create isolated database branches for individual pull requests or experiments. - **Scale-to-Zero Efficiency**: Ephemeral test environments automatically suspend when idle, ensuring organizations only pay for active processing. - **Instant Snapshot and Restore**: With a single API call or button click, engineers can snapshot a database before deploying an agent to perform data or schema migrations; if the result is incorrect, the entire database state rolls back instantly.

- As Nikita Shamgunov explained regarding the trade-offs of object stores: “The lake storage is, you know, has a lot of very, very good properties. It's inexpensive. It's very easy to scale, but it's also slow and transactionally inconsistent.”

[02:14:27](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8067s) Mission-Critical Scale, Performance Benchmarks, and Reliability

Addressing potential skepticism over whether a decoupled, lake-backed PostgreSQL engine can sustain core enterprise workloads, Nikita details Lakebase's scaling and performance characteristics. Administrators define compute boundaries by setting simple minimum and maximum resource guardrails for cost governance. Within those limits, Lakebase dynamically scales compute up during daytime workload spikes and throttles back during nights and weekends. Because storage resides directly in the lake, disk capacity is effectively infinite, eliminating capacity planning, volume re-sizing, and disk-exhaustion outages.

To evaluate real-world speed under high concurrency, Nikita presents benchmark results using the industry-standard TPROC workload (measuring latency against scaling throughput): - A prominent legacy cloud database provider capped out around 130,000 operations per minute. - A second major cloud vendor utilizing a modernized architecture managed approximately 350,000 operations per second before query latency degraded significantly. - Lakebase maintained sub-10-millisecond operational latencies past 600,000 operations per second.

Beyond raw throughput, enterprise readiness requires rigorous compliance, end-to-end encryption, and robust enterprise security controls. Nikita begins outlining their approach to business continuity, emphasizing that while catastrophic cloud outages are rare, their impact on automated, agent-governed businesses can be existential if systems lack rapid multi-region or cross-cloud resilience.

- Lakebase eliminates disk management routines by backing relational tables directly with elastic lake storage. - Operational transactions sustain sub-10ms response times at throughput exceeding 600,000 operations per second. - As Nikita Shamgunov highlighted when presenting the TPROC benchmark: “And LakeBase can scale under 10 milliseconds for each transaction, for each operation, all the way north of 600,000 operations per second.”

[02:17:29](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8249s) Announcing Fully Managed Cross-Cloud Disaster Recovery for Lakebase

Nikita Shamgunov introduces fully managed cross-cloud disaster recovery for Lakebase, defining what it takes to build a truly mission-critical database layer on the lakehouse. Under this capability, organizations can configure their operational database to span multiple cloud providers simultaneously. Shamgunov illustrates this with a concrete deployment model: an active Lakebase instance provisioned in AWS US West paired with a live replica in Azure US East. If AWS suffers an outage, the system performs an instant failover to Azure, ensuring uninterrupted business operations without complex manual intervention.

Highlighting the rapid enterprise adoption of Lakebase—a fully managed, serverless Postgres that runs directly on the lake—Shamgunov notes that although the product has been on the market for only one year, more than 3,500 enterprise customers already trust it with mission-critical production workloads. To showcase how a global financial giant runs on this infrastructure, Shamgunov invites Fed Cohen Freue, Executive Vice President of AI and Data Operations at Mastercard, to the stage.

- Fully managed cross-cloud disaster recovery enables instant, automated failovers between major cloud providers (such as AWS US West to Azure US East). - Lakebase functions as fully managed serverless Postgres built natively on the lakehouse architecture. - Over 3,500 enterprise customers adopted Lakebase within its first year on the market.

```mermaid flowchart LR accTitle: Lakebase Cross-Cloud Disaster Recovery accDescr: Diagram showing primary Lakebase on AWS failing over instantly to a replica on Azure.

subgraph AWS["AWS (US West)"] direction TB P["Primary Lakebase Instance"] end

subgraph Azure["Azure (US East)"] direction TB R["Replica Lakebase Instance"] end

P -.->|"Continuous Replication"| R P ==>|"Outage: Instant Failover"| R ```

[02:18:47](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8327s) Mastercard's Agentic Platform and the Virtual C-Suite

Fed Cohen Freue outlines the immense scale at which Mastercard operates: serving over 200 countries, supporting billions of cardholders, and processing more than 150 billion transactions annually. As Mastercard pursues a strategic corporate goal where data and services account for over half of its total revenue, the company consolidated 80 disparate services into a unified agentic platform. Cohen Freue explains that Lakebase provided the shared data foundation, governance, and isolation necessary for multiple specialized AI agents to reason in real time and collaborate across a consistent business context.

Using Unity Catalog and Lakebase, Mastercard developed a reusable architecture for its new "Virtual C-suite" initiative, unveiled earlier in the year to support small business owners who struggle with executive-level decision-making. The first deployment, the Virtual CFO, directly assists small businesses with critical working capital choices, cash flow forecasting, and payment strategies. Cohen Freue emphasizes the power of having a single transactional-analytical foundation: “So, as one agent takes action or generates insights, it becomes instantly available for the next one.”

- Mastercard operates across 200+ countries, serving billions of cardholders and processing over 150 billion transactions per year. - Consolidating 80 services into one agentic platform supports Mastercard's target of deriving more than half of its revenue from data and services. - The Virtual C-suite's debut agent, the Virtual CFO, automates guidance on cash flow, payments, and working capital.

[02:21:15](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8475s) Multi-Tenant Isolation and 7-Week MVP Delivery at Scale

Turning to the challenges of multi-tenancy, Cohen Freue discusses an upcoming Mastercard capability called Performance Pulse, which requires serving thousands of issuing banks from a shared platform. In financial services, client data isolation is not merely a design preference; it is foundational to client trust, regulatory compliance, and strict data residency mandates. Lakebase allows Mastercard to keep issuer data strictly separated and protected while simultaneously aggregating anonymized, cross-ecosystem insights into a continuous feedback loop. This architecture allows Mastercard to build and validate a capability for a single issuer and horizontally scale it across thousands of financial institutions.

Shamgunov expresses astonishment at Mastercard's velocity, noting that their team moved from concept to a scale-ready MVP demo in just seven weeks—a timeline almost unheard of under strict banking regulations. Cohen Freue explains that they achieved this speed not by cutting corners, but by baking non-negotiable governance, guardrails, and data isolation directly into the foundational Lakebase architecture from day one. Looking ahead toward increasingly autonomous agentic commerce, Cohen Freue emphasizes why deep governance remains paramount: “Bringing that level of trust for us is the currency of innovation.”

- Performance Pulse delivers continuous insights and measurement across thousands of issuing banks on a single platform without breaching data isolation. - Mastercard progressed from initial concept to a production-ready MVP demo in seven weeks. - Embedded guardrails and governance in Lakebase are treated as reusable components across all subsequent agent builds.

[02:25:14](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8714s) Greg Brockman on Model Infrastructure and Connecting AI to the World

Databricks Co-Founder and VP of Engineering Patrick Wendell takes the stage and introduces OpenAI President and Co-Founder Greg Brockman. Wendell introduces Brockman as the technical engine and "Godfather" of the model production process, responsible for the infrastructure, computing frameworks, and talent that have powered OpenAI's foundational breakthroughs. Wendell asks Brockman about his expanding responsibilities as a founder and where his personal attention is concentrated today.

Reflecting on his journey, Brockman describes his operational philosophy as gravitating toward whatever constitutes the single most urgent bottleneck on any given day. He recounts the intense engineering grind during the development of GPT-4: “There was a time like for GPT-4 where, you know, I was up at 2:00 a.m. every night like, you know, getting the the model back up and running, trying to figure out like, you know, deep binary search through our nodes to go find uh which one was was, you know, causing a problem, things like that.” While OpenAI has successfully systematized its model production engine over the past several years to sustain rapid innovation across every layer of the training stack, Brockman notes that the primary frontier has shifted: the most critical challenge now is connecting these advanced frontier models directly to real-world workflows and enterprise applications.

[02:27:21](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8841s) The Rapid Cadence of Frontier Models

Patrick Wendell opens by highlighting the intense, focused effort required to bring cutting-edge research into production products, establishing unified methods for building agents that deliver real enterprise value. Wendell notes the astonishing velocity OpenAI has maintained over recent months, citing rapid successive model releases—GPT-5.3 in February, GPT-5.4 in March, and GPT-5.5 in April—amounting to roughly one major release per month. Having used these models extensively inside Codex, Wendell observes that each iteration represents a dramatic leap forward in quality and asks Brockman what drives this relentless velocity.

```mermaid flowchart TD accTitle: OpenAI Model Production Flywheel accDescr: Diagram illustrating the multidisciplinary layers feeding into continuous rapid model releases.

subgraph CoreTech["Technical Engine"] PT["Pre-training"] PoT["Post-training"] Inf["Inference Optimization & Debugging"] PT --> PoT --> Inf end

subgraph GoToMarket["Deployment & Governance"] Safe["Safety Evaluations"] Pack["Pricing & Packaging"] Safe --> Pack end

subgraph Flywheel["Organizational Execution"] Cross["Cross-team Coordination & Tooling"] Muscle["Iterative Release Muscle"] Cross --> Muscle end

CoreTech --> GoToMarket GoToMarket --> Flywheel Flywheel -->|"Continuous iteration on ~6-week cadence"| CoreTech ```

Greg Brockman responds with a classic waterfowl metaphor: “Well, it's very much the the you know, the duck analogy where it's like, you know, it all looks very smooth and seamless and you know, here is these models on a 6-week cadence, that kind of thing. Under the hood, it is a lot of work.” Drawing on his background building Stripe prior to OpenAI, Brockman remarks that while deep tech engineering might look entirely distinct from standard technology companies, the underlying organizational mechanics are fundamentally the same. Creating models requires disassembling a vast problem space into coordinated operational layers:

- **Pre-training and post-training synchronization:** Bridging foundational research with post-training alignment and optimization. - **Low-level systems engineering:** Identifying and squashing elusive race conditions and tuning inference to run reliably under heavy load. - **End-to-end productization:** Weaving safety protocols, deployment pipelines, packaging, and pricing directly into every release cycle. - **Internal flywheel dynamics:** Creating an organizational muscle across research and infrastructure teams so that training massive frontier models, which was an arduous struggle two years prior, operates as a repeatable, coordinated engine.

---

[02:29:44](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=8984s) The Exponential Curve and Moving from Chat to Useful Work

Wendell shares that Databricks saw major productivity gains across its own engineering organization upon deploying GPT-5.5, validating that the models are operating directly at the frontier. Brockman emphasizes that the technological exponential shows no signs of slowing down: “And the the exponential continues, right? And that is the wild thing about this field, right? It's like every single time a model drops, you try it, you're like, 'Wow, like mind is blown. Here's this problem I couldn't do before, and now it's easy.'” He argues this trajectory will soon encompass all knowledge work conducted on computers, acting as a rocket ship for productivity that every enterprise must actively incorporate into their strategy.

When Wendell jokingly presses for non-public details on future model dates and capabilities, Brockman frames the current paradigm shift by contrasting it with the GPT-4 era. GPT-4 was the breakthrough chat experience—the first time interactive, bespoke machine text was genuinely worth reading. Today's frontier models, by contrast, focus on executing concrete, useful work by wielding tools within live workflows:

- **The engineering transition:** In software engineering, task delegation has flipped dramatically; where models once handled 20% of routine "grind work," they now complete roughly 80%, completely restructuring developer tooling. - **Broader knowledge work impact:** Similar step-function gains are rolling out across everyday business workflows, from drafting presentations and slide decks to orchestrating cross-system data integration. - **Mission-driven distribution:** OpenAI intends to maintain its aggressive release cadence while working alongside enterprise partners to ensure broad, safe access to step-function improvements across all business functions.

---

[02:32:02](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9122s) How Data Powers Frontier AI at OpenAI

Wendell brings the conversation to the intersection of data and AI, pointing out an underappreciated facet of OpenAI: how deeply the company relies on rigorous data practices to drive model quality and product design. Wendell jokes about spotting distinct Databricks charts whenever Brockman or Sam Altman post graphs online, asking about the role of data platforms within OpenAI’s own development stack.

Brockman agrees that data is foundational to everything they do, comparing data to the indispensable core ingredient of baking: “Fundamentally I think that data is almost the core ingredient upon which—I don't know if you want to think of our models as like a cake or something, right? You know, it's like flour.” High returns come straight from basic data hygiene and ensuring training data remains impeccably clean.

Beyond training, analytics play a decisive role in understanding how millions interact with ChatGPT. Brockman notes that users deploy the platform in unexpected ways—such as patients without direct doctor access uploading complex medical reports to extract healthcare insights. In the agentic domain, raw execution traces are too massive and convoluted for humans to read manually; understanding how an agent navigates a task demands advanced analytical monitoring and recursive AI layers. Brockman reveals that OpenAI now uses its own AI models to evaluate agent behavior, provide oversight, and automate painful operational bottlenecks such as human approval loops in Codex. When Wendell playfully closes by asking if Brockman has any feature requests for Databricks, Brockman responds without hesitation: “for us it's always scalable.”

[02:34:51](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9291s) Infrastructure Demands and the $100M Databricks–OpenAI Partnership

Reflecting on the raw engineering demands at OpenAI, Greg Brockman emphasizes that their primary infrastructure requirement is sheer speed and baseline reliability. With workloads expanding exponentially across global user bases, systems are perpetually pushed to their limits, making fundamental infrastructure stability the foremost priority. When Patrick Wendell quips that the goal boils down to “Better, faster, stronger,” Brockman readily concurs that it comes down to the usual fundamentals of latency and scale.

```mermaid flowchart LR accTitle: Databricks and OpenAI Partnership Integration accDescr: Diagram illustrating the components of the Databricks and OpenAI partnership from enterprise data to Codex and Genie integration. subgraph EnterpriseData ["Enterprise Data Layer"] D["Enterprise Lakehouse Data"] end subgraph PlatformIntegration ["Databricks & OpenAI Integration"] G["Unity AI Gateway"] M["GPT Model Family"] C["Codex Application"] GE["Genie AI Coworker"] end

D --> G G --> M M <--> C C <--> GE ```

Wendell shifts the conversation to the commercial and technical alliance between Databricks and OpenAI, which originated as a $100 million partnership focused on connecting enterprise data directly to the GPT family of models. Over the preceding six months, this collaboration deepened around OpenAI's Codex. Databricks added native support for Codex into its Unity AI Gateway and worked alongside OpenAI to enable Databricks Genie to run directly from within the Codex environment. Asked how OpenAI views the broader role of ecosystem partners, Brockman explains that OpenAI cannot construct the entirety of the AI-driven economy in isolation: “Well, number one is that we view what we're doing is that we're kind of a a spark in the overall engine of of the economy of what you know, this AI revolution that we're all building together.”

- The Databricks–OpenAI partnership was established as a $100 million initiative focused on combining enterprise data with frontier GPT models. - Databricks integrated Codex into its AI Gateway and enabled native invocation of Genie within Codex. - Thousands of joint enterprise customers currently run production applications using the combined stack.

---

[02:36:33](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9393s) The Rapid Evolution of Codex and the Builder Era

Brockman asks the conference audience how many people have experimented with Codex, strongly advising every developer in the room to try it out. He describes the standalone Codex desktop application as a fundamentally distinct form factor that introduces a completely different way of interacting with a computer. Wendell shares that Databricks adopted Codex internally as early adopters, noting that their engineering teams are experiencing substantial productivity gains. He remarks on the remarkable velocity of updates, observing that with every new model checkpoint and desktop client refresh, the capabilities improve visibly. Wendell notes that while Databricks always advises enterprises to evaluate multi-vendor AI strategies, Codex has quickly become an indispensable layer of their internal engineering workflow.

Brockman reflects on the competitive landscape, acknowledging that OpenAI was comparatively late to entering the dedicated coding assistant form factor, but accelerated rapidly through focused execution. He highlights a fact that many in the software community underappreciate: Codex itself is open source, which aligns with OpenAI’s founding DNA of providing open tools that developers can inspect, modify, and integrate across varied workflows. Wendell reinforces that connecting Databricks with Codex through the AI Gateway takes roughly one minute to configure. Both speakers marvel at the sheer accessibility of building software today, with Brockman remarking on the creative liberation enabled by these tools: “If you can imagine it, you can build it.”

---

[02:38:40](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9520s) Reframing AGI: Jagged Intelligence over Binary Definitions

Turning to the day's opening keynote where Ali Ghodsi posited that AGI has effectively arrived for core enterprise workflows, Wendell asks Brockman for his perspective on whether artificial general intelligence is here, far away, or simply poorly defined. Brockman observes that AGI has become deeply subjective—almost an emotional sentiment rather than a rigorous technical benchmark. He argues that treating AGI as a binary milestone or an end state represents a conceptual mistake, proposing instead that AI progress is an ongoing continuum without a fixed finish line.

Brockman describes the true mark of AGI as an assistant capable of autonomously executing any arbitrary objective with the proficiency of world-class human experts. However, he characterizes today's frontier models as exhibiting “jagged intelligence”—hyper-capable and dramatically accelerating in specific domains, while remaining brittle and stunted in others. This uneven progress provides an essential window of opportunity for society to ensure human oversight remains paramount: “And I think this is a great opportunity because one thing we really stand for at OpenAI is keeping humans at the center, right? We really want humans to be setting the goals, being in control, and to make sure that these these systems are in benefit of humanity.” When Wendell asks whether the binary framing of AGI is ultimately counterproductive, Brockman concludes: “I think that the right framing is what are the capabilities, what do we want these AIs to be doing, how can we benefit from them, how can they help us, right?”

---

[02:41:28](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9688s) Returning to the Data Realm: Analytical and Transactional Convergence

Following Brockman's departure to audience applause, Ali Ghodsi returns to the stage to pivot back to core data systems, emphasizing the symbiotic relationship between the two companies. He notes that OpenAI relies on Databricks to process and analyze telemetry across nearly one billion active users, specifically utilizing Databricks' analytical engine to detect jailbreak attempts, monitor abuse, and ensure platform safety.

Ghodsi reviews the morning's announcements across both sides of the database landscape: 1. **The Analytical (OLAP) Sphere:** Organizations run complex business intelligence, data warehousing, and data science queries, supercharged by the newly unveiled Reyden engine and Lakehouse//RT to deliver world-record low latencies over lakehouse data. 2. **The Transactional (OLTP) Sphere:** Nikita Shamgunov introduced Lakebase, a serverless Postgres offering native to the lakehouse that provides low-latency transactional execution, cost efficiencies, and cross-cloud disaster recovery.

Ghodsi poses the fundamental question facing modern data architecture: as enterprises face an intensifying onslaught of autonomous AI agents and real-time operational applications, how can organizations break down the artificial divide separating analytical and transactional data? This sets the stage for the keynote's final architectural revelation: unifying transactional and analytical processing directly on the lakehouse.

[02:42:32](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9752s) The Fragility of CDC and the Failure of HTAP

Reynold Xin returns to the stage to tackle what he frames as the single biggest unresolved challenge in data infrastructure: the persistent chasm between operational and analytical systems. While earlier talks covered Bilal Aslam simplifying data engineering, Reynold's own presentation on Reyden unifying analytics, and Nikita Shamgunov modernizing OLTP, applications still default to landing data on the giant continent of OLTP databases. Once an application succeeds and teams need to analyze that data, engineering teams are forced to construct change data capture (CDC) pipelines. Asking the audience for a show of hands on who actually loves their CDC pipelines, Reynold spots only a faint few in the crowd and jokes that he would love to learn their secret for keeping them running.

In practice, CDC pipelines tail the binlog of an OLTP database, capture individual row-level delta changes, and ship those deltas over to an analytics warehouse to reconstruct database state. This architecture is notoriously brittle and prone to breaking unexpectedly. Data engineers frequently find themselves woken up at 3:00 a.m. because a pipeline failure resulted in silent data corruption. Reynold highlights how widespread the frustration is, noting: “As a matter of fact, some of people at Databricks actually joke about CDC doesn't stand for change data capture. It really stands for continuous data corruption.” To alleviate this pain, cloud providers have marketed concepts like zero-ETL, zero-CDC, or mirroring, but Reynold dismisses these as marketing terms that merely disguise managed pipelines under the hood, leaving all the same underlying failure modes intact.

The database industry previously attempted to resolve this through HTAP (Hybrid Transactional/Analytical Processing)—the theoretical holy grail of database engineering where a single engine handles both transactional writes and analytical scans simultaneously. Yet HTAP systems largely failed to achieve mainstream adoption. Building a single engine forced proprietary architectures without healthy open ecosystems, while compelling systems to compromise on both OLTP transaction speeds and analytical query throughput.

- **CDC pipeline mechanics:** Reads OLTP database binlogs, streams incremental deltas, and attempts to reconstruct state downstream in analytical storage. - **Industry alternatives:** Mirroring and zero-ETL frameworks merely hide pipeline operations rather than removing them. - **Why HTAP failed:** Created closed, proprietary systems that compromised on both OLTP transactional latency and OLAP query performance.

---

[02:45:18](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=9918s) Unifying Row and Column Storage: Introducing LTAP

To solve the divide without the pitfalls of HTAP or CDC, Reynold returns to first principles: the fundamental tension between row-oriented and column-oriented storage. OLTP workloads demand row-oriented storage to perform fast point-lookups (finding a needle in a haystack) and rapid row updates. Analytics systems, by contrast, perform massive scans across billions of records and depend heavily on columnar formats. Looking into the architecture of Lakebase that Nikita Shamgunov introduced—specifically the Safekeepers and Page Servers managing write logs and base storage—Databricks profiled the underlying infrastructure and discovered a critical bottleneck imbalance: these storage services are intensely I/O-bound with local disk and network operations, leaving their CPUs largely idle.

Because those storage nodes possess substantial idle CPU capacity, Databricks realized it could execute real-time transcoding directly on the storage servers at the moment data is ingested, transforming row-oriented records into open columnar formats (Delta Lake and Apache Iceberg). Far from degrading performance, converting to columnar format dramatically shrinks data volume through superior compression ratios—typically between 10:1 and 100:1. Because storage servers are bound by I/O rather than compute, slashing data volume through immediate columnar compression actually accelerates storage throughput instead of bogging it down. As rows land in Lakebase, the underlying storage engine transacts them directly into Delta and Iceberg, allowing analytical engines across the lakehouse to query the freshest operational data with zero pipelines.

Databricks terms this architecture LTAP, short for Lake Transactional/Analytical Processing. Reynold argues that LTAP succeeds where HTAP failed because it unifies storage rather than attempting to force a single compute engine to execute both workloads: “We think LTAP is HTAP done right and accomplished the goals of HTAP without actually having a single query engine for it, but we able to actually unify the storage, which is by far the most important part.” By keeping storage unified, organizations govern a single copy of data across open standards like PostgreSQL, Delta Lake, Iceberg, and Parquet. Databricks is also open-sourcing a foundational library featuring a Parquet elephant logo that directly converts PostgreSQL data into Parquet. The LTAP capability will roll out automatically to all Lakebase instances in the coming weeks, requiring zero configuration or migration steps from users.

```mermaid flowchart TD accTitle: LTAP Architecture Flow accDescr: Diagram illustrating how Lakebase ingest converts row-oriented data into columnar lakehouse tables via storage-layer transcoding without CDC pipelines.

subgraph OperationalApp ["Application Layer"] App["App Writes (Inserts / Updates)"] end

subgraph StorageLayer ["Lakebase Storage Services"] Ingest["Row Ingestion (Safekeepers / Page Servers)"] Transcode["In-Memory Transcoding (Idle CPU)"] Compress["10:1 to 100:1 Columnar Compression"] end

subgraph LakehouseStorage ["Unified Lakehouse Storage"] UnifiedData["Open Formats (Delta Lake / Apache Iceberg / Parquet)"] end

subgraph AnalyticalCompute ["Query & AI Engines"] OLAP["Lakehouse SQL & Analytics Engines"] Agents["Genie & Real-time AI Agents"] end

App --> Ingest Ingest --> Transcode Transcode --> Compress Compress --> UnifiedData UnifiedData --> OLAP UnifiedData --> Agents ```

- **Compression efficiency:** Columnar transcoding achieves data compression ratios ranging from 10:1 up to 100:1. - **Hardware profiling insight:** Storage services (Safekeepers and Page Servers) run I/O-bound on disk and network, leaving abundant CPU headroom for inline transcoding. - **Unified governance:** One physical dataset governed in Unity Catalog supports operational OLTP applications and ad-hoc analytical queries simultaneously. - **Open-source release:** Databricks is releasing an open-source library for direct Postgres-to-Parquet conversion. - **Availability:** Rolling out over coming weeks; every Lakebase table will automatically appear in the user's Lakehouse without manual pipeline configuration.

---

[02:50:04](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=10204s) Demo: Real-Time Banking Intelligence with Lakebase and LTAP

Holly Smith joins the stage to demonstrate Lakebase and LTAP working together in an enterprise scenario: running real-time AI agents on operational banking transactions. Holly sets up a high-stakes financial use case: commercial banks across the Americas generating billions of operational transactions per day. The institution wants an automated agent capable of identifying potential VIP customers in real time as they transact, cross-referencing their financial history, local regulations, and advisor qualifications, and arming the bank branch teller with an intervention recommendation before the customer walks out the door. Emphasizing why previous data architectures failed this test, Holly remarks: “Previously, this would have been impossible. Uh the delay would have been too slow and the intervention would have come too late, but not anymore.”

Navigating through the Databricks console, Holly demonstrates how easy it is to provision Lakebase—a fully managed implementation of open-source PostgreSQL featuring decoupled compute and storage. Creating a new project requires simply assigning a name and accepting defaults, after which the database instance spins up immediately. Compute sizes are not locked into static configurations; users can adjust compute bounds on the fly or configure minimum and maximum autoscale thresholds directly from the management interface.

Holly then highlights Lakebase's Git-like database branching capabilities. From the navigation sidebar, administrators can create an isolated branch off production, branch off an exact historical point in time, or even branch off an existing branch. Because storage is decoupled and uses copy-on-write pointers rather than physically duplicating data blocks, branches are provisioned almost instantaneously without storage bloat. Similarly, Lakebase includes built-in backup and restore controls configured to retain seven days of continuous point-in-time history by default (customizable to longer or shorter retention windows), as well as point-in-time restorations against automated snapshot schedules. Moving past the empty initial configuration, Holly pivots into an active production instance already ingesting live banking streams to inspect its operational tables and analytics integration.

- **Use case scale:** Processing billions of daily transactions across retail bank branches to match VIP customers with qualified advisors mid-visit. - **Lakebase provisioning:** Fully managed open-source Postgres with isolated compute and decoupled storage. - **Elastic scaling:** Dynamic adjustment of minimum and maximum compute bounds directly within the Databricks UI. - **Instant branching:** Zero-copy database branching off production, historical timestamps, or child branches without data duplication. - **Point-in-time recovery:** Built-in continuous rewind capability retaining 7 days of operational history by default alongside scheduled snapshots.

[02:51:58](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=10318s) Live Demo: Benchmarking Approaches to Unify Real-Time and Analytics

Holly Smith walks through a live demonstration evaluating how to calculate high-value outliers and prospective VIP customers by scanning historic account balance data across transactional systems. The live dashboard tracks three critical operational health metrics:

- **Transactions Per Second (TPS)**: Displayed as a live green line chart on the right side. If this drops, customer transactions stall and branch staff face an outage. - **Analytical Query Latency**: Displayed in blue at the top left, measuring how many seconds the analytical job takes to return. - **Data Staleness**: Displayed in purple at the top right, measuring data age in seconds to show how up-to-date the results are.

```mermaid flowchart TD accTitle: Comparison of Query Approaches for Analytical Workloads on Transactional Data accDescr: Three architectural patterns tested in the demo showing trade-offs between OLTP direct compute, CDC pipelines, and LTAP.

subgraph S1["1. Direct OLTP Compute"] D1["Direct Query on Lakebase Compute"] -->|Runs >60s| F1["TPS Drops Sharply & Threatens Outage"] end

subgraph S2["2. Change Data Feed (CDC)"] D2["Lakebase Change Data Feed"] -->|Takes 13s| F2["25s Stale Data & Extra Maintenance"] end

subgraph S3["3. LTAP Architecture"] D3["Real-Time Warehouse + Unity Catalog"] -->|Runs in Milliseconds| F3["0s Staleness & Zero TPS Impact"] end ```

Smith tests the first conventional path: routing the complex analytics query directly to existing Lakebase compute via the Databricks SQL editor. As the query executes, the green TPS metric drops visibly, demonstrating severe compute contention. The analytical query hangs for so long that Smith jokes with the audience:

> “Well, you all know my name. It looks like we have time for everyone to go around the room and introduce themselves.”

The query finally completes after more than 60 seconds. The severe TPS degradation shows that running complex analytics directly on operational database compute risks taking down production workloads, making it unusable for real-time agentic applications.

---

[02:54:22](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=10462s) Evaluating Change Data Capture (CDC) vs. Native LTAP

Smith turns to the second traditional approach: utilizing change data capture via Lakebase's change data feed (CDF). While CDF is designed for audit logging rather than sub-second analytics, it isolates operational throughput so the green TPS line remains completely flat. The query finishes in 13 seconds, but the underlying data is already 25 seconds stale. Smith highlights that in external PostgreSQL environments, cross-database replication pipelines often inflate data staleness to several minutes while introducing significant infrastructure bills and engineering overhead to maintain intermediary tools.

Smith then demonstrates the third approach: Databricks LTAP (Lake Transactional/Analytical Processing). Within Unity Catalog, the Lakebase operational table appears as a standard managed table accessible to both Delta Lake and Apache Iceberg readers without building data pipelines or replication jobs. Because the underlying storage exposes an open Amazon S3 path, external engines like open-source Apache Spark can query it directly.

Running the exact same query against an LTAP real-time warehouse returns results in milliseconds. The query achieves zero staleness, leaves operational TPS unaffected, and cuts compute costs significantly by avoiding heavy, long-running queries and intermediate infrastructure. Smith summarizes the breakthrough:

> “Finally, we no longer have a divided view of the world between operations and analytics. We no longer have to choose.”

- **Lakebase Direct Compute**: Latency > 60 seconds; severe TPS drop; dangerous production interference. - **Change Data Feed (CDC)**: Latency 13 seconds; TPS stable; data staleness of 25 seconds; requires pipeline management. - **LTAP Architecture**: Latency measured in milliseconds; zero TPS degradation; instant freshness; open Delta/Iceberg format storage backed by Amazon S3.

---

[02:57:08](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=10628s) Unifying the Data Islands: Day 1 Keynote Architecture Recap

Ali Ghodsi returns to close the keynote, framing the evolution of data architecture as turning isolated data silos into a unified continent:

> “We were able to unify all these different islands, remove all of the data movement into just this one big Pangea.”

Ghodsi reviews how Databricks has systematically unified fragmented technology tiers over time: 1. **Data Engineering & Data Science**: Unified through Apache Spark. 2. **Data Warehousing & BI**: Unified through the Lakehouse. 3. **Streaming & Real-Time Analytics**: Unified through the Reyden engine and Lakehouse//RT. 4. **OLTP & OLAP (Operational & Analytical)**: Unified through Lakebase, open table formats (Delta Lake and Apache Iceberg), and LTAP.

```mermaid flowchart LR accTitle: Databricks Evolution of Architectural Unification accDescr: Historical progression of Databricks unification across analytics, data warehousing, real-time, and transactional workloads.

U1["Spark\n(DE + Data Science)"] --> U2["Lakehouse\n(Warehousing + BI)"] U2 --> U3["Lakehouse//RT & Reyden\n(Real-Time Engine)"] U3 --> U4["Lakebase & LTAP\n(OLTP + OLAP in Open Formats)"] ```

Ghodsi recaps the major product milestones unveiled across the three tiers of the enterprise agent stack:

- **Agentic Data Foundation**: Lakehouse//RT, Reyden engine, Lakebase (featuring cross-cloud disaster recovery), LTAP, and native Lakeflow connectors. - **Context Layer**: Genie Ontology (serving as the foundational business context graph), Unity AI Gateway (providing multi-model governance, observability, and cost management), Open Sharing, and Omnigent. - **Agentic Developer & Work Tier**: Genie One, Genie Code, and Genie Agents. - **Packaged Lakehouse Applications**: Lakewatch (agentic security information and event management built on the security lakehouse following the Panther acquisition) and CustomerLake (an agentic Customer Data Platform).

Ghodsi concludes by directing attendees to breakout sessions beginning at 11:30 AM and previewing the Day 2 morning keynote by co-founder Matei Zaharia focused on Omnigent.

## Notable quotes > [36:44](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=2204s) AIs are plenty smart. AI does not have an intelligence problem, it has a context problem. — Ali Ghodsi

> [01:40:51](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=6051s) In data engineering, failures are silent and permanent. — Bilal Aslam

> [02:09:23](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=7763s) And now we can confidently say we're going to have more software generated in the next year than in the history of humanity. — Nikita Shamgunov

> [01:24:11](https://www.youtube.com/watch?v=Qux8E-L1mk8&t=5051s) For us what we truly believe is an AI transformation is a data transformation, and that's what we've undertaken. — Magesh Bagavathi

## Mentioned Agent Bricks, Agno, Ali Ghodsi, Alteryx, Amazon EMR, Amazon S3, Anthropic, Apache Airflow, Apache Flink, Apache Iceberg, Apache Kafka, Apache Spark, Arsalan Tavakoli-Shiraji, Atlan, AtScale, AWS, Bilal Aslam, Claude Code, Claude Opus, Cognition, Coinbase, CrewAI, CustomerLake, Databricks, Databricks Runtime (DBR), dbt, Decagon, Delta Lake, Delta Lake 5, Delta Sharing, Elise Georis, Enverus, Expo, Fable, Fed Cohen Freue, Foot Locker, General Motors, Genie, Genie Agents, Genie App Builder, Genie Code, Genie One, Genie Ontology, Genie ZeroOps, Glean, Google, Google Analytics, Google Calendar, Google Cloud Platform, Google Docs, Google Drive, Google Gemini, GPT-4, GPT-5.3, GPT-5.4, GPT-5.5, Greg Brockman, HackerOne, Holly Smith, Humanity's Last Exam, Iceberg v3, Iceberg v4, Insulet, Jack Naglieri, Jira, Ken Wong, Lakebase, LakeBridge, Lakeflow, Lakeflow Connect, Lakeflow Designer, Lakeflow Jobs, Lakehouse, Lakehouse//RT, Lakewatch, LangChain, LlamaIndex, Lovable, LTAP, Magesh Bagavathi, Mastercard, Matei Zaharia, MemSQL, Merck, Meta, Michael Stonebraker, Microsoft Azure, Microsoft SQL Server, Microsoft Teams, MLflow, Model Context Protocol (MCP), Mythos, Neon, NetSuite, Nikita Shamgunov, Omnigent, Omnipod, OntoRank, Open Sharing, OpenAI, OpenAI Codex, OpenClaw, OpenCodeInterpreter, Oracle, Panther Labs, Parquet, Patrick Wendell, PepsiCo, Performance Pulse, Plaid, PostgreSQL, PySpark, Replit, Reyden, Reynold Xin, Ryan Blue, Salesforce, Satya Nadella, SharePoint, SingleStore, Slack, Snowflake, Spark Real-Time Mode (RTM), SpendWise, Tabular, TEDD (Transformer-Enabled Drug Discovery), Tonal, Uber, UC Berkeley AMPLab, UniForm, Unity AI Gateway, Unity Catalog, Vercel, Warner Music Group, Workday, ZeroBus

## Video description Watch the full keynote from day 1 of Data + AI Summit, the world’s largest data and AI conference.

Databricks co-founders, product leaders, AI builders, and global business leaders unveil breakthrough product launches and the stories of organizations rewriting what's possible with AI.

0:00 Databricks Co-founder & CEO Ali Ghodsi welcomes attendees, customers, partners, and the ecosystem to Data + AI Summit 7:59 Is AGI here today? 10:12 AI doesn’t have an intelligence problem, it has a context problem 11:21 What it takes to get AI to work in the enterprise 19:29 Open table format unification 21:58 The agentic data foundation 31:16 Announcing Unity AI Gateway: multi-AI governance and cost control 36:45 AI and the context problem 38:50 Announcing Genie Ontology: a living context graph for your business 43:16 Announcing Genie One: the data-smart AI coworker 45:02 Announcing Genie Agents and Genie App Builder: build reusable agents and vibe code business apps 45:52 Genie Code: automate building data and ML pipelines 46:33 Genie ZeroOps: automate maintaining data and ML pipelines 48:27 Omnigent: a meta-harness of AI agents 49:20 An agent system of record, the future software stack 54:09 Lakewatch: an agentic SIEM built on the security lakehouse 55:58 Databricks to acquire Panther, a leading AI SOC platform 57:05 Announcing CustomerLake: the agentic CDP built on the Lakehouse 59:36 Genie: the new AI coworker that knows your business 1:00:59 Genie + context 1:07:29 How Genie Ontology works 1:11:20 Demo: Genie One 1:19:22 Genie One provides full co-work capabilities and is now GA 1:23:42 Magesh Bagavathi (Global Chief Data and AI Officer, PepsiCo) on AI transformation 1:31:15 A new era of agentic data engineering with Lakeflow 1:43:00 Announcing Genie ZeroOps: put data and AI operations on autopilot 1:57:37 Announcing Reyden: a breakthrough new engine for real-time workloads 2:06:22 Announcing Lakehouse//RT - real-time performance on a unified lakehouse 2:09:23 Nikita Shamgunov: Serverless Postgres for apps and agents 2:18:47 Fed Cohen Freue, EVP of AI & Data Operations at Mastercard, on standardizing on Lakebase 2:25:15 Greg Brockman, President and Co-Founder, OpenAI & Patrick Wendell Co-Founder and VP of Engineering, Databricks 2:42:45 Reynold Xin on solving the biggest database engineering challenge: how do we unify transactional and analytical data 2:47:45 Announcing LTAP: lake transactional/analytical processing architecture 2:50:01 Demo: Lakebase and LTAP and why you need to combine your transactional and analytical data 2:57:07 Product announcement recap

Speakers: Ali Ghodsi, CEO and Co–Founder, Databricks Reynold Xin, Co-Founder and Chief Architect, Databricks Patrick Wendell, Co-Founder and VP of Engineering, Databricks Arsalan Tavakoli-Shiraji, Co-founder & SVP of Field Engineering, Databricks Greg Brockman, President and Co-Founder, OpenAI Fed Cohen Freue, EVP, AI & Data Operations, Mastercard Magesh Bagavathi, Chief Data & AI Officer, PepsiCo Nikita Shamgunov, VP, Engineering, Databricks Ken Wong, Sr. Director, Product Management, Databricks Bilal Aslam, Sr. Director of Product Management, Databricks Elise Georis, Senior Manager, Product Management, Databricks Holly Smith, DevRel, Databricks

[Source](https://www.youtube.com/watch?v=Qux8E-L1mk8)

Reading summary

Databricks leadership argues that enterprise artificial intelligence is constrained not by raw reasoning intelligence, but by a lack of context, governance, and unified data architecture. To solve this, the company presented an integrated stack designed to eliminate the boundary between analytical and transactional data while deploying governed, autonomous agents across business operations. The Context Gap and Agent Traversal Limits (36:44) Ali Ghodsi posited that frontier AI models have already reached expert-level capabilities on theoretical benchmarks—such as Humanity's Last Exam—yet fail to deliver enterprise utility when separated from internal business data. According to Ghodsi, the dominant approach of deploying coding and knowledge agents on live traversal loops across enterprise files leads to extreme latency (queries stalling 10 to 15 minutes), exorbitant token costs, and low accuracy. Furthermore, ungoverned agent adoption presents immediate operational risks; Databricks highlighted that over 10% of community-contributed skills in open agent tools were found to contain malicious code. Unifying Transactional and Analytical Data with LTAP (165:18) Historically, organizations separated row-oriented transactional databases (OLTP) from columnar analytical warehouses (OLAP), using fragile ETL or Change Data Capture pipelines that introduced data staleness and synchronization drift. To address this, Databricks unveiled Lakehouse//RT powered by Reyden, an optimization engine trained on execution traces from trillions of queries. Databricks claims Reyden sustains 12,000 QPS on TPC-H benchmarks with tail latencies below one second. For transactional workloads, Databricks introduced Lakebase, a lake-backed, serverless PostgreSQL engine that demonstrated over 600,000 operations per second in TPROC benchmarks. To bridge both systems without performance trade-offs, Databricks introduced LTAP. Because Lakebase storage nodes are intensely I/O-bound with largely idle CPU capacity, the architecture executes real-time transcoding on ingestion, converting row-oriented writes directly into open columnar formats (Delta Lake and Apache Iceberg). In benchmark demonstrations, this transcoding delivered millisecond analytical queries on live operational data with zero staleness and without reducing transactions per second. Enterprise-Wide Automation via Genie and ZeroOps (43:16) Databricks launched Genie One, expanding its conversational data coworker across entire enterprises without requiring non-technical workers to provision backend compute. In production, enterprise design partner PepsiCo used Genie to replace fragile custom Retrieval-Augmented Generation stacks, transitioning its procurement platform SpendWise to conversational interactions. PepsiCo Global Chief Data and AI Officer Magesh Bagavathi reported processing nearly 30,000 conversational interactions post-launch, with an overarching goal to consolidate over 30,000 legacy static reports down to roughly 50 specialized Genie consoles across 320,000 employees. To manage the operational burden of agent-generated pipelines, Databricks announced Genie ZeroOps. The background agent continuously monitors machine learning and data pipelines using per-table ML models to detect failures. It evaluates lineage graphs via Unity Catalog, establishes consensus on root causes using specialized sub-agents, and verifies synthesized code fixes inside isolated shallow clones of production data before alerting engineers for single-click remediation. Reframing AGI as Jagged Intelligence (158:40) In a joint discussion, OpenAI President Greg Brockman argued against treating Artificial General Intelligence as a binary, all-or-nothing threshold, describing it instead as a continuous trajectory of "jagged intelligence." Current models display superhuman mastery in specific domains—such as software development, where Brockman noted AI has shifted from handling 20% of routine grind work to approximately 80%—while remaining stunted or brittle in others. Brockman stressed that this uneven capability creates an essential window to institutionalize human-in-the-loop governance. The commercial alliance between OpenAI and Databricks, initiated as a $100 million partnership, links OpenAI's model family and Codex directly into Databricks' governed data layer. Conversely, Ghodsi highlighted that OpenAI relies on Databricks' analytical engine to process telemetry across nearly one billion active users to detect abuse, jailbreaks, and platform safety violations. Databricks envisions a unified "data Pangea" where open lakehouse storage eliminates siloed operational databases, point-serving layers, and fragmented analytics. However, realizing this agentic architecture depends on whether enterprises can contain runaway inference budgets, prevent sprawl from automated code generation, and trust background remediation agents in mission-critical environments.

Source quotes

[36:44] Ali Ghodsi: AIs are plenty smart. AI does not have an intelligence problem, it has a context problem. [100:51] Bilal Aslam: In data engineering, failures are silent and permanent. [129:23] Nikita Shamgunov: And now we can confidently say we're going to have more software generated in the next year than in the history of humanity. [84:11] Magesh Bagavathi: For us what we truly believe is an AI transformation is a data transformation, and that's what we've undertaken.

Original transcript

[0:08] The world created an [music] artificial [0:09] brain [0:12] then locked it in a room. [0:13] >> [music] [0:15] >> It can answer almost any question, [0:18] but it's isolated, cut off [music] from [0:20] your operations, [0:22] your world. [0:24] >> [music] [0:25] >> It knows how to think. [0:28] Now, let's show it what's real. [0:33] Real is every decision, [0:36] every transaction, [0:38] every signal, [0:40] every discovery, [music] [0:41] every promise, [0:43] every life changed. [0:46] Together, we give the brain context with [0:48] all of your data, [0:50] but you're in control. [0:53] You decide what it knows, [0:55] what it can access, [music] [0:56] who can access it, [0:58] what it can do. [1:00] Run it on anything, any way you want. [1:06] Because [music] the ultimate brain was [1:08] just the beginning. [1:10] The real breakthrough [1:11] is what you build with it. [1:15] The systems you transform, [1:17] the discoveries you unlock, [1:20] the lives you change. [1:25] The possibilities are in your hands. [1:33] Let's make it real. [1:42] >> Welcome to the stage, Databricks [1:44] co-founder and CEO, Ali Ghodsi. [1:59] >> All right, super excited to be here. Um [2:02] that was awesome video. I don't know if [2:03] many of you in the audience maybe [2:05] identified many of those uh clips that [2:06] we were showing are actually customers [2:08] and their use cases that you could see [2:10] in that We're hopefully we play it [2:11] again. You can see it again. Okay, so [2:13] super excited to be here. [2:15] Uh we have an awesome event. This is [2:16] actually the largest data and AI summit [2:18] that we've ever done, okay? So, super [2:20] cool. [2:23] >> [applause] [2:24] >> So, just a couple of days I was looking [2:26] and we only had 97,000 people signed up, [2:29] but we now crossed 100,000. So, super [2:31] excited about that. Um [2:33] >> [cheering] [2:33] >> my favorite thing is actually like we [2:34] have 174 countries represented, which is [2:38] an insane number. Okay, this is the [2:39] number of languages. So, it's truly a [2:41] international global event. There's [2:43] people from all over the world that [2:45] actually flown in for this. So, uh super [2:48] exciting. Uh and we actually have over [2:51] 31 actually 31 They have the number [2:52] exact down. Okay, so 31,309 [2:56] people, uh which makes it the largest [2:59] data and AI conference in the world. Uh [3:01] so, super cool. [3:04] >> [applause] [3:06] >> And this conference, as you know, has [3:08] really a community event, and it started [3:11] in its roots with open source. We [3:13] started data and AI summit. It was [3:15] called Spark Summit back in the day in [3:17] 2013, and we started with the open [3:20] source project Spark, [3:22] which now actually has over 3 billion [3:23] downloads a year. [3:25] Uh but we've been adding open source [3:26] projects over the years. So, you know, [3:28] Delta Lake, Iceberg, MLflow, Unity [3:31] Catalog that we open sourced. Uh but [3:33] we've also added really Postgres. We've [3:34] really embraced the Postgres open source [3:37] project. So, you'll see much more about [3:39] that uh throughout this conference. Um [3:42] in fact, Postgres just Databricks is [3:44] really embracing it. We have 1/7 of the [3:47] governors of the project actually work [3:48] for us, and over 6% of the committers of [3:51] the project. We're really excited about [3:52] the Postgres project. We think that's [3:54] the future of databases. [3:56] Uh and this weekend, my co-founder Matei [3:59] open sourced a project called Omnigent. [4:01] Okay? So, it's a meta hardness. Uh so, [4:03] check it out. Um I'll talk more about it [4:05] and you'll also hear from him uh talking [4:07] about it. Okay. So, that's the [4:09] community. We also have a lot of people [4:10] from the community that are speaking at [4:12] this conference. We're excited about [4:13] that. Uh we're going to hear from the [4:15] co-founder and CTO of OpenAI, Greg [4:17] Brockman, Satya Nadella, many other [4:20] leaders and CEOs uh and co-founders of [4:23] various companies, AI companies like [4:24] Anthropic, Cognition, Glean, Decagon, [4:27] Agno, Lama Index, Crew AI. So, all of [4:31] those are here. Super exciting program. [4:33] And they're all part of a bigger [4:34] ecosystem that's gathered here this [4:36] week. [4:37] Uh ecosystem includes system [4:39] integrators, people who built on top of [4:41] Databricks, or data sharing and AI [4:44] partners that we have. [4:45] Um this is super exciting. And I want to [4:47] give a special thank you to all of our [4:49] partners that made this event possible. [4:51] So, let's give them a round of applause. [4:54] >> [applause] [4:58] >> So, please check them out. They, as well [5:00] as the companies that I showed in the [5:01] previous slide, are in the expo hall. [5:03] They have a lot of interesting [5:04] innovations. So, go check those out. Um [5:07] and this conference would not be [5:08] possible without all of our customers. [5:11] They're the ones that are actually [5:12] chewing the doing the truly amazing [5:14] things on top of the platform. All those [5:17] use cases that we talk about, that's [5:18] really where it's happening. It's all of [5:20] you that are actually implementing all [5:21] these use cases and actually driving the [5:23] data and AI change in the world that's [5:25] happening. [5:26] Um so, very excited about those. I want [5:29] to highlight just three of them cuz I [5:31] think they're inspirational. [5:32] Uh number one is Insulet. So, Insulet [5:35] has a product called Omnipod. And what [5:38] the Omnipod does is that it helps people [5:41] with diabetes. [5:43] Um it collects actually has a glucose [5:45] monitor. And the glucose monitor just [5:47] measures the level of glucose in the [5:49] blood. And then, of course, leveraging [5:51] lots of data and AI, they can actually [5:54] personalize exactly what's happening [5:57] with your [5:58] glucose level and release insulin so you [6:00] can manage your diabetes. So, [6:02] game-changing for all the folks that [6:04] have diabetes in the world. [6:06] So, that's just one of the main many [6:08] 20,000 impacts that, you know, we've [6:10] talked about here. [6:11] Another favorite example of mine, and [6:13] actually many of the early employees [6:15] have one of these devices, [6:17] is a tonal device, okay? So, this is a [6:19] really cool device. It's just that big, [6:20] and you put it on the wall, but then you [6:22] can do all your workouts on it, and it [6:23] just creates this crazy resistance so [6:26] you can do the workout on the device. [6:29] But what's really cool about it is that [6:30] they're collecting 100 billion seconds [6:32] every year of workout data, and they're [6:35] personalizing it for you so that you get [6:37] actually a personalized workout, just [6:40] like your personal PT, [6:42] with diet [6:43] advice, and all of that, of course, [6:46] possible thanks to leveraging data and [6:47] AI on the platform. So, super awesome. [6:51] Okay, and then the final one, [6:53] I'm really excited about this one, is [6:55] Merck. [6:56] Merck actually trained a [6:57] transformer-based [6:59] language model. It's called Teddy. Teddy [7:02] stands for transformer-enabled [7:05] drug discovery. Okay, so it's one of [7:07] those transformer models. It's like an [7:09] LLM, you know, next token predictor. [7:12] It was trained on Databricks, actually [7:13] with the Databricks research team, but [7:15] it's not a language model. It doesn't [7:16] predict the next word. It predicts gene [7:20] regulatory networks. So, it can predict [7:23] which genes in the body are actually [7:26] causal for diseases. So, it can predict, [7:28] okay, if you change this cell, then this [7:31] will happen with the next cell. This [7:32] cell is just a causal cell. This cell [7:34] is, you know, just a reactive cell. So, [7:36] this helps speed up drug development [7:39] significantly. So, over 100 million [7:40] cells that it's been trained on, [7:42] uh so it's super super cool. So, uh [7:45] truly inspiring AI use case. [7:47] Uh so, AI is top of mind for all of us. [7:50] We're here at the Data and AI Summit. Uh [7:52] so, uh I want to have a little bit of [7:54] audience participation here today. Okay? [7:57] Um so, [7:59] is AGI here yet today? So, that's a [8:02] question I have for the audience. I want [8:03] to ask everybody, raise your hands if [8:06] you think AGI is already here. That we [8:07] already have AGI. So, let's see, raise [8:09] your hands. [8:12] Okay, I'm going to do like a statistical [8:13] estimation here. [8:15] Uh I'm going to say 5% at least in this [8:17] area. I I can't see all the way down [8:19] there, okay? Uh are raising their hands [8:21] saying that AGI is here. Because I would [8:22] say like 90 to 95% of you say that AGI [8:26] is not here. Okay. So, I have a question [8:29] for you then. [8:31] Okay. Which one of you can solve this? [8:33] So, the question is, compute the reduced [8:35] 12-dimensional spin bordism of the [8:37] classifying space of the Lie group G2. [8:39] Okay? So, [8:41] audience participation. You can just [8:42] shout it out if you know the answer. [8:43] Okay, actually you can come up here and [8:45] just share the answer with us if you [8:46] know the answer. Okay? How many know the [8:48] answer? Can we see? Raise your hands. [8:52] Oh, there's like one person. Okay. [8:53] >> [laughter] [8:54] >> I should get them on stage. Okay. [8:56] All the frontier agents and AIs today [9:00] can solve this. Okay? So, this is like a [9:02] simple question. This is not a hard [9:03] question. It's a hard question for us, [9:05] uh but for the uh frontier agents, this [9:08] is trivial. It's part of actually um uh [9:12] humanity's last exam, which is a [9:14] benchmark that they're all uh really [9:15] focused on. That benchmark has 2,500 [9:18] questions. Uh they look kind of like [9:20] that, but in all kinds of categories, [9:22] you know, it could be like image [9:23] recognition or various kinds. And today, [9:27] all of them pretty much can solve half [9:28] of those questions. So, we believe that [9:31] AGI is already here. Okay? So, [9:33] artificial general intelligence has [9:35] already arrived. Um in fact, I came to [9:38] the United States to do research at UC [9:40] Berkeley at the lab called the AMP Lab, [9:43] where the A was for algorithms and AI. [9:45] It was the largest AI lab at this time, [9:47] 2009. And actually, this conference came [9:49] out of software that we built at that [9:50] lab. And the definition of AGI that we [9:53] had back then, we've already satisfied [9:55] that by leaps and bounds. In fact, went [9:57] back and I asked many of the people that [9:59] were in the lab, "What do you think? Do [10:01] Do we have Have you passed AGI or not [10:03] the way we defined it back then in [10:04] 2009?" And everybody says, "Yeah, of [10:06] course. Of course we do it the way it [10:07] was back then." But, you know, goalposts [10:10] are changing and so on. So, AGI is [10:12] already here. Okay, it's plenty smart. [10:15] Uh, we don't need it to be more [10:16] intelligent. Um, AI does not have an [10:19] intelligence problem right now. These [10:20] are not the kind of problems that we [10:21] need inside of our organizations. The [10:24] problem we have is that AGI is not [10:26] really, [10:27] um, permi- [10:28] completely, um, permeating our [10:31] organizations. So, inside of your [10:33] companies, you don't have each hundreds [10:35] of agents working for you doing [10:37] autonomous work, collaborating with each [10:39] other, coming up with proposals, [10:41] negotiating, and then we're just [10:42] managing agents. [10:44] Most of us are just using chatbots and [10:46] asking them. Or maybe we're using coding [10:48] tools and they're writing code for us [10:50] and we're using it agentically like [10:51] that. Um, but we're not actually sort of [10:54] fully realized that vision that [10:55] everybody's talking about. And that's my [10:57] why maybe many of you didn't raise your [10:59] hand when I said, "Is AGI here?" Uh, so [11:01] the question is, how do we actually [11:03] enable this at work? So, that's what the [11:06] next couple of days will be about. We [11:08] want to figure out, how do we enable all [11:10] of you, uh, to sort of lead this [11:12] revolution. It's basically our jobs here [11:14] to figure out, how can we actually make [11:16] this AI work? That's what you all are [11:18] doing. [11:19] So, that's what we're focused on. So, in [11:21] a nutshell, what we want to do is we [11:22] have all these AIs. They've been [11:23] released by many of the vendors. There's [11:25] open source, there's frontier models. [11:28] How do we figure out to take all the [11:30] data, all the processes in your [11:32] organization, everything that you have [11:34] in your head, and provide it as context [11:37] to the AI. If we can do that, [11:40] we think that the AI can already solve [11:42] many of the tasks that we want it to [11:43] solve. Okay? So, if we just give the [11:46] context to these very smart models that [11:48] can solve those super hard problems, [11:50] they'll be able to do amazing things [11:51] inside of our companies. You know, most [11:53] of the tasks we're doing is not about [11:55] solving 12-dimensional spin boardisms. [11:58] They're about, you know, getting the [11:59] information out of Salesforce, [12:01] summarizing it, putting it in another [12:02] place, going through it, preparing it, [12:04] and those kind of tasks. So, you just [12:06] need to get all that context. [12:08] So, it's as simple as that. [12:10] But, getting this context into the AI is [12:13] harder than one could imagine. Um it's [12:16] actually really, really difficult. [12:17] And getting the perfect enterprise [12:19] context is elusive. So, this is the [12:20] problem we've been focused on, and we're [12:22] really focused on this problem at [12:23] Databricks. [12:24] Um so, context is actually itself many [12:27] other things. It's not just saying [12:29] context, how do we get it to the AI? [12:31] Context means [12:33] how do we get all the data that you have [12:34] in your organization, and actually [12:36] connect it to the AI? [12:38] Okay? Data could be what are all the [12:40] meeting notes? So, every meeting that [12:41] happens in the organization should be [12:42] recorded, and we should get the [12:44] transcripts of that. And we need to [12:46] reorganize our processes so that we can [12:47] do that. And actually at Databricks, we [12:49] have forward deployed engineers that [12:50] come and help reorganize things like [12:52] that. But, it also means connecting to [12:55] data silos that you have, or data that's [12:56] sitting tucked away, and you don't have [12:59] any uh access to it because maybe [13:01] security has not approved it. So, this [13:03] is step number one. How do we get your [13:04] data AI ready so that we can actually [13:06] give it to the [13:08] uh agents? [13:09] Okay, but let's say we figured this out, [13:11] and we figured out how to get access to [13:12] all the data, then we're still not done. [13:14] The context problem isn't solved because [13:17] we could just take open claw [13:19] and just unleash it on all of your data. [13:21] Okay? When open claw was released, over [13:23] 10% of the skills in it uh were actually [13:26] malicious. Okay? So, step number two, [13:29] how do we do this securely? How do we do [13:31] that with control? Okay, so first is [13:33] context. Now, how do we do this with [13:35] control? [13:36] Uh so, that's really really important. [13:38] If we're going to have agents roaming [13:39] around doing various things, we have to [13:41] make sure that they uh are satisfying [13:44] our security policies, that we can audit [13:45] them, and that they don't uh do [13:46] something that we don't uh want them to [13:48] do. [13:49] Okay, let's say we solve this as well. [13:51] It's still really hard. Um if we just [13:53] have agents running around for looping, [13:56] live checking through all of the data, [13:58] then this going to become extremely [13:59] costly. So, we have to also solve the [14:01] cost problem. Okay? This is sort of just [14:04] shooting up at most of our customers. [14:05] We're seeing the cost just completely [14:07] run away. Uh Uber CEO said that in one [14:09] quarter they blew through the whole [14:11] year's annual AI budget. Okay? If this [14:14] continues, um [14:16] it's completely unsustainable for more [14:18] most organizations out there. You cannot [14:20] have your costs completely outrun your [14:22] revenue. You will go bankrupt. Okay? And [14:25] your companies will not allow that. So, [14:27] they'll just shut it down or stop it. [14:28] Okay? So, it's just infeasible. So, we [14:30] have to also solve the cost problem. [14:32] But even if we solve all of these [14:34] things, including the cost, we get to [14:36] the most important thing, in my opinion, [14:38] which is how do we make sure that we can [14:40] do that uh without lock-in? How can we [14:42] do that in a way where you have choice? [14:45] Okay? This is super important. This is [14:46] the hardest one [14:48] because some of you are representing [14:49] organizations that are less than 10 [14:51] years old. [14:52] So, this maybe doesn't apply to you, but [14:54] most of the organizations in the room, [14:56] over 90% of you represent companies who [14:58] have a long history. They've been around [14:59] for 20, 30, 50, hundreds years, some of [15:02] them. And the thing I've seen in the [15:04] last 20 13 years [15:07] is that the stack that you have at your [15:09] companies is just very complicated. Over [15:11] the years, you keep buying new software. [15:14] You keep [clears throat] adding more [15:14] things to that stack. And you just get [15:16] locked in to all those vendors. And they [15:18] come and go, but the software remains. [15:20] And there's always someone who wants [15:21] that piece of software in your stack, so [15:23] it's hard to rip it out. So, how do we [15:25] do this without getting locked in both [15:27] to the data, the context, but most [15:29] importantly also now to the AI? We don't [15:31] want to get locked in just to one [15:32] particular AI. We want to have choice [15:34] when it comes to the AI. [15:35] So, that's what we're focused on today. [15:37] My whole keynote is just focused on how [15:39] do we solve these four, okay? [15:41] Uh so, I hope I can convince you uh [15:43] today that we're making some steps [15:45] towards solving these four. Uh I'm not [15:47] going to say we've completely solved it. [15:48] Context is solved, but uh at least we're [15:51] making uh some big leaps towards solving [15:53] these four things. So, the four things [15:54] we're focused on is we want you to have [15:56] choice and not get locked in. Uh we want [15:58] to be able to get the security and the [16:00] control and the cost control, as well as [16:03] the context, and then get that to the [16:05] AI, and that way we can have more impact [16:07] together. Okay. So, how do we do that? [16:09] So, let's start with choice, okay? It [16:12] started with your data, as I mentioned. [16:14] So, we have all the data, [16:16] and we want to get that data [16:18] in an AI-ready form. So, step number one [16:20] is [16:21] we've actually made huge progress just [16:23] in the last year to get that data and [16:25] move it into an open lakehouse. So, open [16:29] infrastructure on top of any of the [16:30] clouds, and we get your data ready [16:32] there. That's what Lakeflow does. [16:34] So, Lakeflow has actually been [16:37] developing a lot of connectors over the [16:39] last year. So, 2 years ago when I was [16:41] here, we barely had any connectors, and [16:43] today we now have over 100 connectors. [16:46] So, Lakeflow can now connect to all [16:47] these different systems that you have. [16:48] It's not just Salesforce, Workday, [16:49] NetSuite. It can pull data from Meta or [16:53] Google Analytics or from any of your [16:55] docs or any of the corpus. And if [16:56] there's any data source that you do not [16:58] have, please let us know, and we can [17:00] help you get those into the open uh [17:03] lake. [17:04] Okay. [17:05] And it's not just getting the data in [17:06] there. We can do it in many different [17:07] modalities. So, three things that I [17:09] would like to highlight. One is we [17:10] released Zero Bus last year. It is now [17:12] GA, then actually we've added a lot of [17:14] capabilities to it. So, what does Zero [17:16] Bus do? So, as the name implies, you [17:18] don't need a bus anymore for the data [17:21] infrastructure. [17:22] So, instead of using something like [17:23] Apache Kafka, you can just hit this API [17:27] of ZeroBus, and you can give it data as [17:29] it comes in. It could be bursty, it [17:30] could be tiny rows. You don't need to [17:32] worry about having millions of files or [17:34] any buffering or anything like that. [17:36] ZeroBus just takes care of taking all [17:38] those entries that you hit it with at [17:39] very high rate, and it just makes sure [17:41] that those appear in your open lake [17:45] in a way that you can probe it later. [17:47] Okay? Scalable, manageable. [17:50] Okay. So, that's great. Um [17:52] and then last year we announced Spark [17:55] real-time mode. Spark real-time mode, or [17:57] RTM as we call it, is an open-source [18:00] framework that lets us actually get down [18:01] to tens of milliseconds. So, this is [18:03] really amazing. Uh so, we can do now [18:05] operations super fast. Historically, [18:08] Spark was great for everything, but you [18:10] couldn't really get down to 10 [18:11] milliseconds. Okay? So, the real-time [18:13] streaming would always take this [18:15] microbatch of 1 second or so. But with [18:17] real-time mode, we can now really make [18:19] that super fast. So, many of you, for [18:22] your streaming infrastructure, you might [18:23] have Apache Flink. That was the only [18:25] option that uh many of you would pick. [18:28] But now you can actually do a lot of [18:29] those workloads directly in Apache Spark [18:32] with real-time mode, which is open [18:33] source. [18:34] Okay. And the final one is LakeFlow [18:36] Designer. [18:37] Uh I personally actually really like [18:39] doing all of my data engineering and [18:40] data movement using LakeFlow Designer. [18:43] So, the way LakeFlow Designer works is [18:45] that it kind of looks like Alteryx. Um [18:48] it's visual. You don't need to see any [18:50] code, but you speak to the AI, which is [18:52] Genie, and you tell it, "Hey, I want to [18:53] do this transformation. I want to join [18:55] these two tables." And it just visually [18:57] shows you what that looks like, and then [18:59] you approve it. [19:00] But under the hood, it's all just Spark, [19:03] open-source Spark, and you can actually [19:05] inspect the code, you can version [19:06] control it, you can make it production [19:08] ready. Uh so, that's super awesome. [19:11] Okay. So, those are three things. [19:12] They're all GA. [19:14] Um [19:14] so you can now get all this data, use [19:16] any of these modalities, all these [19:17] connectors from these different systems, [19:19] and you can get them into, you know, [19:21] well, which format should you pick? [19:23] Should you pick Delta Lake or should you [19:25] pick Apache Iceberg? Okay, so that's a [19:28] big question, and actually I want to [19:30] welcome to stage the original creator of [19:33] Apache Iceberg, Ryan Blue, who is the [19:35] CEO and co-founder of Tabular that we [19:36] acquired last year. So, let's welcome [19:38] him on stage. [19:49] Hey Ryan, how are you? [19:49] >> Good, how are you? [19:50] >> Good. I want to embarrass you with a [19:52] video uh from a bunch of years ago. So, [19:54] this video actually had a very [19:55] interesting title. Uh it said [19:59] why you shouldn't care about Apache [20:00] Iceberg. Let me listen to it. [20:03] >> Um and I I sincerely mean that. Um you [20:05] shouldn't care about Apache Iceberg. [20:09] >> Okay, we shouldn't care about Apache [20:11] Iceberg. Actually, I agree. I think we [20:12] shouldn't also care about Delta Lake, [20:14] but why did you say that? [20:16] >> No one should have to care about [20:18] formats. You you should be able to use [20:20] whatever tools you want with all of your [20:22] data. Uh [20:24] use the right tool for the job, which is [20:26] why we built both formats in the [20:28] beginning. [20:28] >> Yeah, in fact, this is super funny. Last [20:31] a few years ago, you know, I would get [20:33] to customer meetings and they would get [20:34] into the nitty-gritties of how Iceberg [20:36] was implemented versus Delta, and they [20:38] would ask us like, "Yeah, but can you do [20:39] this, you know, deletion vectors this [20:41] way and that way?" And I was like, "This [20:42] is not where the industry should be [20:44] going." Uh so, how are we doing? Where [20:45] are we? What's what's going on with the [20:47] versions and all of that? [20:48] >> Well, we just just did the GA release of [20:51] uh Iceberg V3 support, managed Iceberg [20:54] tables in DBR. Uh that is going really [20:58] well. We're following that up later this [21:00] year with the unified metadata layer [21:03] that is going to be built into Delta 5 [21:05] and Iceberg V4. Um so, we are very close [21:09] to the full unification vision. [21:11] >> So, what's special about V3? [21:13] >> Uh V3 is the unified data layer. So, you [21:16] no longer have to rewrite any data files [21:18] to share them across Delta and Iceberg [21:20] tables. [21:21] >> That's awesome. So, basically, the data [21:22] that's laid out, whether it's Iceberg or [21:24] Delta, looks like identical to big data [21:26] now, right? [21:26] >> Exactly. Yes. [21:27] >> Awesome. And then with V4, just a little [21:29] bit of metadata that's left. And [21:31] hopefully, when do we get that? [21:33] >> Uh we're aiming for later this year. I [21:35] don't want to make too many promises, [21:37] but I think that we should finish the [21:38] spec, uh you know, sometime in maybe Q4. [21:42] >> Awesome. Super excited about this. Great [21:44] job by Ryan and the whole community, [21:46] Iceberg community, Delta community. You [21:48] shouldn't have to care about this. Can [21:49] we stop talking about this now? [21:51] >> Yeah. I got work to do if if we're done. [21:54] >> All right. [21:55] >> Thank you. [22:02] >> All right. So, it doesn't really matter. [22:03] Actually, if you're using Databricks, it [22:04] doesn't matter if it's in Delta or [22:05] Iceberg. We don't care. [22:07] Um it's actually the same now. Thanks to [22:09] V3, it's the same data format that's on [22:11] the disk. Okay, perfect. So, that's [22:13] that. [22:14] So, now we're done with how you can [22:15] actually get your data into this open [22:17] format. It's really seamless. Just one [22:19] way of doing it. It's all Spark-based [22:21] under the hood. [22:22] Um so, let's move on and look at how we [22:24] actually then uh start doing analytics [22:26] on that data. So, the data warehouse [22:28] that we announced about 4-5 years ago, [22:31] uh we call it lakehouse. In fact, our [22:33] tagline was the best data warehouse is a [22:35] lakehouse because it's open, so you [22:37] control the data yourself, and it works [22:39] with machine learning workloads. And we [22:41] made huge progress on our lakehouse in [22:44] the last year since last Data + AI [22:46] Summit. We've added over 110 features [22:48] that [22:50] legacy data warehouses have so that we [22:52] can more easily migrate those over. [22:54] Um we also released a lot of AI [22:56] functions. This is the main thing that [22:58] people want with lakehouse. So, there [23:00] are all these AI functions you can just [23:02] So, you have your data, maybe they're in [23:04] tables. That's the kind of thing you [23:06] work with in data warehousing. But now [23:08] you can just in your SQL queries call [23:11] this SQL AI functions and you can say, [23:14] "Can you do sentiment analysis? Tell me [23:16] for each of these cells, what do people [23:17] think?" Or you can tell, "Hey, for each [23:19] of these cells, can you extract for me [23:21] the company names that you see in it?" [23:23] Or can you do a prediction of what this [23:24] number is going to be? And it can just [23:26] do that at really high scale. So, this [23:27] is the most popular thing that people do [23:29] actually on data house. [23:31] And we've taken Lake Bridge, which is an [23:33] automated converter that lets you [23:35] migrate from existing data warehouses [23:37] that you might have on prem or elsewhere [23:39] in the cloud, and we can do that now [23:40] really, really well. So, very excited [23:42] about that. And actually, you all are [23:44] using this. So, it's we've over doubled [23:46] the consumption of our data warehouse [23:48] just in the last year. So, we're very [23:50] excited about this. Um but what I'm [23:52] really excited about is my co-founder [23:54] Reynold getting on stage and talking [23:56] about the new engine. Okay, so we have a [23:58] new engine. He's going to talk about it. [24:00] It's called Raiden. And what's really [24:02] cool about Raiden is that it's the [24:03] world's fastest engine that we've ever [24:06] seen. We've benchmarked it against [24:07] anything else. We've gotten the latency [24:09] of it down to just tens of milliseconds. [24:12] So, far faster than anything we had done [24:14] before. [24:15] Uh so, it's very, very exciting for us. [24:17] So, we'll have him uh in a couple of [24:19] talks in the keynote today [24:21] talking about this. [24:23] Okay, so that's our data warehouse. [24:24] That's the lake house. [24:26] Um what about transaction processing? [24:28] So, I mentioned in the beginning we [24:30] really love Postgres. We really embraced [24:32] Postgres. We think AI's today really [24:34] prefer Postgres. If you ask any of the [24:36] AI's today, "What kind of database [24:38] should I pick?" they will say open [24:40] source Postgres. Okay? Uh but we're [24:42] excited about what we call Lake Base, [24:44] uh which is a way in which you take [24:46] Postgres and you separate it out so that [24:49] you have all of its storage just sit on [24:52] cheap lakes. [24:54] Okay, so now we're talking Lake Base. [24:55] So, this is OLTP, right? It's like my [24:57] sequel, Oracle, transaction processing. [25:00] And once you have that data sitting on [25:01] the lake, you can now start doing [25:03] amazing things on top. So, what can we [25:05] do on top? Well, we have a little [25:07] animation here. If you look, you can get [25:09] serverless auto scaling. So, you can [25:11] scale up and down Postgres. And this is [25:14] really fast. So, you can do it in less [25:15] than a second. You can scale it even [25:17] down to zero. [25:18] That's really awesome cuz if nobody's [25:19] using it in the middle of the night, you [25:21] don't have to pay for it. You just pay [25:23] for cheap storage that's sitting in the [25:24] lake. Okay? This was not possible [25:26] before. You know, others had specialized [25:28] storage, but this is just sitting in [25:30] open source format on your lake. [25:33] So, why do you care about that? Why [25:34] should your company care about that? [25:35] Because it significantly cut cuts the [25:37] cost of running databases down. So, you [25:40] get significant reduction in TCO by [25:43] doing this kind of auto scaling. So, [25:44] that's a game changer. [25:45] But that's actually not even the coolest [25:47] feature Lake Base Postgres. The best [25:50] thing about Lake Base Postgres is that [25:52] it has this feature that agents really [25:53] love. And that feature is called [25:55] branching. So, it kind of looks like [25:57] that. I don't know, that's that's our [25:58] attempt to animate and visualize that. [26:01] But what that means is that you can take [26:02] a gigantic database, petabytes, and you [26:04] can say, "Hey, can you make a clone of [26:06] this for me?" [26:07] And it will in less than a second say, [26:08] "Done." [26:10] What it's doing is not actually copying [26:11] the whole database. It's just keeping [26:12] track. The data is anyway sitting on the [26:14] lake. It'll just say, "Okay, we have two [26:16] copies." But it's actually just one [26:17] copy. And then when you make a change to [26:19] one, it'll just keep track of like, [26:21] "Okay, this change is in this copy. This [26:23] other change doesn't exist in the first [26:24] copy." So, what's called uh copy on [26:26] right. So, it actually does that for [26:28] you. [26:29] Why does that matter? Well, it matters [26:31] because it actually completely changes [26:32] the way you organize databases inside of [26:34] an enterprise. [26:36] Most of enterprises have thousands of [26:37] thousands of database instances that [26:39] they're running. And we have DBAs that [26:40] are running around trying to manage all [26:41] of these. And the way you do that, the [26:43] reason you have thousands of these and [26:44] you pay for all these thousands of [26:45] copies of the data that are siloed [26:47] inside of the enterprise is because you [26:50] need to have uh user acceptance testing [26:53] database. You need to have production [26:54] database one, production database number [26:56] two. You need to have staging database, [26:58] R&D database. So, you have all these [27:00] copies, okay? But, branching what it [27:02] enables is to just have one instance. [27:04] And then you just have different [27:05] branches of it, and that's just lower [27:08] TCO, easier to manage, and agents love [27:11] that. Cuz agents love to just branch out [27:15] and just experiment with the data, try [27:16] something out, and they want to do it [27:18] quickly. So, that 1-second latency is [27:19] very important for them. They don't want [27:20] to wait 10 minutes on a database to come [27:22] up. And then, finally, if they if [27:25] something goes wrong, they want to be [27:27] able to go back to an earlier snapshot. [27:28] If the data gets corrupted or the agents [27:30] did something by mistake, they should be [27:32] able to go back to an earlier version. [27:33] So, this is really awesome. Um so, [27:36] please try this out. If you're doing any [27:38] development going forward, please use [27:40] LakeBase Postgres. Um in the last in the [27:43] next 12 months, we're going to see more [27:45] software be written than ever in the [27:47] history of mankind. Okay? [27:49] Uh all that software [27:51] are going to write using LLMs and coding [27:53] tools need a database behind the scenes. [27:55] Let's make sure it's just open source [27:57] LakeBase Postgres with low cost. That's [27:59] what these agents love. [28:01] But, today we have another talk that I'm [28:02] really excited about, and I'm not going [28:04] to talk too much about it here, but [28:05] Reynold's going to come back and he's [28:07] going to talk about something we call [28:08] LCap, okay? LakeBase and LakeHouse [28:11] unification. So, it's like a [28:11] breakthrough that actually the industry [28:14] been working on for 40 years. We think [28:16] we finally pulled it off. Uh so, he's [28:17] going to come and talk about that as [28:18] well. [28:19] Okay. So, we talked about LakeFlow to [28:21] get the data in. We talked about [28:22] LakeHouse, which is a data warehouse to [28:24] do analytics. We talked about LakeBase, [28:26] which is transaction processing and uh [28:29] the talks that we're going to have on [28:30] that. So, hopefully we now have the [28:32] choice, the data layer at the bottom [28:34] ready for the AI, okay? So, that it can [28:36] be consumed fast by the agents, and it's [28:39] seamless for them, and they can [28:40] experiment with the data. So, this is [28:41] really important. We got to get that [28:42] data layer right. [28:44] But, as I mentioned, we also need to [28:46] figure out control. Okay? So, we can't [28:48] just unleash the agents on all of the [28:50] data like that. We need to make sure [28:51] that we're controlling what they're [28:53] doing. So, that's why we started the [28:54] Unity Catalog project a bunch of years [28:56] ago. When we started that project, [28:58] what the industry was doing is that they [29:00] were having catalogs for structured [29:02] tables. So, data warehouses were looking [29:04] at structured tables, rows and columns, [29:07] and they were saying, "Okay, well, this [29:08] column is somebody's salary information. [29:10] So, can we hide that?" So, that's what [29:11] this was about. Access control on [29:13] structured data like that. But, we [29:16] called it Unity because we think you [29:17] shouldn't just do control on structured [29:20] data, you should do it on all the data [29:22] assets that you have in an organization, [29:23] any data and AI assets. Okay? So, you [29:26] could have unstructured files, PDFs. You [29:28] might have AI models, uh and 50 other [29:31] asset types that we added over the [29:32] years. We want to be able to do it with [29:34] all of these. Okay? So, that's idea [29:36] number one for Unity Catalog. Number [29:38] two, you don't want to just do access [29:40] control. You can lock everything down [29:42] and you're done. You want to also be [29:44] able to discover and democratize the [29:45] data in the organization. You want to [29:47] also be able to track the lineage of the [29:49] different data sets. Where does this [29:51] data come from and visualize that graph? [29:53] You want to be able to do cost controls [29:55] on everything that's being done cuz [29:56] otherwise the cost just will run out. Um [29:59] you want to be able to look at the data [30:00] quality of everything you have in your [30:02] organization. So, these are all the [30:04] things that are part of Unity Catalog [30:05] and it's extremely important. [30:07] That's why we never actually charge for [30:08] Unity Catalog. Unity Catalog is just [30:10] free and we actually open sourced it a [30:11] few years ago and we're doubling down on [30:12] open sourcing more. In the last month, [30:14] we added more people that are open [30:15] sourcing pieces of it. Uh so, it's [30:16] moving very very fast. We're excited [30:18] about that. [30:19] And also, we announced that you want to [30:21] be able to share data. So, this was [30:23] really important for us. Um [30:25] the rest of the industry, they were [30:26] doing this kind of locked-in sharing [30:28] where they were saying, "Buy my software [30:30] and then if you want to share it with [30:31] someone else, they should buy also my [30:33] software and they can share data sets [30:34] between it." That's why we open sourced [30:36] Delta Sharing a bunch of years ago as [30:37] part of the Delta Lake project. And now [30:40] you can take any data assets and you can [30:41] share it with anyone in the ecosystem. [30:42] Exactly have great growth, lots of [30:44] partners in that. But what we're very [30:46] excited about this year is that we [30:47] actually open sourcing a new project [30:48] that's a super set of Delta Sharing. We [30:51] call it Open Sharing. [30:53] And what this enables you to do is share [30:55] not just with Delta, but also with [30:57] Iceberg cuz they're now the same format. [30:59] But also not just share data assets, but [31:01] also AI assets. Okay? So now you can [31:03] share your agents, your skills, and your [31:06] models and you can do that also on [31:08] premise. Okay? So we're very excited [31:10] about that. This is very cool. [31:14] >> [applause] [31:16] [cheering] [31:17] >> Okay. So we got all these different [31:19] things you can do in Unity Catalog. We [31:20] have Open Sharing. [31:21] But in the last year what has happened [31:23] is that uh governance, which always has [31:26] been AI governance has always been part [31:28] of Unity Catalog, is becoming very [31:30] complicated in most organizations. Okay? [31:32] So we're seeing that organizations now [31:35] have uh lots of agents. You're all [31:37] building agents, maybe using Llama [31:39] Index, maybe using LangChain. Um the [31:41] SaaS providers are now each providing [31:44] you an agent that they want you to use. [31:45] They have agent builders that they want [31:47] you to use. They have their own MCP [31:48] servers that the SaaS vendors are [31:50] saying, "Please use our MCP servers. [31:51] Here are the skills you can use with [31:52] those." [31:53] Your organizations, your developers are [31:55] themselves developing MCP servers. [31:57] They're themselves developing skills [31:59] files. They're [32:00] um [32:01] All these models that are being [32:02] developed now by different tier vendors, [32:04] there's many different versions of them. [32:05] They all have different capabilities, [32:07] you know, they exist, they don't exist. [32:09] Um there's coding tools out there, um [32:11] vibe coding things, and then now we're [32:13] starting to see more and more co-work [32:14] like agents in the organization. So it's [32:17] kind of a quagmire. It's a complete [32:19] mess. Okay? [32:20] >> Yeah. [32:21] >> So what is the problem with this? [32:24] So problem number one is that actually [32:25] cost is running away. I think this is [32:26] the biggest problem right now. Uh anyone [32:28] we talk to, they're really worried. The [32:30] skies of this are just skyrocketing. [32:32] Um and no one has visibility. Is it [32:35] someone's agents? Which model is it? [32:37] What were they doing? So, all this token [32:39] maxing is happening and there's no way [32:41] to actually set limits on everything [32:42] holistically or rate limit it. So, [32:44] that's problem number one. Problem [32:46] number two is that there's no way to [32:47] control what the agents are doing. How [32:49] do we make sure that the agents actually [32:51] have access to the appropriate data, [32:52] that we can audit them, that we have [32:55] guardrails [32:56] for their inputs and outputs, and how do [32:58] we manage their identities? Who are they [33:00] be acting on behalf of? [33:02] How do we track this? And then finally, [33:04] and this is the most important one as I [33:05] said earlier, how do we do this with [33:06] choice? Without locking. Because the [33:09] state of the art models now have a [33:10] lifespan of 1 month. November last year [33:13] we were super excited about Gemini came [33:15] out from Google and it was like the best [33:17] model and we all were switching over to [33:18] it and we said, "Okay, that's that's it. [33:20] This is the best model ever." Then [33:22] February time frame, January February [33:23] time frame it was like clearly Opus is [33:25] the best ever. And then OpenAI came out [33:27] with GPT-5.5 and it was the greatest [33:29] model ever. And then last week uh we had [33:33] Mythos or Fable and it was out and we [33:34] were all going to switch to it but now [33:35] they canceled it and then you know, it's [33:37] back and forth. So, uh how do we make [33:39] sure that we can actually have [33:41] flexibility uh so that we can pick any [33:43] of them and that uh we will have choice [33:46] here, okay? Uh so, those are three [33:48] problems that are really important. [33:49] That's why we're super excited to [33:51] announce as part of Unity Catalog uh we [33:53] are breaking out one place where you can [33:55] manage and control all of your agents, [33:59] all of your AI spend. We call it Unity [34:00] AI Gateway. [34:02] Okay? [34:03] So, excited about this? [34:05] >> [applause] [34:09] >> So, what is Unity AI Gateway? It's a [34:10] single pane of glass [34:12] and there you can actually have your one [34:15] entry point for all of your agents, for [34:18] all of your AI, for everything that [34:20] happens, any of the harnesses that [34:22] you're using can go through this. Again, [34:24] uh this is part of Unity Catalog which [34:26] we open sourced. We don't charge for [34:28] this. And also the AI Gateway is part of [34:30] ML Flow. So, it's also open sourced. Um [34:33] but one of the most important things [34:34] that we do that not maybe everyone in [34:36] the room knows is that we will actually [34:37] provide you capacity uh for the frontier [34:40] models. So, let's say you committed to [34:42] pay $100,000 to Databricks. You might [34:45] use Lake Flow for that to consume [34:47] things, you might use our Lakehouse, but [34:48] you can also just use tokens for that [34:51] $100,000 directly from us on OpenAI's [34:54] models or Anthropic's models or Gemini. [34:57] And you can do this on any cloud, [34:58] whether you're on Azure, GCP or AWS, you [35:00] can just use this. So, this is available [35:02] on the platform. Uh so, we actually [35:03] offer that. [35:05] Two, you can actually get perfect uh [35:07] observability on all of the spend inside [35:09] the organization with dashboards that [35:12] give you the observability of how that [35:13] spend is happening, and you can set [35:15] limits. You can set budgets. You can set [35:17] budgets on different groups, subgroups, [35:20] all the way down to the individual [35:21] level. And when someone exhausts that [35:23] budget, uh you will get an alert. Uh or [35:26] you can actually just stop it and rate [35:28] limit it so that you uh you can make [35:30] sure that nobody's spending too much or [35:31] that they every day get an email saying [35:33] how much they've spent. So, this is [35:34] super important. And then finally, we [35:36] can enforce safety, compliance, [35:39] auditing, and identity management of all [35:42] your AI. Okay, this includes harnesses, [35:44] this includes models and agents. So, [35:47] very excited about this. So, that's uh [35:49] what Unity AI Gateway is, and now you [35:51] can manage all these asset types, MCP [35:53] tools, models, agent skills that you [35:56] have in the organization. And I just [35:58] want to emphasize, this is not just MCP [36:01] or the tools that you have in [36:01] Databricks. [36:03] These are any MCPs that you have in the [36:04] organization by any vendor. You register [36:07] them in Unity AI Gateway, and you will [36:09] track all the usage, and you will see [36:11] how it's going. And in fact, it has a [36:12] really awesome capability where you just [36:14] authenticate once for the MCPs, and [36:16] it'll do it for all of them, so you [36:17] don't have to authenticate again and [36:18] again and again. So, this is really [36:20] awesome. [36:21] Check this out. [36:22] Okay. [36:23] So, now we have uh Uh, also control, and [36:26] we have cost. So, now we've dealt with [36:30] how do we get your data ready? So, [36:31] hopefully we have a data ready in an [36:33] open format that's fast for the agents. [36:36] Um, we figured out how to do control, so [36:38] we can do governance, and how do we do [36:39] cost management with Unity AI Gateway. [36:42] Okay. So, that brings us uh, to the [36:44] context layer. So, as I said, [36:47] AIs are plenty smart. AI does not have [36:49] an intelligence problem, it has a [36:50] context problem. So, how do we actually [36:51] get that context into the AI? So, this [36:54] is the most important thing. [36:56] Um, so what do I mean here? So, let me [36:57] give you an example of how it works [36:59] today. [37:00] Today, you're using these [37:02] agents. You might be using a coding [37:03] agent. And if you look at the best [37:05] agents out there, they kind of work like [37:07] this. There's a big graph of all the [37:09] documents and everything that everyone [37:11] is doing inside of your organization, [37:13] and what they're doing is there in this [37:14] red dot you can see they go when you ask [37:16] a question. For instance, say, "How is [37:17] my product revenue [37:19] um, last 2 weeks in Europe?" It goes and [37:22] it looks at one document. And then it [37:24] reads that document, and it says, "Okay, [37:25] is there anything interesting?" And [37:26] then, "Oh, there's a link." And then it [37:27] goes, opens up an MCP server to another [37:29] document. It reads that document. And [37:31] it's sort of for looping live through [37:34] your corpus to get the data. Okay? [37:37] But there's a big problem with this [37:38] approach that all the agents today are [37:40] using, all the frontier ones. Uh, [37:42] problem number one is that it's very [37:44] time consuming. This live loop of going [37:46] searching for your data live, [37:48] uh, it's just going to take very long [37:49] time. Okay? And we all see it. Like you [37:51] ask a question, even a simple question, [37:53] and it takes it to 10 minutes, 15 [37:54] minutes to just come back with an [37:55] answer. Because it's live going through [37:57] the MCP servers, it doesn't know where [38:00] to search. It's just trying live to do [38:02] this. It doesn't have the context. [38:05] Number two, the costs are super high. [38:07] The joke is you tell it, "Please rename [38:09] this file for me." And it takes 10 [38:11] minutes, and it charges you five bucks. [38:13] Okay? So, that's kind of how it works [38:15] because it's again, it's going through [38:16] and it's doing the search, and it's [38:17] using up a lot of tokens to construct [38:19] all of this. But these two are not even [38:21] the biggest problem. This This gives you [38:22] really bad user experience and it's [38:23] expensive. The biggest problem is that [38:25] the quality suffers. [38:26] The reason that is is that all of the [38:28] documents and all the data assets in [38:29] your organization is gigantic, okay? So, [38:32] you're really just doing a small random [38:33] walk of a small subset to get to the [38:35] answer. It's remarkable the answers it [38:37] gives you. It's game-changing. We're all [38:39] excited about it and we all were using a [38:40] bunch of it and the costs were going up. [38:42] Uh but, actually if you look at the [38:43] accuracy, it's uh quite problematic. [38:46] Okay? So, [38:47] um how do we solve this? Uh so, we are [38:50] very excited to announce, and this is [38:52] one of the most important things I'm [38:53] going to announce, Genie Ontology, okay? [38:57] >> [applause] [39:01] >> So, what's Genie Ontology? So, let me [39:03] explain. So, Genie Ontology what it does [39:06] is that it actually connects with all [39:08] the data that you have in an [39:09] organization and in the background, so [39:12] this is not when you're asking a [39:13] question, this is happening in the [39:14] background, uh it's constructing a graph [39:17] of the most important knowledge that you [39:19] have inside of your organization and [39:21] it's making it ready so that we can give [39:23] that context to any of the agents, okay? [39:26] And one thing that's really important is [39:27] that it's not just everything that you [39:28] have in the lake house and everything [39:29] you have in Databricks. [39:31] It also connects to your Google Drive, [39:33] it connects to your uh SharePoint, it [39:35] connects to your email, Google Calendar, [39:38] anything that you set up and it can [39:40] actually branch out and access all of [39:42] these things that you have there. [39:43] And the research team developed an [39:45] algorithm uh called OntoRank, okay? I [39:48] know it's a bad name, but you know, this [39:50] is the research team came up with it. [39:52] And it's like ontology rank. Um [39:55] but, what what is OntoRank? So, what [39:57] OntoRank does, it's it's kind of like [39:59] PageRank for those of you who remember. [40:00] So, PageRank was how Google [40:02] uh 25 years ago or 30 years ago figured [40:04] out how to take the whole web and create [40:07] an index for what are the most important [40:10] sites. Before Google, AltaVista, Yahoo, [40:13] these sites, [40:14] um they were just looking at times is [40:15] this word mentioned on the site. It led [40:17] to these SEO uh search engine [40:20] optimization where people adding the [40:21] same word thousand times on a web page [40:23] to get higher ranking. But PageRank it [40:25] could just figure out from the graph [40:27] itself what's important, okay? So, [40:29] OntoRank does a similar thing on all of [40:32] the assets that you have in an [40:33] organization, but it's not PageRank. [40:35] This problem is harder because it's not [40:37] just web pages with links. [40:40] It's all the assets that you have, but [40:41] they're actually different asset types. [40:43] So, it needs to identify and you know, [40:45] source code is different from Google [40:46] Doc. [40:48] Also, here we have users. And the users [40:50] inside of your organization are part of [40:51] an org chart and it actually matters how [40:53] often they access things. So, we have [40:54] this information as well. So, we take [40:56] all of these different asset types, all [40:58] the people in the organization and we [41:00] construct this ontology graph that then [41:02] helps us actually [41:04] do amazing things with this Ford agent. [41:06] So, this is the context that we're going [41:07] to feed. [41:09] Now, you might say, "Hey, I already put [41:10] together context. I already built [41:12] semantic layers at my company. I already [41:14] have a business glossary that I built [41:15] over the years." [41:17] Genie can take that from any of the [41:18] sources. So, you can actually bring your [41:20] own semantics as well. So, as part of [41:23] Unity Catalog, we have something called [41:24] Unity Catalog semantics where you can [41:26] bring from our partners, whether it's [41:28] from Atlan or AtScale or from [41:31] existing BI tools that have semantics or [41:33] any business glossary that you already [41:35] have. You can have humans curate that. [41:37] You'll happily take that and it'll help [41:39] it get smarter. So, depending on [41:41] whichever camp you're in, you think it [41:42] should be completely autonomic and [41:44] happen by itself, Genie ontology does [41:46] that. If you want to have lots of manual [41:48] curation, you can do that with model [41:50] semantics. They both work well together. [41:53] Then this These are fed into our agents. [41:55] So, we'll talk about the agents that we [41:56] have and we'll give you demos, but [41:58] there's three categories of agents that [41:59] are important. I'll talk about them. [42:01] Genie one, Genie agents, and Genie code. [42:04] And they all are fed this context which [42:06] makes them faster, it makes them much [42:10] more cost-effective, and it makes them [42:13] much higher quality. Okay, so you can [42:14] much faster get the quality answers you [42:16] want without paying too much. Okay, so [42:18] that's what this is. Um, if you're using [42:20] other agents, you want to use other [42:21] agents, they can use Genie as an MCP [42:24] server, and they can just connect to it [42:26] that agent Genie, and they can actually [42:28] get the semantics from Genie. So, you [42:30] can also do that this way. You can bring [42:32] any client, any agent you want, you can [42:34] connect to this ontology that you have [42:35] there, [42:36] uh, so that you can get that semantics [42:38] in the context into any existing AI that [42:40] you have. [42:41] Um, so this is what it looks like in the [42:43] product. You can actually click on the [42:44] graph, and you can sort of explore it. [42:46] In this particular graph that we're [42:47] exploring here, this is just one [42:49] instance of Databricks for ourselves. [42:51] This is our own Databricks that we have [42:52] for Databricks employees. It has over [42:54] 4.5 million ontology data snippets that [42:58] it has collected and constructed. So, [43:00] obviously nothing you could do live. [43:01] Like, it would be infeasible for any of [43:04] these agents to live go one by one and [43:06] look through 4.5 million ontology [43:09] snippets, uh, but it's done that, and [43:10] it's constructed this so that you can [43:12] actually have this. Okay, so, um, that's [43:15] super awesome. [43:16] So, this is now fed to our agents. Uh, [43:19] the most important agent that I'm [43:20] excited about is Genie 1. Okay? So, let [43:23] me tell you about Genie 1. [43:25] Some of you might already be using Genie [43:26] at Databricks for the last, uh, 3 years. [43:29] Genie has been part of the platform. [43:30] This is not that Genie. This is Genie 1. [43:33] So, it's the one place you go to access [43:35] all of the data assets in your [43:37] organization. Okay? So, Genie that we [43:39] had previously after using it still [43:41] there, um, [43:42] but that's confined to a certain domain, [43:45] and you have, say, marketing people [43:46] asking just marketing questions in a [43:47] Genie, [43:48] uh, space or room. Genie 1 is just one [43:52] place any of your business users in your [43:53] organization can log into. So, you [43:56] should just enable automatic identity [43:58] management for all of the employees. If [44:00] you have 100,000 employees, they then [44:01] log in, they just get Genie 1. They [44:03] don't get the data warehouses, they [44:04] don't get all the rest of the Databricks [44:06] infrastructure. You know, they're not [44:08] going to be able to go and spend spin up [44:10] data warehouses and spend a lot of [44:11] money. They're just getting a simple [44:12] interface that looks like this. [44:14] And they just ask their questions there. [44:16] So, it looks like any other agent that [44:18] you might have. [44:19] Uh but there's some differences here. [44:20] First of all, it now can get data from [44:22] any of the sources that you have at the [44:23] bottom of the screen. So, whether it's [44:25] Google Drive or SharePoint or whatever [44:27] it is. It of course has all the goodies [44:29] that other agents have like you can have [44:30] skills, you can schedule task and do [44:32] co-work [44:34] uh activities for you. Uh but the key [44:37] thing is it has access to this Genie [44:38] ontology. Okay? And that changes [44:41] everything. [44:42] So, very excited about Genie 1. Uh [44:43] please enable your organization to have [44:45] it and also use it. Uh my favorite way [44:47] of using it is on the phone. Most of us [44:48] use it on the phone on our iPhone or on [44:50] uh [44:50] Android. Uh and you can see it's visual, [44:53] you see all the graphs, and it's [44:54] integrated with our dashboards, AI BI. [44:57] So, it's basically like a BI tool or a [44:58] dashboard. [45:00] Okay? So, that's Genie 1. [45:01] Uh we're also announcing uh Genie [45:03] agents. So, what's Genie agents? [45:05] You can take any conversation that you [45:07] had with Genie 1 and you can say, "Hey, [45:10] turn it into an agent." You can take [45:12] that agent, you can now make it [45:13] available to the whole company. So, for [45:14] instance, you can say, "Hey, here is all [45:16] my HR documents." And create that agent, [45:19] have a conversation, make sure that it [45:20] can answer questions about your HR [45:21] documents. Now you can make that [45:23] available to the whole organization. [45:24] Anyone can ask any HR questions they [45:25] have from that agent. You can put it in [45:27] Slack, you can put it in Teams, any [45:29] modality they want. Um but it also can [45:32] do more. It can actually completely do [45:34] uh autonomous work for you. It can go uh [45:37] every day fetch information from say [45:39] Salesforce, prepare it, put it in [45:41] Workday, uh send you a report, uh work [45:44] with other agents. Uh so, very excited [45:46] about Genie agents. [45:48] And then we have Genie code. [45:49] Genie code is special because um Genie [45:52] code is similar to other coding agents [45:54] that are there, but it's really good at [45:55] two things. Okay? It's really good at [45:58] data engineering. So, you know, 13 years [46:01] of data engineering on the platform, [46:02] we've really made it really good at [46:04] writing pipelines for you, writing code [46:06] for you, doing data engineering. So, [46:08] that's super awesome. Uh it's also [46:10] really, really good at machine learning [46:12] and data science, okay? So, it can go [46:14] and build you a machine learning model. [46:15] You can train it on GPUs. It can test [46:18] that the accuracy of those machine [46:19] learning models is really high, and it [46:21] can really democratize access to machine [46:23] learning inside of your organization. [46:24] So, that's really, really important. Uh [46:26] so, these are two skills that it's, you [46:28] know, uniquely capable of on the [46:30] platform. So, check out Genie code. [46:32] And uh we also are launching something [46:35] I'm really excited about. I think this [46:36] agent actually is going to be the big [46:38] biggest transform transformative agent [46:40] for most of your organizations. We call [46:42] it Genie Zero Ops. So, Zero Ops, as the [46:45] name implies, what it does for you is it [46:48] runs in the background. [46:49] It looks at all your data pipelines, or [46:51] if you have data science machine [46:52] learning models, and it sees that, okay, [46:54] this pipeline is down at 2:00 a.m. It [46:56] will then go investigate, why is this [46:58] pipeline down? Oh, the pipeline is down [47:00] because it's there's an error. It's [47:01] saying that, you know, this data I'm I'm [47:03] not familiar with it. There seems to be [47:05] a new column in this data. It'll go see, [47:07] okay, what's the name of that column? [47:08] Can I add that column? It'll actually [47:09] experiment with writing the code in a [47:11] separate environment without affecting [47:13] your production pipeline. And once it's [47:14] ready, it'll prepare it for you, and [47:16] you'll send you a notification on your [47:18] phone, and you can see and you can [47:19] review it. And if it looks good, you can [47:21] just click accept, and it'll fix your [47:23] pipeline, okay? So, there's still a [47:24] human in the loop. Of course, you can [47:26] just click do it automatically, but I [47:28] would not recommend that. Yes, people [47:29] are excited. [47:31] >> [applause] [47:33] >> Yeah. [47:34] I'm personally really excited about [47:35] this. No one wants to wake up at 2:00 [47:36] a.m. You know, we have all these folks [47:37] that are on call, and you know, this [47:39] pipeline broke. Why did it break? Oh, [47:41] you have to go investigate, and you have [47:42] to spend, you know, 2 hours in the [47:43] middle of the morning, and everybody's [47:44] yelling. [47:45] Uh this is the world of data [47:46] engineering. Uh so, with Zero Ops, we're [47:48] really excited that you can actually [47:49] make this much simpler. [47:51] And then finally, we'll talk about apps, [47:52] building your own apps to democratize [47:54] access to data and AI inside of your [47:55] organization. [47:57] Uh but now we make that really simple to [47:58] vibe code your own apps. You can of [48:00] course use our partners. You know, you [48:01] can use lovable, replit, for sale, any [48:03] of these. But we also now give you on [48:05] Genie Genie app builder, which is which [48:07] can do that for you. [48:09] Okay, that's Genie. [48:11] And key thing here is Genie ontology [48:13] that we automatically construct for you. [48:14] That's really the secret sauce. [48:16] But the co-work and accessing the data [48:18] outside of the lake house is also really [48:20] important. A lot of people think of data [48:21] bricks as, okay, this is the data I have [48:22] in my lake house. No, this is any of the [48:25] data you have in your organization. [48:27] But what about developers who want to do [48:29] low-level coding? They want to do coding [48:31] agents. Don't want to write software. [48:33] You know, they want to use different [48:34] harnesses. Maybe they're using cloud [48:35] code. Maybe they're using codex. Maybe [48:37] they're using open code or pie or [48:38] something else. So, that's where agent [48:40] bricks comes in. So, this is for [48:42] developers who are doing development [48:43] work. And we've expanded that developer [48:46] platform so that we now have sandboxes [48:49] so that you can run your agents in [48:50] really fast sandboxes that come up and [48:52] isolate them. We have agent memory. [48:54] And we announced this weekend uh [48:58] meta harness called omnigent. So, Matei [49:01] announced that and he'll be giving a [49:02] talk on this and explain what that is [49:04] and how that works. But it's a harness [49:06] of harnesses. So, it can leverage all [49:08] the existing harnesses and it can [49:09] actually have them play against each [49:11] other. It can have cloud code fight [49:13] codex and actually get a much better [49:15] result than either of them. So, he will [49:17] give a talk on that as well today. So, [49:18] very excited about that. [49:20] Okay. So, [49:22] that's what it looks like. So, hopefully [49:24] I've shown you how we can now take all [49:26] the data that you have in your [49:27] organization. Okay, this is the first [49:28] half of my talk. How do we take the [49:30] data, make it ready for AI, [49:33] how do we actually make sure that we [49:34] have cost and governance control and [49:37] security figured out, and then how do we [49:39] get the Genie ontology, this context, [49:41] into the agents. [49:43] So, that's great. [49:45] But I think actually something more [49:46] profound is happening in the industry. [49:49] Something is happening with software. [49:50] Okay, and people are talking about it. [49:52] So, what is that? So, what is the [49:54] classic software as a service SaaS stack [49:57] look like over the last two, three [49:59] decades? So, it kind of looked like [50:01] this. [50:02] You had a system of record, which was a [50:03] database. [50:05] That's where you had this most important [50:07] golden records about your HR data or [50:09] your CRM data, whatever it was. And you [50:12] had some custom workflows [50:14] inside of that. [50:15] Somewhat rigid, but you could customize [50:17] them if you brought in [50:18] contractors that would augment that for [50:20] you. It would take a little while. And [50:21] there was a UI. It was customizable. [50:24] No one really loved those UIs, but most [50:26] of us learned them. And then once we [50:27] learned them, we didn't want to switch [50:28] to a new one. [50:30] And we had one of these for each of the [50:31] domains inside of the organization. So, [50:33] you have one for sales. You have one for [50:35] ops. You have one for finance. So, [50:38] there's one of these for each. [50:40] But what's happened in the last 2 years [50:41] is that each of these vendors are now [50:42] providing you an agent of their own. So, [50:44] they're saying, "Hey, we have an agent [50:46] for this. Here's Use my agent to access [50:48] my system of record data here. You know, [50:50] we're headless. You can just access it [50:52] with an agent." [50:53] But the problem is, when you go to these [50:54] agents and you start asking questions, [50:56] you never have just questions about the [50:57] data in that system of record. Your [50:59] question spans things. And you need to [51:01] access data that's elsewhere. So, it [51:03] needs to go and access systems that are [51:04] elsewhere, okay? And this gets quickly [51:06] very complicated. And most of these [51:08] vendors don't want to play well together [51:09] anyway cuz they're competing with each [51:10] other. So, it's sort of very messy [51:12] inside of the enterprise right now. And [51:14] it's very confusing how do you make that [51:15] actually work and go across these [51:16] different systems. So, the big question [51:18] is, [51:19] what does the future of software stack [51:21] look like, you know, or the SaaS [51:23] software or the apps of the future? [51:26] What do they look like? What are they [51:27] going to look like, okay? And we don't [51:28] know, but we think it should look [51:30] something like this, right? You have all [51:32] your agents. [51:34] And they should just access [51:36] agent system of record or agentic system [51:39] of record. They should just be there and [51:40] you should be able to access that [51:42] seamlessly. It should be very fast. You [51:44] shouldn't have to go through lots of [51:45] other agents. You shouldn't have to [51:47] try to go to a production database. [51:48] That's too risky. You should just be [51:49] able to have it at your fingertips and [51:51] let these agents do that really fast, [51:53] quickly, cost-efficiently, securely. So, [51:57] what would the agent system of record [51:59] look like in the future? What do we [52:00] think that's going to look like? Well, [52:02] we think it probably looks something [52:03] like this. [52:04] You have all your data [52:06] and you can access it all in one place [52:08] in an open format. Okay, you see where [52:10] this is headed? [52:11] Okay. [52:12] So, I can just click through the next [52:13] few slides. [52:14] On top of that, you want to have unified [52:15] governance, right? So, you want to be [52:17] able to have control. You want to be [52:18] able to do cost management on top of [52:20] that. And of course, you don't want [52:22] these agents to go and go one doc at the [52:25] time and spin up tokens and not know [52:27] what they're doing. You want them to [52:28] have the enterprise context so they can [52:30] quickly do that. And the context should [52:31] span all of these different [52:32] organizations. Okay? So, we think that [52:35] the new system of record is actually the [52:37] data and AI platform that [52:40] we've been talking about all along. We [52:41] think that the world will look more like [52:42] this, whether it's Databricks or someone [52:44] else, but this is going to be the [52:45] structure of the agentic system of [52:47] record of the future. That way, you can [52:49] get the choice. You get choice of [52:50] running this on any cloud, any SaaS [52:52] vendor, any of the AI models. You can do [52:55] the cost controls, you can do the [52:56] governance control. And of course, you [52:57] should have the context for this. So, [52:59] that's what we think the future looks [53:00] like. So, let's augment the stack with [53:02] that as well. So, this is what I already [53:04] presented to you, those three layers. [53:06] But now, on top of that, we also get [53:07] agentic apps. [53:09] So, Databricks apps is very exciting. [53:12] We'll have a whole talk dedicated to [53:13] this. Databricks apps is how you [53:15] democratize access to data and AI inside [53:17] of organization to hundreds of thousands [53:18] of people. And it could be custom [53:21] apps that you've built. Maybe you vibe [53:22] coded these. [53:24] There's a huge proliferation of this. [53:25] This is the fastest growing actually [53:27] product for us. [53:28] So, that's what apps look like. And we [53:30] actually now introduce a marketplace [53:32] where you can actually get these apps. [53:34] So, anyone can go and access these apps. [53:36] And one of the things that we're also [53:36] excited about that we're announcing is [53:38] that you can actually transact and pay [53:39] for these apps so that you can buy them [53:41] from those vendors that are actually [53:42] offering those. Of course, you can have [53:44] your own, your company can build your [53:46] own, uh but all of these that also we've [53:48] listed here exist as apps that you can [53:50] actually get access to [53:51] uh or work with. So, that's a very [53:52] exciting development. Okay. So, that's [53:55] what that looked like. [53:56] Um but there's a few apps that we're [53:58] really excited about and we want to [53:59] build ourselves that we think that we uh [54:02] uniquely uh have a point of view in [54:04] where we can really add value. Where we [54:06] think it's really data intensive and [54:07] it's kind of our really forte. So, the [54:10] first one is security. And we announced [54:13] 2 months ago that we're launching a SIM. [54:15] Uh it's called Lake Watch. It's an [54:17] agentic SIM. Uh we call it the security [54:19] lakehouse. It looks kind of like this. [54:21] Let me explain what this is. It's built [54:23] on the agent system of record that I [54:25] mentioned earlier. So, you collect all [54:27] of your security data in a lakehouse. We [54:30] have to do it that way. The current [54:32] approach of taking data and injecting it [54:34] into these proprietary SIMs uh is super [54:38] expensive. So, people can't afford and [54:40] the data is exploding. So, you don't [54:42] want to actually ingest all these [54:43] expensive data into these classic legacy [54:46] SIMs. So, a lot of people what they're [54:49] doing is they're filtering out the data [54:50] before to make sure that, you know, only [54:52] the most important data is in there. [54:54] Well, this is not going to work cuz [54:55] we're getting now massive attacks from [54:57] automated agents. In the last 6 months, [54:59] the best hacker in the world is agentic [55:01] system called Expo. You can look it up [55:03] on the HackerOne [55:04] um top list. Uh people are finding [55:07] exploits using things like Mythos and [55:08] even before Mythos and actually after [55:10] Mythos, we're going to see lots and lots [55:11] of exploits. Uh so, we need all of the [55:13] data. It needs to be very cheap. It [55:15] can't be skyrocketing cost and we should [55:17] just store it all in any means we have [55:19] in an open secure lakehouse. And then on [55:22] top of that, we've built agents in Lake [55:23] Watch. [55:24] Those agents, what do they do? They [55:27] automatically make sure that they create [55:29] detections. They when you wake up and [55:32] there's like 100 alerts of what's going [55:34] on with your data, they go and triage it [55:36] for you automatically, Very similar to [55:37] the zero ops, but this is really for the [55:39] sock. Sock is security operation center. [55:42] So, the sock analysts uh wake up and can [55:45] do that. [55:46] Um and then finally, it can do threat [55:48] hunting for you. It can actually find [55:50] out that there's a new zero-day attack [55:51] and it'll add the pipeline to it and [55:53] automatically do all of that work. So, [55:54] we're very excited about Lake Watch. Uh [55:56] so, you should check that out. And we [55:58] are very excited to announce a piece of [56:00] news uh which is that Databricks has [56:03] agreed to acquire Panther Labs. [56:05] So, yes. [56:08] >> [applause] [56:11] >> So, Panther is a really amazing um [56:14] ahead of its time [56:16] SIM that based on Python, okay? So, what [56:19] Jack Naglieri and team that we're very [56:21] excited to welcome to Databricks and [56:23] you'll actually see him on stage here [56:24] tomorrow. [56:25] Um [56:26] they really bet on everything should be [56:28] Pythonic. So, they were way ahead of [56:29] their time. Actually, I remember meeting [56:31] him 5 years ago and he came from Airbnb. [56:33] He was a sock analyst himself and he [56:34] said that you know, we can't have these [56:36] complicated drag-and-drop UIs. It needs [56:38] to be Python-based. Which turns out to [56:40] be really prescient because today [56:42] um [56:43] all the AIs want to speak Python. [56:46] So, that's really awesome and he has [56:47] hundreds of connectors that they've [56:48] already built that we're all adding. Uh [56:49] so, now we have all these connectors and [56:51] we have all these Pythonic detections. [56:53] Uh so, very excited about this and they [56:54] have really marquee customers. So, [56:56] Anthropic is using them. Uh Coinbase is [56:59] using them. Plaid is using them. So, [57:01] we're very excited to welcome that to [57:02] the family. [57:03] Um so, that's the agenda for today. [57:06] Okay. There's one more space that we [57:08] want to we're excited to get into, which [57:10] is the marketing stack. [57:12] So, [57:13] um what is that? So, we're very excited [57:15] to announce what we call customer lake, [57:18] okay? [57:19] Yeah, so what's [57:21] >> [applause] [57:24] >> Okay. So, what's customer lake? So, it's [57:26] an agentic CDP, customer data platform [57:28] or customer 360 if you will, [57:30] uh built on the lakehouse. Um it looks [57:32] like this. [57:33] And so, of course, you can get now all [57:36] of your data into a lakehouse. [57:38] Um, but it's unique. So, it has in- two [57:41] agents that are really special. [57:43] One agent, which is a profile agent, [57:45] what it can do is it can automatically [57:46] do all of the identity dedupe that [57:49] these, uh, [57:50] frameworks in the past have done, but it [57:52] can do that, uh, using LLMs. So, it's [57:54] really accurate. It also has a campaign [57:56] agent, which implements what we call a [57:58] new concept called infinity campaigns. [58:01] And infinity campaigns basically let you [58:03] tell your personalize every interaction [58:07] that you have. So, in the past, what [58:09] people would do is they would say, "The [58:10] whole world, let's split it up into 100 [58:12] different audiences, and then let's [58:13] market to each of the 100 audiences." [58:15] But now, we can really personalize with [58:17] really cheap, small, uh, distilled LLMs [58:20] one-to-one, uh, continuous campaigns. [58:23] So, we're very excited about customer [58:24] lake as well. And we've developed this, [58:27] uh, in partnership, uh, with all the [58:29] vendors that you see down here. And you [58:31] can see at the top, uh, the logos of our [58:33] development partners, uh, that are [58:34] helping us actually develop this. So, [58:35] very excited about this. There'll be a [58:36] talk by Tasso on this as well. Okay. So, [58:39] that brings me to the end of my keynote. [58:42] Uh, we have an exciting program. This is [58:44] what I kind of showed you. So, let's [58:46] simplify this a little bit and zoom out [58:48] what I've told you today. [58:50] So, basically said that we have this [58:52] data, we have all these processes. If we [58:54] can just capture all of that and make it [58:55] AI-ready, we can get the context and we [58:57] can give it to the AI, um, then we can [58:59] have huge impact in the organization. [59:01] So, really, it's about how do we get [59:03] choice [59:04] so that you're not locked in. How do we [59:05] get governance? How do we get cost [59:07] control? And how do we get context? [59:09] Okay. So, the first one is really about [59:12] any data, any model, and any cloud [59:14] without lock-in. The second one is [59:16] really one place where you can do all of [59:17] your governance and all of your cost [59:18] controls. And the final one is about how [59:20] we get enterprise context to your [59:23] agents. So, Well, concludes my talk. I [59:26] hope I've shown you that uh, Databricks [59:28] Data and AI platform will help you do [59:29] that. Thank you so much. [59:31] >> [applause] [60:30] [music] [60:48] >> Welcome to the stage Databricks senior [60:51] director of product management Ken Wong. [61:01] >> All right. Good morning. [61:04] So today I get to introduce you to Genie [61:07] 1. Genie 1 is an AI co-worker that can [61:11] connect to all of your data and all of [61:13] your apps so that everyone in your [61:15] organization can get insights and even [61:18] automate action. You can even create on [61:20] autonomous agents just by talking to [61:22] Genie. [61:24] But, I know what you're all thinking. [61:26] In 2026, [61:27] why do we need another one of these AI [61:30] agents? [61:32] Well, the answer's actually very simple. [61:34] The ones that you have today aren't very [61:36] good with enterprise data. They either [61:38] can't connect and reason about data at [61:41] all, or they're just not accurate enough [61:43] for you to actually trust them to make [61:46] real decisions, or let alone automating [61:49] your work. [61:50] So, let me show you what I mean. So, [61:51] yesterday we had our product advisory [61:54] board meeting at Databricks. Um shout [61:56] out to the PAB, and I wanted to [61:59] prepare myself for it by getting Genie 1 [62:02] to help me with a comprehensive profile [62:04] of all the customers we have, what's [62:06] going on in their account. So, this [62:08] required Genie 1 to connect to all the [62:10] systems that we have, including my my [62:12] personal email and Slack [62:15] and my calendar to figure out who's in [62:17] the PAB and what we've been talking to [62:18] them about. [62:20] required Genie 1 to connect to [62:21] Salesforce to pull in the use case [62:22] information. [62:24] But, it also meant that Genie 1 needed [62:26] to connect to Databricks in order to [62:29] pull all the consumption data that they [62:31] had, because I had a lot of nuanced [62:33] questions that I wanted to include in [62:34] the brief about what they're using Genie [62:36] for. [62:37] And this is what Genie 1 was able to [62:39] come up with. Um as you can see, it gave [62:41] me a comprehensive profile at the blur [62:43] some of it so you don't see the details. [62:45] Um but, it after you know, querying [62:48] Databricks and querying these systems, [62:49] they able to give me a comprehensive and [62:51] visual profile of all of our customers. [62:54] So, what I wanted to do was try this [62:56] exact same prompt with five of the [62:59] leading AI agents that were out there. I [63:00] tried two of the AI assistants that came [63:02] bundled with some of the software we [63:04] had, and three of the leading coding and [63:06] co-work agents. And this is what I saw. [63:09] So, the first AI assistant is actually [63:11] one that I use a lot, one I use to write [63:13] docs and things like that. And it [63:17] it came back with a response to my [63:18] prompt very very quickly. Actually, a [63:20] little bit too quickly. And there was [63:22] something a little bit off about it. And [63:24] the first thing I noticed was that it [63:25] said that we had 24 customers in the [63:27] pub. And I knew this was not true. So, I [63:30] asked it and said, "Hey, like where did [63:32] you get this 24 number from?" [63:35] And [63:36] guess what it said. [63:37] It confessed that it just completely [63:39] made it up. I was stunned, right? Cuz [63:42] this is actually I use this thing a lot. [63:44] Uh not just for at work, but I also have [63:46] a personal subscription. Um but it just [63:49] shows you goes to show you how important [63:51] enterprise grounding actually is. Now, [63:54] the other AI assistant that I use [63:55] actually did a lot better. It actually [63:56] searched existing documents and pulled [63:58] out account briefs. Um but they were all [64:01] sort of stale, you know, they were all [64:04] from a quarter ago, information that was [64:06] no longer relevant to the conversation I [64:08] wanted to have. [64:10] So, the results of this thing completely [64:12] unusable. The good news it was very very [64:14] quick. [64:15] Now, the the coding agents and the [64:17] co-work agents actually made a genuine [64:19] attempt and connected to all the all the [64:22] systems and the structured source in [64:24] order to pull live data out. But if [64:26] you've ever used one of these things, [64:27] you've probably experienced what I [64:29] experienced, which is that it went off [64:31] and tried very very hard to pull this [64:33] information out. It actually hit its [64:35] time limit um where it said asked for [64:37] permission to continue burning Lord [64:39] knows how many tokens in the process. [64:41] And then after I allowed it to to [64:43] continue, it went off a a a a away and [64:46] it it [64:48] iterated again and again. And it came [64:49] back with some cool ASCII art. Um but [64:52] then [64:52] confessed again that actually we were [64:55] far from done, you know, we had only [64:57] gone partway through. So, we're 30 [64:59] minutes into this thing at a partial [65:00] report. It's like a huge contrast with [65:03] what Genie 1 was able to do. [65:05] So, you might say like hey, this is sort [65:07] of a one-off test. Um you know, you [65:09] cherry-picked an example. But you know, [65:11] it's actually very, very consistent with [65:13] the research that Databricks research [65:15] had been doing on the ability of coding [65:18] agents to answer novel questions. [65:20] Actually, what we saw we created a [65:22] benchmark out of real questions that our [65:24] employees were asking of Genie, and we [65:26] asked coding agents to solve the same [65:28] problems. Now, these weren't [65:29] run-of-the-mill simple questions that [65:31] could be easily, you know, [65:33] um solved with some semantic layer. [65:35] These were like nuanced questions like, [65:37] "Hey, with all the all the customers who [65:39] use Genie, what percentage also use a [65:41] third-party agent?" Questions that, you [65:43] know, hadn't been asked before. And we [65:45] saw that coding agents were able to do [65:47] it [65:48] the best of them about half the time, [65:50] and it took minutes. And remember this [65:52] chart, we'll come back to it. But 50% of [65:55] the time for for a real data question is [65:58] basically unusable, right? You cannot [66:00] make You literally can flip a coin. [66:02] Um [66:03] and if you know how these agents work, [66:05] and I I'll explain this a little bit [66:07] already, you're [66:09] it shouldn't be surprising because they [66:10] basically go through this process of the [66:12] agentic loop, where where really getting [66:15] to the right answer is a matter of it, [66:17] you know, kind of making sure happening [66:19] upon like the right sequence of [66:22] reasoning steps to [66:24] chance upon like the right context it [66:26] needs to ask answer the question. So, it [66:28] kind of goes through this walk where it [66:30] it just does its best. And there is a [66:32] very real trade-off between the accuracy [66:34] that you get and the cost and runtime of [66:37] these things. And And this trade-off [66:39] results in what we see. [66:41] Now, our current best approach to [66:43] solving this problem is to inject some [66:45] context, right? Like to give it some [66:47] information that gives it hints on the [66:49] right thing. And if you come from the AI [66:51] world, you might call these things [66:52] skills. And if you come from the BI [66:54] world, you might call these things [66:55] semantic layers, but they're really the [66:56] same idea. It's like, "Hey, we can just [66:58] tell it the things that it needs to do, [67:00] and then it can do a better job." [67:02] And the problem with this is that, well, [67:03] while it's totally reasonable and in [67:05] fact best practices to do it for the [67:08] most stable business concepts you have, [67:10] it's unreasonable to do it for the [67:12] breadth of things that you do within [67:14] your organization. Think about how each [67:16] and every single marketing campaign [67:18] defines leads a little bit differently, [67:20] or how every single scrum team uses all [67:22] the fields in Jira a little bit [67:24] differently. You just can't [67:26] realistically write all this stuff down. [67:29] And before you say, "Hey, can't we just [67:30] use AI to solve this problem by [67:33] generating these models?" Well, it [67:34] actually just doesn't work. Actually, [67:36] Anthropic [67:37] published a great blog on the limits of [67:39] this, talking about that the fact that [67:41] if you do this, what you end up doing is [67:43] just encoding or like anchoring the AI [67:46] on one specific use case, and then it [67:49] actually [67:50] negatively result affects the results [67:52] overall. [67:54] So, what is our solution to this [67:55] problem? Well, I mean, Ali stole my [67:58] thunder a little bit. It is the Genie [68:00] Ontology. Right, the Genie Ontology is [68:03] an automatic context layer that adds a [68:06] learned layer on top of any modeled [68:08] context that you have, whether that's in [68:10] Unity or in one of your existing [68:13] semantic model or uh modeling tools. [68:16] Um it and it basically allows Genie and [68:19] any agents attached to the ontology to [68:21] do a much better job of accurately [68:24] answering questions with data. Let me [68:26] tell you how it works. [68:28] So, first, with the Genie Ontology, what [68:30] we do is connect to the systems that you [68:32] have that contain knowledge. So, for [68:35] example, using the Databricks system, [68:37] you have a ton of information in your [68:39] pipelines, queries, dashboards, and [68:41] things like that that tells an agent how [68:44] they can reason about data. And we [68:45] extract all of those things, things like [68:48] expressions and relationships, but also [68:50] expertise, like that the idea that this [68:52] person is really knowledgeable as about [68:54] a specific topic. And we extract all of [68:57] this into our internal knowledge store. [69:00] And then we go over this knowledge with [69:03] our an algorithm which you know, this is [69:05] how you know the engineers still run the [69:06] show at Databricks, we call Onto Rank. [69:09] And it's a page rank out like algorithm [69:12] that helps us determine which snippets [69:14] are actually authoritative. [69:16] And then with with this information, [69:18] when a when a question comes into Genie, [69:21] we're able to look up this information, [69:23] apply permissions because we also [69:26] extracted out the permissions of the [69:27] underlying source access so that we [69:29] don't have to you don't have to worry [69:31] about leaking permissions, and then we [69:33] just inject that context into the agent [69:36] loops so that Genie is able to answer [69:38] much better uh [69:39] questions much more accurately. [69:41] And the results of this is so that if if [69:43] I ask a question like how many people [69:45] are registered um without ever having to [69:48] model it, Genie knows about the nuances [69:50] of how [69:52] um how we captured registrations in our [69:54] internal systems because it was able to [69:57] extract that information out of some [69:59] source access. And if you want to know [70:01] where it came from, you can just inspect [70:02] it and see how it actually learned about [70:04] this these calculations from our, you [70:07] know, underlying dashboards. Uh [70:08] marketing team's a little bit mad at me [70:10] about this because I took this video [70:12] from 2 weeks ago, um but the numbers are [70:14] a lot higher now. [70:16] So, [70:18] coming back to this chart. So, we saw [70:19] what happened uh [70:21] with generic coding agents, how does [70:23] Genie with the ontology do? Well, in our [70:26] internal testing, what we've seen is a [70:28] consistent 30 percentage point plus [70:30] improvement in in accuracy and roughly [70:33] half the the run time of the leading [70:36] agents, right? And we're really [70:38] confident that as we continue to tune [70:41] the Onto Rank algorithm and expand the [70:44] the the the breadth of the ontology that [70:45] we're able to push this accuracy much [70:48] much higher. [70:50] So, with this level of accuracy enabled [70:53] by the Genie ontology, now we can truly [70:55] create a data coworker. One that you can [70:59] give to all of your employees to [71:00] actually make decisions with data and [71:02] even automate decision-making, right? [71:04] It's the It's the foundation. [71:07] So [71:08] I'd like to welcome now Elise Joris onto [71:11] the stage to show you what you can do [71:13] now that you have this foundation and [71:14] what you can do with Genie 1. [71:26] >> Hello everyone. [71:27] My name's Elise. I'm a product manager [71:30] here at Databricks for Genie and I use [71:33] Genie in a lot of my day-to-day work. [71:36] We actually have an exact review for our [71:38] Genie OKRs that's coming up and I'm [71:40] going to use Genie to prepare for it. [71:43] I'm starting here in Genie 1 and as you [71:46] can see, it's a pretty simple interface. [71:48] I have my chat here in the middle. We [71:51] have some [71:52] buttons that we can use to draft a doc [71:55] or create a skill. [71:57] On the left side I have my prior chats [72:00] as well as all the agents and the assets [72:02] that are available to me. But more on [72:05] that in just a second. [72:07] I'm going to start by asking Genie to [72:09] create a document for my review. Every [72:11] team at Databricks uses the same [72:13] template. It's got metrics, highlights, [72:15] lowlights. So I've given Genie a link to [72:18] that. [72:19] I've also asked to poke the people who [72:20] have open Jira tickets so that we can [72:22] keep those moving. [72:24] Now as Genie is running, we can actually [72:26] click into what it's thinking. [72:28] So as you can see here, it clearly found [72:30] something interesting in the Unity [72:32] Catalog Glossary. [72:33] Seems like it found some potentially [72:35] relevant ontology snippets and this is [72:38] actually all from data that I do already [72:40] have access to. Genie isn't forcing you [72:43] to create a separate permission system. [72:45] It's enforcing the permissions that you [72:47] already have through Unity Catalog. [72:50] Looks like it also grabbed some Google [72:52] files and found those Jira tickets. [72:55] It used MCP to comment on those tickets [72:57] just like I asked. [72:59] And now it's actually running some SQL [73:01] across both Databricks and BigQuery to [73:04] get the latest ground truth. [73:06] And that's actually really important. A [73:08] lot of generic agents are just reciting [73:11] what already exists in a document, but [73:13] there's no document for this week's [73:15] metrics. That's why I'm doing this. [73:17] Genie understands the numbers and is [73:19] actually able to compute the results [73:21] based on our live operational data. And [73:24] to that end, it looks like I already [73:26] have a doc for my review. [73:28] Looks like Genie pulled the metrics and [73:30] broke them down by region. [73:33] Gave me a bunch of nice visualizations. [73:36] Grab the highlights, the lowlights, [73:38] anomalies. And notice that it actually [73:40] did that fairly quickly. Rather than [73:43] just grinding through an agentic loop of [73:45] trial and error and eating up a lot of [73:48] tokens along the way, Genie just went [73:50] straight to the ground truth. And that [73:52] is really the Genie ontology at work. [73:55] So, how do we know that we can actually [73:57] trust this? [73:58] Well, for that we're going to pop down [73:59] here into the citations. This is [74:01] everything that Genie actually used to [74:03] answer my question. And again, we have [74:04] tables, foreign catalogs, tickets, docs. [74:08] Looks like it did actually use the [74:10] definition from the glossary. You can [74:12] see here that it's verified. [74:14] But I actually want to click into the [74:15] ontology. I want to make sure that Genie [74:17] inferred everything correctly. [74:20] So, what you're seeing here are ontology [74:22] snippets. These are again the facts [74:24] about our business that Genie has [74:26] learned and evaluated automatically. [74:29] It looks like there's a snippet for how [74:31] we define engagement. A snippet for what [74:34] qualifies as a blocker in Jira and one [74:38] for the SQL that we can use to pull [74:39] engagement by region. [74:42] I'm going to click into this first [74:43] snippet. And see here we see the uh full [74:46] definition. We're looking at sessions [74:48] that lasted longer than 30 seconds. And [74:51] it looks like Genie pulled this from our [74:53] mobile KPIs dashboard, which was [74:55] authored by our colleague Chung. [74:58] It's also got a high authority score. [75:00] This is a function of things like how [75:02] often the snippet is used or the asset [75:04] that it was pulled from. It goes back to [75:06] that onto rank algorithm that Ken was [75:08] talking about. And I can actually click [75:10] into Chung's profile and here I see that [75:12] he's engaged with a lot of the domains [75:15] our team cares about. And yeah, it [75:17] actually looks like Genie has pulled [75:18] quite a few snippets from the assets [75:20] that he's created. [75:22] Now, while I'm here, I might as well [75:24] schedule this to run every week, so I [75:26] don't have to manage Jira myself. [75:29] Great. And now I'm just going to take [75:31] this one step further. I'm going to ask [75:33] Genie to add a forecast to my doc. One [75:36] of my teammates already has a [75:37] forecasting skill, so I'm just going to [75:39] use that. And she didn't do anything [75:40] fancy. She just used the built-in [75:42] forecast function and added a few [75:44] instructions for how we like to review [75:46] these forecasts. [75:47] And again, this is really only possible [75:50] because Genie's able to compute these [75:52] results based on our live operational [75:55] data. [75:56] And it looks like it added my forecast. [75:58] And it also gave me a pretty nice [75:59] visualization here. [76:01] So now I'm going to ship this. I'm going [76:03] to send it over email and Slack. [76:07] Now, this OKR review isn't a one-time [76:09] thing. People are asking me for updates [76:11] on the mobile app literally all the [76:13] time. So I'm going to create an agent [76:15] that can answer questions using context [76:18] from this conversation. [76:22] I'm going to name my agent. [76:29] Great. Now I have an agent. This is an [76:32] expert co-worker that can answer [76:34] questions and take action related to the [76:37] Genie mobile app. [76:39] And unlike an OKR doc, which is probably [76:42] going to get stale pretty quickly. This [76:44] agent stays up-to-date, and again, it [76:47] can compute results on the fly. So, [76:49] while I'm here, might as well share this [76:51] with the other PMs at Databricks. [76:53] And now I'm going to click into this [76:55] agent. [76:57] So, as you can see, I'm actually [76:58] chatting directly with my agent. I'm not [77:01] going to have to specify what product [77:03] we're talking about because it's already [77:04] baked into the agent. It's customized to [77:07] the mobile app. And you can also see [77:09] that in these agent details that Genie [77:11] has automatically generated. [77:14] And more broadly, you can probably [77:15] imagine all the different teams that [77:17] might use something like this. Think [77:19] about any function that has to answer a [77:21] bunch of questions. HR, IT, finance. [77:25] Now, everyone in the organization can [77:27] create and collaborate on these types of [77:29] expert co-workers in one place. [77:33] Now, from here, I can go in and manage [77:35] my agent, you know, tighten the scope, [77:37] sharpen the instructions, add some [77:39] context, or I can certify it. And this [77:42] is going to appear on our organization's [77:44] agent list page as a certified agent, [77:47] and the Genie ontology is also going to [77:49] prioritize it. [77:51] Now, I don't do all of my work on the [77:54] desktop. I actually have to do a lot of [77:57] work on the go, which is why I have the [77:59] Genie mobile app. [78:02] And if I open my app, [78:05] on my phone, that is, it looks like I [78:07] just got a notification. [78:09] Uh engagement with the app is spiking. [78:12] It's all very meta when you have to work [78:13] on Genie with Genie. And I can click [78:16] into the app to learn more, or maybe ask [78:19] a follow-up question. [78:21] And to be clear, I didn't actually have [78:23] to set up this insight. Genie knows that [78:26] I care about engagement with the app [78:28] because of my activity, and it can [78:30] actually proactively notify me because [78:32] something interesting happened. And I I [78:35] that's actually really powerful. Genie's [78:37] no longer just a reactive Q&A tool. It's [78:40] now a proactive AI co-worker that can [78:43] bring me things that matter before I [78:45] even know to ask. So, in just a couple [78:47] of minutes, we moved the project along, [78:49] prepped for an exact review, got some [78:51] colleagues questions answered, and even [78:54] learned something that we didn't know to [78:55] ask about. That is a super high-quality [78:59] AI co-worker, and it's really only [79:01] possible because Genie understands our [79:03] business through the ontology. It can [79:05] compute results rather than just [79:07] reciting documents, and it can enforce [79:10] our organization's governance along the [79:12] way. So, we'd encourage you to try this [79:14] on your own data and share your [79:16] feedback. And with that, Ken is going to [79:18] come back out. Thank you all. [79:24] >> All right, what do you think? [79:27] So, [79:28] Genie 1 is a data-smart AI co-worker. It [79:32] has full core capabilities. You can [79:35] create documents, and again, compute [79:37] them live, not just summarize and recite [79:39] existing things that you have, but net [79:41] new analysis. You can connect it them it [79:44] to all of your systems thanks to the [79:45] connectors that we have. You can [79:47] schedule it to work when you're back in [79:49] the background, and you can schedule it [79:51] without having to keep your laptop open. [79:54] Uh and you can integrate it to all of [79:55] your systems, even custom ones, because [79:57] we have custom MCP tools all managed [80:00] through uh Unity AI Gateway, as well. [80:03] And you can develop custom skills that [80:05] you can share and create with your [80:06] co-workers, even create a dedicated [80:09] domain-specific agent just by talking to [80:12] Genie, so you can improve the [80:13] performance of that focused agent over [80:16] time. [80:18] And Genie, again, works on all of your [80:20] data thanks to the large and growing [80:22] ecosystem of connectors that we have [80:24] through federation and ingestion. And [80:26] customer and it's all managed by Unity. [80:29] So, and we'll have a deep dive on all [80:31] the governance features of the Unity AI [80:33] Gateway tomorrow. And what this means is [80:36] that customers like Warner Music Group [80:38] is able to use Genie for both Databricks [80:40] and non-Databricks [80:43] data and and customers like General [80:46] Motors is able to deploy hundreds of [80:48] Genie agents into production to improve [80:50] the performance in every single one of [80:52] their teams. And it allows customers [80:55] like Foot Locker to be able to think [80:57] about how to transform their business to [80:59] a world employee is armed with the [81:02] ability to make decisions with data. [81:06] We really want every single user to be [81:08] able to leverage data to do their jobs, [81:11] which is why we're striving to make [81:13] Genie available everywhere. So, I'm [81:14] pleased to announce that Genie is [81:16] integrated now into Teams and Slack [81:19] available today. And it'll be available [81:22] on your mobile devices on both both [81:26] major platforms on Android iOS again [81:28] today. So, as Ali mentioned, this has [81:31] already become the most popular way for [81:33] Databricks employees to experience Genie [81:35] and I'm pretty sure it's going to be [81:36] that way for your for your custom for [81:38] your employees as well. [81:40] Now, for those of you who are using one [81:42] of these existing AI agents or have [81:45] developed your own and would like to [81:47] benefit from the super [81:49] superior accuracy and data smarts of [81:51] Genie, we're also pleased to announce [81:54] the availability of the Genie MCP app. [81:57] And that gives your your existing agents [82:00] or existing tools the ability to tap [82:02] into Genie's ability to compute results [82:05] accurately. And all of and that is also [82:08] available today. [82:10] So, we really believe that Genie one is [82:12] unique with the ontology as the only AI [82:15] agent that allows [82:17] everyone in your organization to [82:18] actually consistently make results and [82:20] not just summarize summarize information [82:23] from existing documents. And we want to [82:25] make sure it's available to to [82:26] absolutely everybody. So to in order to [82:29] to kind of encourage that to happen, [82:31] we're giving $10 worth of tokens um for [82:35] every single user inside your [82:36] organization every single month. So [82:39] there's really no reason not to get [82:40] started with with Genie Genie 1. And [82:44] we're going to make Genie 1 generally [82:47] available today. [82:53] All right. So please connect Genie 1 to [82:56] every everything inside your [82:57] organization and give it to everybody. [83:00] Well, next I'm excited to introduce to [83:01] you Pepsico who's going to tell you a [83:03] little bit about the work that we've [83:05] been doing together. [83:42] >> Welcome to the stage Pepsico's Global [83:45] Chief Data and AI Officer Magesh [83:48] Bagavathi [83:50] and Databricks Co-founder Ali Ghodsi. [84:11] >> So Magesh, we saw a pretty cool demo out [84:13] there talking about Genie, but here [84:14] we're about to talk about all the [84:16] amazing things that Pepsi's been doing. [84:18] Um I'm sure everybody out here is [84:19] familiar. They just saw the video, but [84:22] you guys are pretty massive and complex [84:24] business, but best in class when it [84:26] comes to AI transformation. So, how [84:28] about you tell me a little bit about [84:29] that? [84:30] >> Well, thank you, Arsalan. It's fantastic [84:32] to be here. Great crowd, very talented [84:34] crowd, and I'm here to represent team [84:36] PepsiCo. [84:37] And you know, this journey started off [84:40] for us around 6 years back as we were [84:42] really initiating our full-blown [84:44] end-to-end digital transformation and [84:45] process transformation as well. [84:47] And the focus has always been faster for [84:51] our customers and consumers, stronger [84:53] for our people, products, and our [84:55] businesses, and ensuring we're better [84:57] for the planet. So, that's been the core [84:59] mission of PepsiCo. [85:00] And as we talk about the size and scale [85:03] of PepsiCo, sometimes it amazes myself, [85:06] right? [85:07] Um [85:08] we have 320,000 employees globally, [85:11] operate around 200 countries, [85:14] and this is continuous non-stop [85:16] operation. We have I don't know, some of [85:17] you might not know, we have one of the [85:19] largest fleets in North America. So, [85:21] when you have this level of complexity, [85:24] you also want to most importantly be [85:26] able to service your consumers. [85:28] And one of the things we measure how we [85:30] service the consumers is occasions. [85:33] We do around 1.4 billion occasions a [85:36] day. An occasion is you saw the great [85:37] products in the video. It is really how [85:40] you all consume our fantastic products [85:43] every day. So, that's what we call as an [85:44] occasion. [85:46] So, when we have to service these [85:47] occasions through 6 million retailers [85:49] and also your eB2B and e and and D2C [85:52] avenues, it's a pretty complex endeavor. [85:55] So, for us what we truly believe is an [85:57] AI transformation is a data [85:59] transformation, and that's what we've [86:00] undertaken. So, what the team has done [86:02] really is fundamentally over the last 6 [86:05] years really moved us from a legacy [86:08] architecture [86:09] where we had 60 plus data lakes to one [86:12] lakehouse right now running on [86:13] Databricks. [86:15] And also from [86:17] uh [86:18] a monolithic architecture to a truly [86:21] pluggable Lego-based architecture. That [86:23] transformation for us is really how we [86:26] start unlocking the value of the data [86:28] that's in this platform because now 90% [86:31] of PepsiCo's universe, data universe, is [86:34] now on the platform. And this which is [86:36] what we call as enterprise data [86:37] foundation. So for us this is a living [86:39] breathing platform that we double down [86:41] on. [86:41] >> That's awesome. And look, you were one [86:43] of the early ones that talked to us [86:45] about Genie. I know you had efforts [86:47] internally and you basically came to Ali [86:48] and I and you're like, we have to build [86:50] something like this. [86:51] Why has Genie been such a game-changer [86:53] for you guys? [86:54] >> Yes, we did come to you guys a few years [86:57] back. And [86:59] this was when we were actually, as we [87:01] started onboarding all this data into [87:03] the platform, we saw that there's going [87:05] to be so much richness this platform can [87:07] offer for us in terms of the core [87:08] capabilities. We call this AI for BI and [87:11] our ability to dialogue with data. [87:13] Whether it's a warehouse operator, [87:15] whether it's a frontline employee, [87:16] whether it's a knowledge worker, we want [87:18] to be able to dialogue with data just as [87:19] Ken showed. And for us, as we were [87:22] looking at this AI for BI consoles, it [87:25] was taking us anywhere between three to [87:26] six months. If you want to vectorize it, [87:28] you want to rag, do do rag patterns and [87:31] try to bring the data on board and [87:32] really then start executing this. As we [87:34] were having conversations, we were [87:36] starting to see [87:37] you know, the accuracy was not where we [87:39] wanted it to be. So that's why it was [87:41] taking six three to six months to [87:42] productionize. And this is when we [87:44] reached out to to you guys. [87:46] So at this point in time, having almost [87:49] 95% [87:50] accurate and and catalog data in the [87:53] enterprise data foundation, for us Genie [87:55] is all about unlocking the value from [87:57] the Databricks environment. [87:59] >> So [88:00] that's kind of an amazing story. Now, [88:02] bring that home for me. Make it real, [88:04] right? You guys actually have actual use [88:06] cases of how you've been using it. And I [88:07] know even though you're getting started, [88:08] talk to me about some of the value and [88:10] how you've been using it. [88:11] >> Yeah, it's not real until you [88:12] productionize it, right? And um [88:16] what we have in PepsiCo's and ambition [88:18] is we want to identify PepsiCo. [88:20] And as we think about identifying [88:22] PepsiCo, we think about concepts such as [88:24] the supply chain AI brain. One of the [88:27] components of the supply chain AI brain [88:29] is the procurement brain. [88:31] Very strong uh sponsorship from our [88:33] chief procurement officer to really [88:35] start looking at insights for all things [88:37] procurement, both direct and indirect [88:39] spend. [88:40] We're a $95 billion revenue company. So, [88:42] there is a decent amount of direct and [88:43] indirect spend. So, [88:46] um within this platform called Spend [88:48] Wise, where our procurement leaders and [88:51] our key senior leaders operate to [88:52] understand what their overall indirect [88:53] spend is, [88:55] we've had dashboards, capabilities, and [88:57] reports. But as we unleashed Genie [88:59] within that that space, [89:01] it was truly a big game changer. Just in [89:04] the first few weeks of launch, we almost [89:06] had 30,000 queries, engagements back and [89:09] forth. And all of a sudden you start [89:10] seeing this user population shift from [89:13] reports and dashboards into a Genie type [89:16] engagement. One of the things I love [89:17] about Genie Genie, by the way, is a [89:19] little bit of what Ken showed was that [89:21] ability for you to go into agent mode [89:24] and really then start looking at deep [89:26] research. So, you have tremendous amount [89:29] of flexibility within the platform, [89:31] which is what me and a lot of the [89:33] leaders within PepsiCo really like. And [89:35] now what we're looking to do is we're [89:36] now looking to expand this use case [89:38] across all parts of PepsiCo, including [89:40] commercial reporting for Europe and also [89:42] for Latin America as well. [89:44] >> Fair enough. And then I know they're [89:45] going to yell at us cuz we're running [89:46] late, but I'd be remiss to not ask you, [89:48] you know, what's next for you guys? You [89:50] guys have such ambitions of what you [89:52] want to drive. So, what's next? [89:54] >> You You know, for us, it's all about how [89:56] do we get to an agentic PepsiCo. And [90:00] we've got huge ambitions. For us, it's [90:02] really about going from insights to [90:04] action to outcomes for our end user and [90:08] for our population. So, our goal is now [90:10] to go from these 30,000 plus reports [90:12] down to 50 plus AI for BI or Genie [90:15] consoles. That's a big outcome for us. [90:18] That's how we're going to drive our our [90:19] business going forward. This will really [90:22] help us agentify PepsiCo. This will help [90:24] us [90:24] agentify our 320,000 associates. [90:27] >> Awesome. Well, it's been an incredible [90:29] journey and excited to see what comes [90:30] next. Thanks so much. [90:31] >> Thanks for the partnership. [90:32] >> Appreciate it. [90:32] >> Thank you all. [90:47] >> Welcome to the stage, Databricks senior [90:49] director of product management, Bilal [90:52] Aslam. [90:58] >> Good morning. Good morning. [91:01] I'm going to talk to you about Lakehouse [91:03] but just a little quick note first. I [91:05] wore this I wore this purple jacket two [91:07] years ago. Talked about Lakehouse. I did [91:09] not wear it last year and the biggest [91:11] piece of feedback was bring it back. So, [91:13] here it is. [91:14] Uh my name is Bilal Aslam. I'm a senior [91:16] director of product management at [91:17] Databricks. Super excited to talk to you [91:19] about Lakehouse. Now, uh first of all, [91:21] thank you to Mugesh and Arslan and Ali [91:23] and Kenan and everybody who's come here [91:24] and talked about AI and agents. Now, uh [91:27] super exciting future uh but if you're [91:29] anything like me, if you're a data [91:30] engineer, uh you're probably sitting [91:32] there thinking, "This is great but all [91:33] of this needs data." Uh all these apps, [91:36] agents, AI, everything needs data and [91:39] you might be thinking, "My job got a [91:40] little bit harder." [91:41] Uh and that's because uh with data [91:44] engineering, the destination is simple. [91:46] The destination is get to the right. Get [91:48] to the business outcomes we want. We [91:49] want agents, applications. We want [91:51] real-time operations but we're actually [91:53] starting from the left. We're starting [91:55] from raw data sources. We have data in [91:58] Kafka, Salesforce, NetSuite. We have [92:00] on-premise databases like Oracle and SQL [92:02] Server, and the messy middle, that's [92:04] data engineering. That's our job to take [92:06] that raw data, turn it into insights. [92:09] And over the years, we have built [92:11] architectures like these. I put this up [92:13] here because it's representative. You [92:15] don't have to look at every box here. [92:16] It's a lot of logos. It's a lot of [92:18] boxes. It's a lot of lines between [92:20] boxes. And that's just to get data from [92:23] one place to the other. [92:24] And you know, we've made peace with [92:26] complexity. Sure, there's a lot of [92:27] tools, but you're probably sitting in [92:28] here thinking, "Okay, at least it [92:30] works." [92:31] Uh [92:32] even though you know instinctively that [92:34] this complicated, not unified [92:36] architecture is missing a bunch of [92:37] critical things. We want to version [92:39] control everything as data engineers, [92:41] but in this stack, you can version [92:42] control some things, not the others. Uh [92:44] you can version version control your [92:45] Spark code, but not your Kafka [92:47] infrastructure. [92:48] You want governance everywhere, but it's [92:49] pretty much impossible. You're reading [92:51] and writing to two too many different [92:52] places. And there are parts of the [92:54] stack, as we will see soon, that are [92:55] just locked up. And finally, you want [92:57] everything to scale, and you want to [92:59] save money. Uh so you can build more [93:01] applications and outcomes for your [93:02] business. But not all parts of the stack [93:04] scale equally well. [93:07] I'm here to tell you that your job, and [93:09] my job as a data engineer, is about to [93:11] get quite a bit harder. [93:13] What you're seeing here is Genie code. [93:15] It's a purpose-built coding agent for [93:16] data science, machine learning, and data [93:18] engineering. [93:19] Uh if you're using it, that's awesome [93:21] because uh 60% of lake full pipelines, [93:24] this is a crazy statistic, are already [93:26] written by Genie code in just 3 months. [93:30] We just released this. [93:31] It's pretty awesome. [93:33] On the other hand, this statistic scares [93:34] me a little bit as a data engineer. [93:36] And that's because I have a crush on [93:38] agents. Agents are great, but all agents [93:40] mean new data, new pipelines, more to [93:43] build, more to manage on that fragile [93:45] infrastructure. We're going to get [93:47] swamped. [93:49] So what do we do? We start by removing [93:51] the moving pieces. And actually, it's [93:52] super important. We don't just want to [93:54] remove complexity with more complexity. [93:56] We want to replace that with open [93:58] formats, open open frameworks, open [94:00] tools. We ultimately need to simplify [94:02] the data stack. [94:05] I'm here to make a pretty bold claim, [94:07] and my bold claim is that Lake Flow is [94:09] that unified stack. That it gives you [94:11] that open foundation for AI and agents, [94:13] so that you can build the future that [94:15] your company deserves. And now, we work [94:17] with hundreds of partners, lots of [94:18] tools. They're all interoperable with [94:20] Lake Flow and the Lakehouse. You can use [94:22] them as well. [94:23] So, let's get started. I'm going to do a [94:24] little bit of a speed run through this. [94:26] Uh how do you simplify data [94:27] transformation? ETL. This is where [94:29] you're spending most of your time and [94:30] money. [94:31] Uh you have Spark, you have lots of [94:33] legacy Spark jobs. You may have a Spark [94:36] distribution on EMR or some other [94:38] vendor. [94:39] I'm going to replace that with Apache [94:40] Spark declarative pipelines. If you're [94:42] not using them, please try them out on [94:44] Lake Flow. They're open, you can run [94:46] them anywhere, including on your laptop, [94:48] and they're declarative, so you can just [94:50] focus on the work to be done. And [94:52] because they're open, agents are really, [94:54] really good at writing these pipelines. [94:56] They unify batch and streaming. [94:58] Many years ago, DBT was the only game in [94:59] town when it came to SQL. I like writing [95:01] ETL in SQL, but Spark declarative [95:03] pipelines unifies that as well. It has [95:05] SQL and Python. You don't have to [95:07] choose. And finally, if you're using [95:09] Flink for real-time streaming, that's [95:11] probably because actually low latency [95:12] streaming was really hard on Databricks [95:15] and Spark. But now, last year we [95:17] open-sourced real-time mode, which is a [95:19] brand new engine for low latency stream [95:21] streaming, and I'm super excited to [95:22] share that it's now available inside [95:24] Spark declarative pipelines. So, you can [95:26] get millisecond streaming on an open [95:28] framework that runs anywhere. [95:30] Awesome. So, we're taking care of this [95:32] complexity, we're getting rid of this, [95:33] we're building an open foundation for [95:34] agents and AI. [95:36] Well, what about all the shadow IT [95:38] that's happening? So, you see, you may [95:40] have 20 data engineers in your company, [95:42] but you have hundreds of analysts who [95:43] are building pipelines in drag-and-drop [95:45] tools. These pipelines are built on [95:47] their laptops, proprietary formats, [95:50] proprietary tools, and if somebody [95:52] leaves the team, you're out of luck. [95:55] Super excited to share that Lake Flow [95:56] Designer, the no-code data prep tool [96:00] powered by Genie, is now available. It's [96:02] generally available. You can try today. [96:04] You can try it right now. [96:07] Thank you. [96:09] >> [applause] [96:10] >> Awesome. And by the way, one super cool [96:12] thing is that Lake Flow Designer doesn't [96:14] build any proprietary pipelines. It just [96:16] builds Spark declarative pipelines under [96:18] the hood. Again, you can just run them [96:20] anywhere. I think you're starting to see [96:22] a pattern. All right, we're going to go [96:23] a little faster. I'm going to go ahead [96:24] and simplify data ingestion. If you're [96:26] using a SaaS that has hundreds of [96:28] connectors, that's great, but it's [96:30] probably landing data in proprietary [96:32] formats. Your agents don't know how to [96:35] operate this SaaS. You cannot automate [96:37] this. You can barely monitor it or [96:38] observe it or govern it. [96:40] As Ali mentioned, I'm super excited to [96:42] share that Lake Flow Connect now has [96:44] more than 100 connectors. If there's a [96:46] connector you're thinking of, we've [96:48] either built it, we're building it, or [96:50] we're going to build it, and there's a [96:51] new community that's building [96:53] open-source connectors. You can even [96:54] build your own. And guess what? I think [96:56] you're going to guess what I'm going to [96:57] say next. This isn't a special type of [96:59] pipeline or ETL. It's just Spark [97:01] declarative pipelines under the hood. [97:03] That was pretty awesome. [97:04] All right, let's talk about Kafka. As [97:07] data engineers, we use Kafka day in and [97:09] day out for high-volume telemetry. We're [97:11] bringing that in. Kafka's a buffer, but [97:13] we also know Kafka's a real pain to [97:16] manage, okay? [97:18] Super excited to share that Zero Bus [97:20] Ingest is a fully managed service that [97:22] is now 100% wire compatible with Kafka. [97:25] So, you can take your Kafka producing [97:26] code, point it at Zero Bus Ingest, and [97:29] you can land data into the lakehouse in [97:31] open formats, ready for AI. So, no small [97:34] files, millions of small files at 12 GB [97:37] a second. [97:38] So, it's pretty awesome. So, you can get [97:39] rid of Kafka for your stack. [97:42] Great. [97:43] Uh okay, so we're almost there, and [97:46] you're going to see we're still building [97:47] everything on an open ETL framework [97:50] that's declarative. But you see you see [97:52] this little Airflow icon hanging out [97:54] there. That's your orchestrator. You [97:55] have to use that to trigger things, to [97:57] schedule things. Let's simplify that as [97:59] well. So, you're probably using Airflow [98:01] because you like writing workflows in [98:02] Python. Well, Lakeflow Jobs supports [98:05] writing workflows in Python now. It's [98:06] just pure Python. You can also build [98:09] point-and-click DAGs. [98:11] It's fully serverless, so you don't have [98:12] to manage any infrastructure. And one [98:14] interesting fact is that we did a survey [98:17] of our customers, and 80% of our [98:18] customers are using old Airflow [98:20] distributions. So, they're managing [98:22] them, maintaining them, they have [98:23] security vulnerabilities, and they're [98:25] not even getting all the new features. [98:27] Okay? [98:28] And I'm the last thing I'm really [98:29] excited about because for the longest [98:30] time, let's face it, with Jobs, it was [98:32] really good at orchestrating Databricks, [98:34] not so great at orchestrating other [98:35] systems. So, super excited. This is [98:38] probably one of my favorites in this [98:39] conference is we're releasing 50-plus [98:40] integrations so you can orchestrate [98:42] everything. You can think of these as [98:43] Airflow operators. They're open source, [98:45] and you can orchestrate everything, [98:47] including Snowflake. Okay. So, where [98:49] where does this bring us? This brings us [98:51] to a an open stack. Everything is pretty [98:54] much building open source declarative [98:56] pipelines. [98:58] And [98:59] this is not a brand new product. We've [99:02] been at this for a while, and it's ready [99:04] for the world's toughest data workloads. [99:06] I want to share some statistics with [99:07] you. Spark declarative pipelines now [99:10] process 200 trillion rows of data every [99:12] single day. [99:14] Lakeflow Jobs [99:16] is running the world's biggest data [99:17] workloads. It's about 1.7 billion job [99:20] runs per month. And finally, serverless [99:23] compute is really popular. You don't [99:25] want to manage clusters. You don't want [99:26] to set up VPCs. 50% of our customers [99:28] have now opted in to use serverless [99:30] compute. That's pretty awesome. [99:33] Great. So, we've I hopefully I have [99:35] convinced you to some degree that we [99:37] made it easier to build, and now you [99:39] might be thinking, "Okay, we have an [99:40] open stack. I can start building. Yeah, [99:43] I can write the code. Maybe you're going [99:44] to go try out Genie and it's going to [99:46] build something for you. [99:48] But what about operations? I haven't [99:50] really solved that for you. Actually, [99:51] I've kind of made the problem worse, [99:52] right? Because now you're going to say, [99:54] "Great, now more people can build more [99:56] pipelines, [99:57] more outages, more problems." [100:00] And here's the reality, simplifying [100:02] pipeline creation is actually the easy [100:04] part. Although this was pretty tough, [100:05] right? [100:06] It's data operations that are really, [100:08] really hard. Here's a survey that is by [100:12] one of our partners and it shows that [100:13] data engineering teams spend more than [100:15] 50% of their time on maintenance. So, [100:18] that's where your time is going. It's [100:19] only going to get worse and even with [100:21] this time spent on maintenance, you [100:23] still see up to 60 hours of downtime [100:26] every month. So, this sucks. [100:29] Now, you might be thinking, "But Bilal, [100:30] you just told me that this is a unified [100:32] stack. Surely it has APIs and endpoints [100:35] and tables. Maybe a coding agent can [100:36] just fix it." And unfortunately, not [100:38] really. Let me walk you very quickly [100:40] through why. And the big idea here is [100:42] that data engineering is not software [100:44] engineering. It's something a bit more. [100:46] In software engineering, your code is [100:48] self-describing. [100:49] It's just literally describes itself. [100:51] And in data engineering, it's data and [100:53] code. [100:55] In software engineering, your tests [100:57] either work or they don't. They're fully [100:59] deterministic. In data engineering, you [101:01] have code and data. You can't just test [101:03] data. It's statistical. [101:06] And finally, in software engineering, [101:08] and this is the big one, failures are [101:10] allowed. When something goes wrong, [101:12] you're going to see a trace. You're [101:13] going to see an exception and then you [101:14] can roll back your deployment. In data [101:16] engineering, failures are silent and [101:18] permanent. You're probably going to get [101:20] an angry phone call from a CEO or a [101:21] stakeholder saying, "Hey, how come this [101:23] data's bad?" Maybe a week later. [101:26] Uh so, why do coding agents trip up [101:28] here? Well, that's because when it comes [101:30] to detection, they're just missing data. [101:33] They're missing telemetry. And you can [101:35] give them Spark logs, for example, but [101:37] these are multi-megabyte traces. You [101:39] have to be very careful to manage the [101:41] the the context window there. [101:42] And similarly, when it comes to [101:44] assessing the root cause, they're [101:45] missing lineage. Now, you have to export [101:47] your lineage out of the system, and to [101:49] some degree you can do this. Coding [101:50] agents are getting better and better, [101:52] and context windows are getting better. [101:54] And then you have to remediate. They [101:56] have to write the code fix. [101:58] But it's actually the final step, which [101:59] I'm going to be a little pedantic about [102:01] and call it verify. [102:02] You can't with data, you can't just run [102:04] your unit tests. You actually have to [102:06] take your code fix, and you have to run [102:08] it on production data. Okay, and if this [102:11] is giving you shivers, that's exactly [102:13] right. It really should. So, what does [102:15] your agent need? Your operations agent [102:17] needs to combine code and data. That's [102:19] tools and skills. To some degree, it can [102:21] do it. [102:23] To figure out the root cause, it needs [102:26] read access to production data. [102:28] Uh [102:29] who's comfortable with that? It's an [102:30] agent you didn't write and you don't [102:32] trust. I I don't see a single hand [102:34] raised here. [102:36] But actually, your coding agent also [102:38] needs write access to production data. [102:41] That's the only way it can verify the [102:43] fix. Like, who's here wants to give a [102:45] random agent write access to their [102:47] production data? [102:48] There's got to be one person. [102:50] Okay, there's no one who wants to do [102:52] that, right? Because it can cause chaos. [102:54] It can drop everything. [102:57] And this is the big insight. So, we've [102:58] been looking at this problem. It's a [102:59] tough problem, and the big realization [103:01] we came to was that your operations [103:03] agent actually needs to live in the data [103:06] plane. It cannot live outside of the [103:08] data plane. And if you think about it, [103:09] that makes sense, because the data plane [103:11] has the data. It has the lineage. It's [103:14] also the right governance boundary. [103:16] So, [103:19] I'm super, super excited to announce [103:21] Genie Zero Ops. A new background genie [103:24] that puts your data and AI operations on [103:26] autopilot. [103:28] Great. I'm going to give you a demo, but [103:30] let me quickly walk you through how it [103:31] works. [103:33] So, for detection, Genie Zero Ops [103:35] autonomously builds per table machine [103:37] learning models, and it continuously [103:39] fine-tunes them. And it has native [103:41] access to metrics, events, and logs. So, [103:43] you don't have to build this plumbing [103:44] yourself. We already have tens of [103:46] thousands of these machine learning [103:47] models in production with Genie Zero [103:49] Ops. [103:51] To assess, it does graph ranking on data [103:54] lineage in Unity Catalog. So, I'll show [103:56] you how it walks the lineage forward and [103:58] backwards. And then to figure out the [104:00] root cause, and this is where you spend [104:01] a lot of your time. The root cause is [104:02] basically what went wrong. It actually [104:04] has a supervisor agent and a fleet of [104:06] sub agents who do research and then come [104:09] to a consensus on what is the likeliest [104:10] cause. [104:12] For remediation, Genie Zero Ops works [104:14] with Genie Code. It cooperates. If Genie [104:16] Code has access to your code, to your [104:18] ticketing system, to your version [104:20] control, it uses all that. So, it can [104:21] also update tickets for you as it goes [104:23] through the life cycle. [104:25] And finally, verification is the [104:27] toughest step. To verify, this is the 10 [104:29] years of building an open lakehouse, it [104:31] builds shallow clones of your production [104:33] data. These are super cheap, super fast [104:36] branches similar to what we do in Lake [104:37] Base. And then it utilizes native [104:40] network and code isolation in [104:41] Databricks. All right. So, I'm going to [104:43] show you a quick demo of what Genie Zero [104:45] Ops actually does. [104:48] Great. [104:49] So, [104:50] let's see. [104:52] All right. So, you're going to notice a [104:53] couple of things here. First of all, [104:55] this does not look anything like a [104:57] dashboard. We're not flooding you with [104:59] metrics, events, alerts, light, uh you [105:02] know, red lights and all that. [105:04] What you see is something that looks and [105:05] feels a lot like an email inbox. It's [105:08] actually we modeled it after a [105:09] prioritized email inbox. [105:12] So, for example here, one cool thing is [105:14] that you get to see severity. So, every [105:16] incident is ranked by severity, so you [105:17] can only you have you can just focus on [105:19] what really matters. [105:20] And also, this isn't just a list of jobs [105:23] and pipelines and failures and things [105:24] like that. Over here, for example, [105:26] here's an alert, and Genie Zero Ops [105:28] figured out that hey, 16 jobs are [105:30] failing. They're actually the same [105:32] incident. So, the natural grouping saves [105:35] you time. [105:36] All right. So, I'm going to go ahead and [105:37] look at an actual incident. And what's [105:39] happening here is that there's an [105:41] upstream table called top fan voters and [105:44] something's going wrong with it. About [105:45] 10 minutes or so ago, [105:47] the row count dropped quite a bit. [105:50] And what Genie Zero Ops does is by the [105:52] time you log in, your background agent [105:55] has actually done all the thinking for [105:57] you. It has done all the investigation [105:58] for you. And as a data engineer, I can [106:01] start looking at things like what's the [106:02] impact. Now, when it comes to impact, [106:05] not all tables are important, not all [106:07] pipelines are important. So, it walks [106:08] the lineage forward. But, in this case, [106:10] it figures out that hey, top fan voters [106:12] is a pretty important table. [106:14] Downstream tables depend on it and this [106:16] fan fan engagement dashboard depends on [106:17] it. So, it marks it as critical. [106:20] Then, it deploys the root cause agents [106:22] to figure out what's going on. And what [106:24] these agents do is this is what they're [106:26] saving you hours of time. [106:28] They start from this table and they fan [106:30] back using lineage and they go [106:32] investigate potential causes upstream. [106:34] So, over here, this is a fairly simple [106:36] DAG. For example, you know, it could be [106:39] any one of these tables, but really it [106:40] finds this one table, this fan [106:42] interactions table, that's the one [106:44] that's problematic. [106:45] Okay. Then, it went ahead and [106:47] autonomously wrote a fix for you, but [106:50] this fix is not deployed to production. [106:52] You're in control. What it did do, and I [106:54] want to share this, this is super cool, [106:56] is that it went and created a shallow [106:58] clone. It's like the lake base branch of [107:00] your production data. By the way, you [107:01] have to give it permission. It won't do [107:03] it automatically. You have to tell it [107:04] you can do it with this pipeline. [107:06] And this could be terabytes of data. [107:08] It's a shallow clone. It deployed the [107:10] fix and it actually verified that the [107:11] right number of rows are returned. This [107:13] is awesome. This is hours, if not days, [107:16] worth of work. And now, at this point, [107:18] I'm just going to go ahead and create a [107:19] pull request and I can follow my uh [107:21] software development life cycle. [107:25] Uh let me show you one other super cool [107:26] thing that Genie Zero Ops can do. So, [107:28] one other thing Genie Zero Ops does is [107:31] if you give it permission, it will [107:33] autonomously scan tables for things like [107:35] PII. So, in this case, our application [107:37] is unwittingly exposing the PII of about [107:40] a thousand users. Okay? That's pretty [107:42] bad. [107:43] And Genie Zero Ops creates a report for [107:44] me. It finds out what the PII, the [107:47] personally identifiable information is. [107:49] It gives me a per-table breakdown. It [107:52] looks at lineage to figure out this is [107:53] important. Yeah, this is pretty bad. And [107:55] then, in this case, it comes up with a [107:57] proposed fix that's not a code fix. It's [108:00] a uh it's a Unity Catalog a policy fix, [108:02] and I can deploy it. [108:04] Okay. So, that's Genie Zero Ops. We're [108:06] super excited about it. [108:08] And for data engineers like me, and [108:10] hopefully for you, it means far less [108:13] time building, far less time [108:15] maintaining, and much more time building [108:18] on a unified and simple data stack. [108:20] Thank you very much. [108:26] >> Welcome back to the stage, Ali Ghodsi. [108:39] >> All right, super cool. I'm very excited [108:41] about Zero Ops. It's going to make our [108:42] life so much easier. Okay. So, now we're [108:44] getting to really exciting part of our [108:46] uh show where we're going to actually [108:47] talk about a lot of the announcements [108:48] that I mentioned. We're going to see [108:49] Reynold uh introduce a lot of those [108:51] things that I mentioned that we uh have [108:53] innovated on in the last year. So, but [108:55] before I do that, I want to put it all [108:56] in perspective, because it can get kind [108:57] of confusing uh in the data world. [108:59] There's so many different technologies, [109:00] there's so many moving pieces. So, we [109:02] wanted to simplify that. So, we call [109:04] this um what you see here on this slide [109:06] the known data realm. Okay? It's kind of [109:09] like the Game of Thrones okay? of data [109:11] infrastructure. Okay? This is like the [109:13] troubled waters. So, I'm going to [109:14] explain it. What does this look like? [109:16] Okay. So, on the west of this planet, we [109:19] have the world of OLTP databases, okay? [109:22] So, this realm of OLTP databases was [109:25] invented in the '80s. And what it was is [109:28] people needed a database behind every [109:29] piece of software. So, this software, [109:32] for instance, you might have an ATM, [109:34] you're going to go withdraw money from [109:35] it, that needs to hit the database. It [109:37] needs to do it really fast. We want that [109:39] latency to be milliseconds. We want it [109:41] to never go down and want it to be super [109:43] reliable. It can't be that it ever [109:45] messes up the transactions, okay? [109:47] And over time, we've had these niche [109:49] databases that also have appeared. We [109:50] have key-value stores, we have now [109:52] vector search with AI. Uh so, that's [109:56] this island of OLTP. Okay. [109:59] On the right-hand side of this map, what [110:01] we have is the world of data [110:03] warehousing. They appeared shortly after [110:05] the '80s because people wanted to ask, [110:07] you know, how many how much data volume [110:10] went through this ATM here? Or how much [110:13] money has actually transacted on all the [110:15] ATMs? [110:16] Asking that question [110:18] uh on those planes of data warehousing [110:20] or uh OLTP on the left uh would be too [110:23] complicated. And it would take down [110:25] those databases and it would affect the [110:26] ATMs, and we never want that. So, we [110:28] want that to be up and running and want [110:29] it to be isolated. Okay. So, on the [110:31] right-hand side, we started doing data [110:33] warehousing, and now you could ask these [110:34] more analytical questions, okay? All [110:36] right. But very soon, in the last 15 [110:38] years, people wanted to ask much more [110:40] advanced questions. Not just what's the [110:42] total transaction volume on the ATMs, [110:45] but how much is Ali going to withdraw [110:47] today? Can you predict the future? So, [110:49] we started doing data science and more [110:50] and more [110:51] uh advanced statistical analysis. Okay. [110:55] And then, last few years, we've now had [110:57] real-time analytics appear. And [110:59] real-time analytics, that's, you know, [111:01] agents or applications or dashboards [111:04] that want really low latency. [111:06] Um they can't quite answer complicated [111:08] questions and doing big joins of data [111:11] and saying, you know, let's do this [111:12] analysis of all this data, but they can [111:14] give you dashboards that give you some [111:15] analytical queries. So, for each of [111:17] these we needed different things, and we [111:19] needed to move data from the left-hand [111:21] side, from the OLTP databases all the [111:23] way to the right. Okay, so for that, [111:25] that's where data engineering came in, [111:27] and that's where we had all the [111:27] complexity of moving data and shuffling [111:29] it from the OLTP databases all the way [111:31] to the OLAP engines on the right-hand [111:33] side. Okay, so this is what the world [111:34] looked like. All right, you just heard [111:36] Bilal. So, this world of data [111:38] engineering, that's what he was focused [111:39] on. How you actually get your data from [111:41] all these systems ready uh so that you [111:43] can start doing operations on it. So, we [111:44] just cleaned this up. Okay, these [111:46] planes, he just cleaned them up and [111:48] added roads. But, now what we're seeing [111:51] is that there's this onslaught [111:54] of agents coming. All right, we're [111:56] having agents that requires us to ship [111:58] way more data between these different [112:00] islands. Okay, and we're going to get [112:01] crushed by these storms that are going [112:03] to come, and we're going to lose ships. [112:05] Okay. Our infrastructure is super [112:06] brittle. We've already seen many of the [112:09] very important infrastructure sites that [112:11] we depend on go down in the last year. [112:14] This is going to happen to every [112:14] organization. It's almost like everybody [112:17] just hired double the number of [112:18] employees. Okay, cuz the agents are [112:20] submitting queries, they want more data [112:21] stuff, more data movement is happening. [112:23] So, we need to simplify this. [112:25] Okay, so how do we simplify this? We [112:26] kind of did that already uh when we [112:29] started Databricks, and what we did is [112:31] that um we kind of merged data science [112:34] and data engineering. Spark already [112:36] could do big data movement, and it [112:38] allowed you to machine learning with [112:40] things like Spark ML. [112:42] And 5 years ago, we announced Lakehouse, [112:45] which lets us do in one engine data [112:47] warehousing, data engineering, and data [112:49] science. So, we've already kind of [112:50] accomplished that. That's kind of great. [112:52] Um but, we still have this real-time [112:54] analytics [112:56] uh island separate, and we need to make [112:58] a separate copy of the data if we want [112:59] to have really, really low latency. [113:02] Because the analytics stacks today, all [113:04] the data warehouses that are there, they [113:06] can't go down to a fraction of a second. [113:08] No matter what you do, there's a hard [113:10] trade-off. They can't do the complex [113:11] queries and the fractional uh second [113:14] queries that real-time analytics has. [113:15] So, we need separate engines for all of [113:17] that. Okay. So, for this, I'm excited to [113:20] announce our latest innovation. I'm [113:22] going to let my co-founder Reynold Xin [113:24] do that. So, please welcome him on [113:25] stage. [113:32] >> All right. [113:34] Thank you, Ali. [113:35] Um I'm super excited coming on stage [113:37] today to talk to you about this new uh [113:39] effort. And honestly, it's probably the [113:41] largest single largest innovation we've [113:43] done since our introduction of [113:44] Lakehouse. [113:46] And as Ali already mentioned, um with [113:49] the Lakehouse, we were able to actually [113:50] unify data science, data engineering, as [113:52] well as data warehousing. [113:54] And uh our data warehousing business has [113:56] taken off exponentially since its [113:58] introduction 5 years ago. It was [113:59] recognized by the Gartner and Forrester [114:02] both as a leader in data warehousing. I [114:04] think over 60% of the Fortune 500 uses [114:07] the Lakehouse for warehousing. And I [114:09] joke about we actually scan exabytes of [114:11] workload depending on which time zone [114:12] you are before you have breakfast. [114:14] The um and over the course of last few [114:17] years, we worked fairly hard to improve [114:19] performance. At every single Data and AI [114:21] Summit, I think me or Shawn somebody [114:23] else would come up on stage introduce to [114:25] you some massive performance performance [114:27] improvements we've done. [114:29] But the reality is kind of the [114:30] warehouses or the Lakehouses hit a wall [114:33] at about a second. [114:34] And what do I mean by hitting a wall? It [114:36] doesn't mean you can never get a [114:38] response from the Lakehouse or warehouse [114:40] in less than a second for a query. It [114:42] just means if you have a very stringent [114:44] workload that you want to have very [114:46] tight SLAs in the range of milliseconds [114:48] or hundreds of milliseconds, it it would [114:50] be very very difficult to accomplish [114:52] that because you might actually get [114:53] latency spikes. [114:54] And [114:55] as a result, I think for many [114:57] organizations that should require [114:59] um workloads that have very stringent [115:01] SLAs, they separating setting up a [115:03] separate serving stack. They would copy [115:06] their data instead of the data warehouse [115:08] or the lakehouse and copy a portion of [115:10] it that's required for serving into a [115:12] separate stack. And this caused a lot of [115:14] problems. [115:15] It's actually data pipeline to maintain [115:17] for every data set. [115:18] Um this serving stacks tend to perform [115:21] super well on very simple queries, but [115:23] doesn't actually work well for the more [115:25] general workloads. It's a governance [115:26] nightmare. You now have to worry about, [115:29] "Hey, did I forget to actually secure my [115:30] data set in this specific platform, but [115:33] I actually did in my underlying [115:34] warehouse?" [115:36] So, if the whole theme here is [115:37] unification, [115:39] what if you never had to move your data? [115:41] What if you never have to copy it? And [115:42] we asked ourselves that question a [115:44] couple years ago. [115:45] And now it turned out it's actually [115:47] fairly fundamental. It's very, very [115:48] difficult to tackle this problem. Um as [115:51] a matter of fact, Michael Stonebraker, [115:53] who's a Turing Award winner as some of [115:54] you might know him as the original [115:56] creator of Postgres, wrote a whole [115:58] academic paper about it called "One Size [116:00] Fits All, an idea whose time has come [116:02] and gone." And in this paper, [116:05] um Stonebraker actually argued it's [116:07] very, very difficult to engineer a [116:09] single database engine that's capable of [116:11] running a wide variety of workloads. And [116:13] ultimately it's because the increasing [116:15] complexity makes it very, very difficult [116:16] to actually engineer the system. [116:19] And the fundamental reason for that is [116:21] if you look at how database engineering [116:23] were done in the last four decades or [116:25] even five decades, um every database [116:27] engineering team started working [116:29] approximately this way. Um Databricks [116:31] included. [116:32] We'd go target stuff, "Hey, here's a new [116:34] workload we might want to actually uh [116:36] study and then understand and then [116:37] actually build a system for it." Um and [116:40] the way to do it is to uh study all the [116:41] latest and the coolest academic papers [116:44] about the specific type of workloads. [116:46] And some of the papers actually go back [116:48] 40, 50 years. Some of the papers are [116:49] probably in the last few years. You try [116:51] to study them, understand, "Hey, am I or [116:53] am I uh so apply in real life?" And you [116:56] spend a lot of time implementing it, and [116:58] at some point you try to roll it out to [117:00] test how well it works with real world [117:01] workloads. And inevitably, this happens. [117:06] So, the new technique you implement, or [117:07] new It could be a new algorithm, could [117:09] be a new data structure. It works [117:12] remarkably well for a subset of the [117:13] queries, and it works terribly for a [117:16] subset of the queries. So, the database [117:18] ends up, "Hey, we're doing better here, [117:20] and actually regressing." And many of [117:22] the algorithm and data structures [117:24] actually work really, really well for [117:26] very, very low latency to guarantee [117:27] maybe a millisecond response time, tend [117:30] to actually backfire when have larger [117:31] data sets, and vice versa. So, this is [117:33] one of the biggest problems. [117:36] Um so, we 2 years ago decided, "Hey, can [117:39] can we break the status quo? How can we [117:41] challenge it and do the best we can?" [117:44] And uh we actually went back to drawing [117:45] board so with amazing engineering team, [117:48] uh we actually flipped the way how [117:50] database engineering were done. Instead [117:51] of starting with ideas and techniques, [117:54] went back to the workloads. [117:55] Um so, luckily, on Databricks we have [117:58] collected over zetabytes of data we [118:00] scanned, traces from all of this data um [118:03] which in the range of quadrillion [118:04] natural traces. I have to learn how to [118:06] pronounce that word. [118:07] Um and all coming from trillions of [118:08] queries that were executed. And based on [118:11] all of this um traces, we're able to [118:13] actually build a machine learning model [118:16] um that give you very high fidelity [118:18] estimation of what might happen in [118:20] practice given the specific new [118:22] technique. And based on this machine [118:24] learning model, we can do two things. [118:26] The first is we can select the right [118:29] algorithm to dispatch very quickly at [118:31] runtime. But there's not an LM cuz LMs [118:33] we have too high latency. We can [118:35] actually pick the right algorithm to [118:37] dispatch at runtime. But even more [118:39] importantly, we can decide and predict [118:41] what algorithms to even implement before [118:44] it hits production. [118:46] Because there are millions of algorithms [118:48] and data structures out there. It would [118:50] It'd short to actually find them all. [118:52] Knowing what to implement is one of the [118:54] biggest advantage one can get. [118:56] Right. So, the resulting engine of that [118:58] is Raiden, which all know is a reference [119:00] to Mortal Kombat. [119:02] The [119:03] So, Raiden [119:05] Um [119:06] and [119:07] the resulting engine is actually pretty [119:09] remarkable. And you might have actually [119:11] used some of the other serving stacks [119:12] out there. Um you have some and if you [119:15] have, you run into many of the issues [119:16] we're talking about. For example, uh [119:18] some of them can't run queries longer [119:19] than 5 seconds. Almost all of them [119:21] require to copy the data into a very [119:23] specific format. It's very engine [119:25] specific. They can't run complex joins [119:27] and all of that. But Raiden can actually [119:28] do it all, right? [119:31] And the way Raiden will show up is going [119:32] to come in the form of a new SQL [119:34] warehouse. [119:36] And um this is kind of abstract. So, let [119:38] me give you maybe a quick demo [119:40] for you to help understand what Raiden [119:42] actually does. [119:44] Right. Can I get the demo on stage? [119:46] Okay. So, here I have a very simple [119:49] setup here. Honestly, it's just a very [119:50] simple query. You have the New York taxi [119:52] data set query. You don't really need to [119:54] understand what it does. But for those [119:56] of you that actually know what the data [119:57] set is, it's fairly small. And this runs [120:00] I First of all, I ran this query in [120:02] Lakehouse today. All right? It finished [120:04] It's not terrible. It finished in about [120:06] a second. All right? And on the [120:09] right-hand side, I have the Raiden [120:11] setup, which we call Lakehouse RT. And [120:13] let's try to run this. [120:16] Okay. I don't know if you can see this [120:18] number. [120:19] Uh [120:22] >> [applause] [120:26] >> Let's try to run it one more time. [120:27] Yep. [120:28] Did we hardcode it 0.007? No, we didn't. [120:32] Um [120:33] Okay. [120:34] But that's just one query. [120:36] All right? What's the fun of running [120:37] just one query? [120:39] Um here I want to show you a demo of [120:41] what would happen in practice when you [120:43] start hammering the system. [120:45] And we've built a simulator simulating [120:47] real dashboard application workloads [120:49] hitting against the written engine. And [120:52] I'm going to show you start with just [120:53] one active session here. So, let's [120:54] imagine this is an actual dashboard [120:56] running. We're just going to load this [120:58] dashboard. It loads fairly quickly, [121:00] right? It's probably what we expect. But [121:02] what happens if you scale up to 10, 100, [121:05] or why don't we try 1,000? [121:07] So, now we're going to simulate 1,000 [121:10] active agents hitting the same engine at [121:13] the same time. Each of the agent, by the [121:15] way, is generating more than one query. [121:17] The dashboard actually takes about eight [121:18] queries before it to load. [121:20] Um and they will all hit exactly the [121:22] same moment. Let's see what happens. [121:25] Done. [121:26] All right. [121:28] So, we executed [121:30] >> [applause] [121:30] >> 8,000 queries at 6,000 queries per [121:33] second with a tail latency of 37 [121:36] milliseconds. This is incredible. [121:39] None of existing systems can do this. [121:41] That's a very simple demo. [121:45] Okay. [121:46] But [121:47] when I told you, "Hey, none of the [121:49] existing systems can do that." [121:51] Let's put it into the perspective of [121:53] existing systems. So, here I want to [121:55] show you a benchmark measuring latency [121:57] versus throughput. Throughput here means [121:59] queries per second. [122:00] Right? So, we have the Y axis, which is [122:02] the [122:03] P90, 90th percentile latency, and X [122:06] axis, the number of queries executing. [122:08] And we're running a very simple query, [122:10] TPC-H [122:11] Q6, which is a canonical query of [122:13] scanning a little bit of data and [122:15] running some aggregation. And let's see [122:16] what happens when we test the different [122:18] systems out there. So, for a competitive [122:21] vendor, we're using their latest [122:23] warehouse. We're able to actually push [122:25] the warehouse. [122:26] It started fairly low latency, but as [122:28] you start pushing up the 160 queries per [122:31] second, the latency started climbing up [122:33] and hitting a wall. [122:35] So, the same vendor actually announced a [122:38] fairly innovative offering last year [122:39] called Interactive Warehouse. [122:41] It's so interactive, they can't actually [122:43] read the same data with a general [122:45] warehouse. The only way to actually have [122:46] it work is to copy the data into a [122:48] separate warehouse. [122:50] Um, and the interactive warehouse does [122:52] do much better than their latest [122:54] generation general warehouse. You're [122:55] able to push like 300 queries per second [122:58] before the latency starts spiking up. [123:01] Um, and we also tested through the open [123:04] source [123:05] um, query engine that's known for being [123:07] able to run very simple queries very, [123:09] very fast with the yellow color. [123:11] Um, and it's actually running much [123:13] better on this specific workload [123:14] compared with the uh [123:16] um, Gen2 and the interactive warehouse. [123:19] It's able to push all the way to a close [123:22] to actually, I think 15,000 queries per [123:25] second. That's pretty good, right? But [123:27] then they just started crashing right [123:28] after. It doesn't actually time out, it [123:29] just crashes. [123:30] So, how can Redondo, right? How does [123:32] Redondo? [123:35] On the same workload, Redondo actually [123:37] run all the way to 12,000 queries per [123:40] second while keeping the tail latency [123:42] still below a second. [123:46] >> [applause] [123:51] >> There's one other benchmark I want to [123:52] show you, [123:53] which is what if we run all of TPC-H, [123:56] which is a standard data warehousing [123:58] benchmark. [123:59] Um, as you can see on the screen, [124:01] when data set is small, actually all [124:03] systems run reasonably well. I mean, [124:05] it's like a few seconds running 22 [124:06] queries. I mean, some are a little bit [124:08] faster than others, but they all run [124:10] pretty well. But as you scale up, for [124:12] example, 100 GB, you started seeing the [124:14] yellow systems [124:15] uh runtime started spiking. And this is [124:18] actually I'm pointing to point out the [124:20] very specific part of my uh talk about [124:21] earlier, which is a specialized serving [124:24] engine doesn't actually work very well [124:25] on the more complex joints, which will [124:27] show up in the full TPC-H queries. And [124:30] of course, uh when we scaled up to [124:32] actually bigger amount of data, both [124:33] systems actually started crashing. The [124:35] interactive warehouse started crash uh [124:37] can't even finish because he cannot run [124:39] any queries longer than 5 seconds and [124:41] the yellow system couldn't actually [124:43] complete because they started running [124:44] out of memory and Raden was actually [124:45] able to complete all queries at all [124:46] scale factor. [124:49] You might have benchmark fatigue at this [124:50] point, so many benchmarks. [124:53] Um the who cares about synthetic [124:55] benchmarks? [124:56] Um so one of the thing that we're really [124:58] excited about in the process of [125:00] developing Raden is the design partners [125:02] we've been working with. Some of which [125:03] I've shown their logo on the screen [125:05] here. Want to share with you some two [125:07] results from all these design partners. [125:10] The first one is Amverce. Amverce is a [125:12] global um leading data and AI platform [125:15] for the energy sector and they tested [125:18] Raden with 11 of their most [125:20] representative production queries. And [125:22] what they found compared with the [125:23] existing serving stack is that with [125:25] Raden the same extremely low latency [125:28] queries remain sub 100 milliseconds [125:30] whereas the longer running queries can [125:32] now actually dramatically shrink their [125:34] run time and across the board on average [125:36] they're actually getting a 16 times [125:38] speed up. [125:39] So Paul from Amverce was super excited [125:42] to see that they're able to now execute [125:44] queries in tens of milliseconds but also [125:47] shrink the run time of the longest [125:49] running queries by almost 100x. [125:53] And Meta also tested Raden and they [125:55] found very similar things. They were [125:57] able to have their typical queries come [125:58] back in tens of milliseconds and the [126:00] most important thing is you don't need a [126:02] separate system on the side just for [126:04] your serving. [126:06] So what Raden does it can give us the [126:08] best low latency possible [126:10] at extremely high concurrency. [126:13] Um it can actually run real world [126:14] workloads really really well and it can [126:16] also handle the complex CDL um EDW [126:19] workloads super well. [126:21] And today very excited to announce the [126:23] first product offering powered by the [126:25] Raden engine. It's called the Lakehouse [126:27] RT. And Lakehouse RT is a new warehouse [126:30] that's showing up starting with read [126:31] only workloads. They will support [126:33] millisecond um performance and massive [126:36] concurrency, but the most important [126:37] thing is directly on the lake, governed [126:39] by Unity Catalog on the same Delta or [126:42] Iceberg data. [126:46] >> [applause] [126:48] >> So, [126:50] what can you do differently with [126:52] Lakehouse RT, where RT stands for real [126:54] time? [126:55] Um challenge your conventional wisdom. [126:57] If you have a separate serving stack, [127:00] test it out and you might be able to [127:02] collapse the two platforms into one. And [127:04] that will give you [127:06] millisecond performance, massive [127:07] concurrency, directly against your data [127:10] lake and open formats. [127:14] And beta starting today, [127:15] try to talk to your account team to get [127:17] early access. [127:18] Um and there's also two talks at this [127:20] conference starting soon today on uh [127:23] Raiden and Lakehouse RT that you can [127:25] learn more about. [127:26] And [127:29] that's it. And next, I think I would [127:30] like to invite Nikita on the stage talk [127:32] about Lake Base. [127:38] >> Not quite. [127:40] Um [127:41] so, Reynold refuses to say what Raiden [127:45] stands for, so I'm going to get up here [127:46] and embarrass him a little bit. Okay, so [127:49] Reynold runs all of the data platform [127:51] stuff. All this innovation you see [127:52] that's part of his team. So, his team [127:54] actually called the engine Raiden [127:56] because it stands for Reynold's dream [127:58] engine. Okay, it's true. True story. [128:01] Um you know, A, I think they wanted to [128:02] embarrass him, [128:04] uh so that's why he doesn't want to call [128:05] it that. But secondly, we are I was [128:07] actually talking to one of the team [128:07] members and he said, "Look, there's so [128:09] many projects at Databricks and you guys [128:11] always, you know, kill the projects [128:13] that, you know, you think you don't want [128:14] us to be spread out thin. So, we [128:15] thought, what if we were name this after [128:18] a co-founder?" [128:19] Okay. So, uh that's the Raiden engine. [128:22] So, this is the map that we had of the [128:24] data world and then you saw that now we [128:26] had real-time analytics on a separate [128:27] island. Now we can shift these tectonic [128:29] plates and merge that in. You get the [128:31] same format of the data under the hood. [128:34] You get the same Unity catalog [128:36] governance. And you can now do all of [128:38] your low-latency processing as well as [128:41] the big data processing in this one [128:43] island using the same technology stack [128:45] under the hood. So it comes out as part [128:47] of Lakehouse RT, which stands for real [128:48] time. [128:50] And we've actually priced it super [128:51] competitive, so it's essentially the [128:52] same price. So very excited about this. [128:54] Okay. So now we're going to turn our [128:57] eyes to the left and we're going to look [128:58] at this island of OLTP. And I'm very [129:01] excited to welcome on stage Nikita [129:04] Shamgunov, who was CEO of MemSQL, later [129:07] SingleStore, and then Neon, which was [129:10] acquired by Databricks, where he leads [129:11] all of this and he's been doing OLTP for [129:13] a very very long time. So he'll tell us [129:15] about Postgres. So let's welcome Nikita [129:17] on stage. [129:27] >> Thanks to AI, [129:28] we now generate software. We're not [129:30] writing it by hand. [129:32] And we're looking at the trends for the [129:34] first 6 months of year 26. And now we [129:37] can confidently say we're going to have [129:39] more software generated in the next year [129:42] than in the history of humanity. [129:46] And every application still needs a [129:48] database. [129:50] So [129:51] what do next generation application and [129:54] agents need from a database? [129:58] So the first thing is it needs to be [130:00] familiar. [130:01] And the reason to that is the more [130:04] agents know about the systems, and they [130:06] know by scanning the internet and [130:07] scanning all the Stack Overflow forums [130:09] and Reddit, the better they are at [130:11] running and operating it. So it needs to [130:14] be open source. [130:15] It needs to be popular. And it needs to [130:17] be extensible, so you can consolidate [130:20] all your main and also niche workloads [130:23] on a single database platform. [130:26] It also needs to be nimble. [130:28] With this onslaught of applications that [130:30] are coming at us and all the various [130:32] dev, test, and staging environments, [130:35] uh the system needs to be serverless and [130:37] it needs to be branchable so you can [130:39] easily create environments for your [130:41] agents in which they can safely operate. [130:44] And finally, as you run those things at [130:47] scale and at volume, uh it needs to be [130:49] cost-effective. [130:52] The final thing it needs to be mission [130:53] critical. Operational databases run your [130:56] business and so [130:58] um that means is it needs to be [131:00] infinitely scalable, it needs to be [131:02] fast, and it needs to be extremely [131:04] reliable. [131:06] So, [131:08] familiar, [131:10] nimble, [131:11] and mission critical. [131:14] So, we decided to start with Postgres, [131:16] the most advanced open-source database [131:19] in the world. [131:20] It has a largest uh ecosystem in the [131:23] world and also lots and lots of [131:25] extensions. Again, you can consolidate [131:28] all your additional workloads to the [131:29] main one, to the operational database [131:31] system one, into the platform. [131:34] It's also open-source and understood by [131:36] every agent on the planet. [131:40] But Postgres is a monolith [131:42] and the compute and storage in the [131:44] monolith are tightly coupled. So, if you [131:46] want to make it nimble, um you need to [131:49] do something. You need to re-architect [131:51] that system. [131:53] The main idea we had is we can decouple [131:56] storage and compute and move storage [131:59] into the lake. [132:01] But it's actually, you know, much harder [132:03] than you might think. [132:05] And so, for that, we need to [132:06] re-architect storage from the ground up. [132:09] The lake storage is, you know, has a lot [132:12] of very, very good properties. It's [132:14] inexpensive. It's very easy to scale, [132:17] but it's also slow and transactionally [132:19] inconsistent. [132:21] So, as we are rebuilding [132:23] uh the storage for Postgres to run it on [132:25] the lake, we had to introduce two [132:27] additional services, one for reads and [132:29] one for writes. [132:31] For writes and transactional consistency [132:33] consistency, [132:34] uh we built safe keepers. They implement [132:36] a consensus protocol called Paxos. [132:39] So, they give you low latency writes. [132:41] And then we introduced another service [132:43] called page servers that serve pages to [132:45] Postgres compute. Those systems are [132:47] called page servers and they deliver on [132:50] low latency reads. [132:52] Then we put it all together and [132:53] integrate with the lake, [132:55] and that gives us uh Lakebase. [132:58] The fully managed serverless Postgres [133:01] that runs on the lake. [133:05] So, what does Lakebase give us? [133:08] Well, um it certainly is nimble, right? [133:11] So, we can provision Postgres in under [133:13] 500 milliseconds. And as you have lots [133:16] and lots of developer [133:18] uh staging [133:20] um environments for every potentially PR [133:22] that you you want to move into GitHub, [133:25] um we can scale that also to zero, [133:27] right? So, with lots and lots of [133:29] environments and they're serverless, we [133:32] need to deliver on TCO. So, we [133:34] automatically shut down environments [133:36] that you don't use. [133:40] We introduced two more patterns that are [133:42] incredibly useful in this new agentic [133:44] world. The first one is branching. [133:46] Every Postgres database you can is in [133:49] easily branch, and it also uh branches [133:52] in about 500 milliseconds. So, you can [133:54] create an additional environment for you [133:56] to create dev test or staging. [134:00] Another another pattern that kind of [134:01] emerged in that agentic development is [134:05] uh snapshot restore. [134:07] With any with a simple mouse click or an [134:09] API call, you can snapshot your [134:11] database, then unleash your agent to do [134:14] work uh [134:15] to build some software, change the [134:16] schema, change the data, whatever. [134:19] If the work is to your liking, you can [134:21] proceed. If not, you can instantly roll [134:23] back to the previous snapshot. [134:27] So now it's built for agents, but is it [134:30] mission critical? Can it support your [134:32] business? [134:34] Let's start with scalability. [134:37] With Lake Base, all you need to set is [134:40] error bars for your compute. You can [134:42] say, "Well, don't scale my compute below [134:44] this minimum or over that maximum." And [134:47] you want to do it for potentially cost [134:49] controls. But within two those error [134:51] bars, you actually can [134:54] scale up and down automatically [134:57] as your workload changes. So we'll scale [135:00] the system up at your peak hours, scale [135:03] things down [135:04] at maybe on weekends or nights. [135:08] Storage, of course, is on the lake and [135:11] that gives you infinite scalability. You [135:13] will never run out of storage and you [135:15] will never run out you will never need [135:17] to run management operations if you're [135:20] approaching, you know, your disk size or [135:21] whatever. [135:25] So now you might be wondering, "So you [135:28] changed the architecture [135:30] of Postgres at the storage level. [135:32] So is it fast? [135:34] How does it compare with the industry [135:36] implementations of Postgres as a [135:38] service?" [135:40] So this is a very similar [135:42] graph to what Relational was showing for [135:44] Aiden [135:45] where we're measuring latency as we [135:47] scale throughput. [135:48] Obviously here it's an operational [135:50] system. So it runs much smaller queries [135:53] that in analytical systems, but they run [135:56] at at very high concurrency. [135:58] And we're comparing it with [136:01] the first cloud vendor, which is quite [136:02] popular. And that cloud vendor taps out [136:05] about at about 130 [136:08] operations per minute. And by the way, [136:10] we're running a fairly standard industry [136:12] benchmark called TPROC. [136:15] Another cloud vendor with slightly [136:17] different architecture was able to push [136:21] its throughput to about 350,000 [136:23] operations a second, but after that, of [136:25] course, latencies started to spike as [136:28] well. [136:29] And I'm very excited to to to showcase [136:32] some of the incredible performance work [136:33] that we've delivered. And LakeBase can [136:36] scale under 10 milliseconds for each [136:39] transaction, for each operation, all the [136:41] way north of 600,000 operations per [136:44] second. [136:48] LakeBase is is ready for [136:51] almost any workload. [136:53] Mission critical also means a lot of [136:55] features, right? And those features are [136:57] security, compliance, encryption, you [137:00] name it, and we have it. [137:03] But I also wanted to showcase something [137:06] that we're in a very very unique [137:08] position to deliver on. [137:10] So, cloud outages don't happen every [137:13] day. [137:14] But when they do happen, they're they're [137:16] devastating to your business. And now [137:18] that a lot of your business is [137:20] increasingly automated and run by agents [137:23] themselves, [137:24] cloud outages could could be really [137:26] really disruptive to your operation. [137:29] So, I'm incredibly excited [137:33] to introduce first fully managed [137:37] cross-cloud disaster recovery. [137:40] >> [applause] [137:45] >> This is what we mean by truly mission [137:47] critical. [137:48] With fully managed cross-cloud disaster [137:50] recovery, you can set up your system to [137:53] run cross-clouds. So, you can provision [137:56] LakeBase, let's just say US West [138:00] AWS, [138:02] and then [138:03] a replica US East Azure. [138:06] And in the case of AWS outage, you can [138:09] instantly failover from one cloud to [138:13] another and continue your uninterrupted [138:15] business operations. [138:18] So this is Lakehouse. It's fully managed [138:21] serverless Postgres that runs on the [138:23] lake and it is mission critical. [138:26] We've only been on the market for a [138:28] year, but we already have over 3,500 [138:31] enterprise customers that trust us with [138:34] their mission critical workloads. One of [138:36] them, I would like to invite here on [138:38] stage. Please welcome Federica Cohen [138:41] from MasterCard. [138:54] >> How are you doing? Thank you for having [138:56] me. [138:57] >> Doing good. [138:58] Excited to have you. [139:00] So MasterCard [139:02] aims to have data and services account [139:04] for over half of its revenue. [139:07] To get there, you consolidated 80 [139:09] different services as the one agentic [139:11] platform. [139:13] So what didn't work in the past and [139:15] what's working now? [139:17] >> Well, it's a great question and maybe [139:18] just to level say just to give a sense [139:20] of the dimension that we're operating [139:22] in. We're operating in about 200 over [139:26] 200 countries. We have billions of card [139:29] holders and we process more than 150 [139:31] billion transactions every year. And so [139:33] at the scale that we have to operate, [139:35] bringing together our services was very [139:37] critical. Now, Lakehouse helped us [139:39] accelerate that by creating a shared [139:42] foundation where we can bring in [139:44] differentiated insights, our governance, [139:47] and isolation when we need it to [139:50] basically bring together agents that can [139:53] reason in real time, but more [139:54] importantly can work together with that [139:57] shared context and that architecture. [140:02] >> So, you've built a a reusable [140:04] architecture um here using uh Unity [140:07] Catalog and Lake Base to help with the [140:09] creation of a virtual C-suite. [140:12] How does standardizing on Lake Base [140:14] help? [140:15] >> Well, standardizing on Lake Base has [140:17] been pretty critical. [140:19] Uh for example, in March we announced a [140:21] new solution called virtual C-suite. [140:23] It's designed to help small businesses [140:25] operate with executive-level [140:26] decisioning. Think about the average [140:29] small business owner. They're very busy. [140:31] It is very difficult for them to take [140:33] decisions on cash flow, [140:36] on forecasting. And so, what we've done [140:38] is we created a suite of agents that [140:40] will help each as a digital executive [140:42] aiding in a number of digital [140:44] responsibilities. The first one we're [140:45] starting with is the virtual CFO. It [140:47] brings and helps uh make decisions on [140:49] cash flow, on payments, and how to make [140:52] working capital, all of which are [140:54] make-or-break decisions. And so, we use [140:56] Lake Base to create a foundation that [140:59] allows us to take in insights from their [141:01] data, from their performance, bring it [141:04] all together, and the benefit is because [141:06] we're using Lake Base, we're able to [141:07] have that shared understanding. So, as [141:10] one agent takes action or generates [141:11] insights, it becomes instantly available [141:13] for the next one. [141:15] >> So, let's talk about multi-tenancy. With [141:17] thousands of issuer banks on a single [141:19] platform, [141:21] um multi-tenant isolation is a [141:22] high-stakes challenge. How did you use [141:25] Lake Base to approach a secure [141:27] bank-level data separation while [141:29] enabling AI-driven insights that can [141:31] continuously learn and improve over [141:33] time? [141:34] >> It's a great question. So, there is [141:36] another solution that we've been working [141:37] on that's coming soon called uh [141:39] Performance Pools. And for those some of [141:41] these solutions, we have to support [141:43] thousands of issuing banks all on a [141:44] shared platform. Now, think about the [141:46] challenge that we have to have. For us, [141:48] data isolation isn't optional. It's [141:50] foundational to the trust that our [141:51] clients have with us uh and it helps us [141:54] manage things like data residency and [141:56] other requirements that we have. And so [141:59] this capability that is coming soon that [142:01] helps us bring always on insights into [142:03] action to measurement and continuous [142:05] loop requires us to keep our issuing [142:07] data isolated and separate for [142:10] protecting each of our clients but at [142:12] the same time we have to have the [142:13] ability to consolidate insights in a [142:16] secure and transparent and anonymized [142:18] way so to draw it. Lake Base allowed us [142:20] to basically create this solution by [142:23] establishing a single foundation that [142:25] then can apply to thousands of issuers [142:27] at once. And so for us it's enabled us [142:30] to have that trust and that embedded [142:32] foundation within that we can create for [142:34] one client and then expand and scale [142:35] horizontally to all of them. [142:39] >> Your team [142:40] went from concept to scale ready MVP [142:42] demo in 7 weeks. My mind is kind of [142:45] blown. I've never seen anything like it. [142:47] Um and and for a company with with [142:49] MasterCard's regulatory bar this is kind [142:52] of wild. Um [142:54] so how did keeping agent memory, model [142:57] serving, governance inside of a single [142:59] platform allow you to move fast without [143:02] sacrificing trust? [143:04] >> We're incredibly proud of the speed that [143:06] we've been able to achieve but part of [143:08] the key piece is that we are [143:09] uncompromising in our approach to how we [143:12] manage and govern our AI and our data. [143:14] And so what we were able to do with Lake [143:16] Base was to build all of the foundations [143:18] from the start. So when we start [143:20] developing a new MVP what we what we [143:22] have the ability to do is embed that [143:24] governance, the guardrails that we need [143:26] and a number of the other isolation and [143:28] foundations that we need to have in [143:29] place from the get-go. By building this [143:31] in with the Lake Base architecture that [143:33] allowed us to move much faster than [143:36] usual because all of those trust [143:38] components become non-negotiable. So [143:40] instead of supposed to accelerating by [143:42] cutting corners we accelerated by [143:43] building the foundation from the start [143:45] into something that frankly now we can [143:47] reuse every time we build additional [143:48] agents. [143:50] >> So, looking ahead, what becomes possible [143:53] as your agents get more autonomous, what [143:55] are you looking forward to building [143:57] on top of Databricks platform? [143:59] >> Well, I'll go back to something you said [144:01] at the start, Nikita. Um we aspire to [144:03] have our services power more and more of [144:05] Mastercard. Well, what that means in an [144:07] agentic world is that over time we're [144:09] going to have an increasing set of [144:10] agents that our clients in the ecosystem [144:12] uses, but also it means that they have [144:15] to be trusted and that they have to have [144:18] that shared foundation within. And so, [144:20] as we start moving from agents that not [144:22] only provide insights and information, [144:23] but can help you take action, help drive [144:25] the ecosystem, [144:28] that shared foundation is what's going [144:29] to help them learn from each other and [144:31] drive their power even further. So, when [144:33] you think about some of the things I [144:34] mentioned like the virtual C-suite, like [144:36] performance pulse, like capabilities [144:37] we're driving in agentic commerce, [144:39] bringing that level of trust for us is [144:41] the currency of innovation. And we [144:43] believe it's going to let us scale much [144:44] further thanks to that shared components [144:47] that we'll be able to build in. [144:49] >> Thank you for being an incredible [144:50] customer, Fred. [144:51] >> Thank you for having us. Thank you, [144:52] everybody. [144:53] >> [applause] [144:55] >> And now, [144:56] please [144:58] a co-founder of Databricks, Patrick [145:00] Wendell. [145:14] >> Welcome to the stage OpenAI's president [145:17] and co-founder, Greg Brockman, and [145:20] Databricks co-founder and vice president [145:23] of engineering, Patrick Wendell. [145:33] >> [music] [145:35] >> Take your seat here. [145:42] >> Um well, my guest uh needs no [145:44] introduction, uh but I will I will do it [145:46] anyways. I I'm Greg Brockman is [145:48] co-founder of OpenAI and uh we're [145:51] thrilled to have him here. Uh Greg, I'd [145:53] actually love to start by talking about [145:55] your uh ever-expanding role at OpenAI. [145:58] Uh you're known as sort of the Godfather [146:00] of the model production process, the [146:02] infrastructure, and talent, and [146:04] everything needed to to power that core [146:06] engine, but uh but I know you've [146:08] recently started taking on a a much [146:10] broader role uh as a founder, and I'd [146:12] love to hear the motivation behind that [146:13] and where your focus is these days. [146:16] >> Well, first of all, thank you for having [146:17] me. It's amazing to see so many people [146:19] who are passionate about Databricks [146:20] about where technology is going in in [146:23] one place. And I at OpenAI, you know, [146:26] I've been involved in basically every [146:28] aspect of of our technology that I can, [146:30] you know, help help with since our very [146:32] early days. And the way that I tend to [146:35] operate is whatever the most important [146:37] pressing problem of the day is, I'll try [146:39] to go and and make a difference there. [146:41] Um, [146:42] there was a time like for GPT-4 where, [146:44] you know, I was up at 2:00 a.m. every [146:46] night like, you know, getting the the [146:47] model back up and running, trying to [146:48] figure out like, you know, deep [146:50] binary search through our nodes to go [146:51] find uh which one was was, you know, [146:53] causing a problem, things like that. [146:56] Build build out the training frameworks. [146:58] And the these days, one of our most [147:01] important problems is that we've made we [147:04] have such a great model production [147:05] process, right? That we really have [147:06] spent a bunch of time over the past [147:08] couple years to I be able to innovate at [147:11] every single layer. And connecting those [147:13] models to the world is actually becoming [147:15] this most important critical challenge. [147:18] And so, I've been spending a lot of time [147:19] really helping the teams focus on [147:21] what do we need to do to bring all this [147:23] amazing research to bear, building the [147:24] products, um having a really focused [147:26] effort to have one way of building [147:29] agents uh that are able to add value to, [147:32] you know, people people in this room. [147:34] >> Yeah, and certainly uh allowing these [147:36] models to do useful work inside of an [147:38] organization is something everyone here [147:39] cares about a lot. And I will get into [147:42] that, but before we get into it, I do [147:43] want to talk quickly about the models [147:45] themselves. [147:46] You've been on such a tear in the last 6 [147:48] months or even really 3 months. [147:50] If I have this right, you know, 3 [147:52] GPT-3.3 came out in February, 3.4 in [147:54] March, sorry, 5.3, 5.4 in March, and 5.5 [147:59] in April. So, it's almost once a month [148:00] and and as a user of those models, I use [148:03] them largely in Codex. Like each each [148:05] model was just a huge leap from the [148:07] prior one. [148:08] I'm curious, what accounts for this like [148:10] really rapid development in model [148:11] quality and and how do you see things [148:13] moving forward? [148:14] >> Well, it's very much the the you know, [148:16] the duck analogy where it's like, you [148:18] know, it all looks very smooth and [148:19] seamless and you know, here is these [148:20] models on a 6-week cadence, that kind of [148:21] thing. Under the hood, it is a lot of [148:24] work. It is a lot of teams coordinating [148:26] together. And the interesting thing for [148:28] me as a technologist, you know, I [148:30] have you know, before OpenAI, I was I [148:32] was working on Stripe and there, you [148:34] know, you sort of see what it's like to [148:35] build a technology company. And you [148:37] know, you might think that building [148:39] these models, it's like totally [148:40] different, but it's actually pretty much [148:42] the same just in a very different domain [148:44] that requires a lot of like really sort [148:46] of [148:47] you know, parsing apart this complicated [148:49] space into a bunch of different layers. [148:51] So, there's pre-training, there's [148:52] post-training, there's a bunch of just [148:54] like dealing with like you have some [148:56] race condition, you got to go find that, [148:57] figuring out how to get the inference [148:58] into good shape. So, there's a lot of [149:01] technological problems that all stack [149:02] together, thinking about the safety, [149:05] how we actually are going to release [149:06] this pricing, packaging, the whole [149:07] thing. [149:08] Um [149:09] and that we basically have a flywheel [149:14] internally that I think it just goes [149:15] faster and faster in terms of just like [149:17] how does every team contribute to each [149:19] of these releases. And so, at a [149:20] technology level, I think that we've [149:22] just made huge strides. Like we were [149:24] kind of you know, 2 years ago, I think [149:26] that releasing a model, even just like [149:27] training a big model was a huge amount [149:29] of pain and that the teams at every [149:32] single layer, infrastructure, the [149:34] research, everyone has come together in [149:35] order to make that process smoother, [149:37] better, and to be able to just work [149:39] together as one to get these models out. [149:41] So, it really is a repeated process [149:42] where we're building a muscle. [149:44] >> Like we at Databricks certainly saw when [149:45] we rolled out 5.5 in our development [149:47] process, there was like huge wins. And [149:50] and I think our engineers certainly saw [149:51] it as like absolutely at the frontier [149:54] around those types of workloads. [149:55] >> Yes. And the the exponential continues, [149:58] right? And that is the wild thing about [149:59] this field, right? It's like every [150:00] single time a model drops, you try it, [150:02] you're like, "Wow, like mind is blown. [150:04] Here's this problem I couldn't do [150:05] before, and now it's easy." And [150:07] you just project that forward, right? [150:09] And it's like every single problem both [150:11] in software engineering, but also now in [150:13] broad knowledge work, right? Any sort of [150:14] work you do with a computer, it's [150:16] starting to be something these models [150:17] can accelerate, right? It's like this [150:19] rocket ship that just moves you forward [150:20] on anything, and the exponential [150:23] it's not going to stop. And I think [150:24] that's something that's worth pricing in [150:25] as people think about how they're [150:27] building their AI strategy. [150:29] >> So, am I hearing that you're going to be [150:31] able to tell us something about the next [150:33] model timing-wise, capabilities-wise? [150:35] How much How much can you We'll keep [150:37] this just between us, you know. Don't [150:38] worry about these guys. [150:39] >> Of course, of course. I mean, look, like [150:41] the the we we are continuing to work [150:44] super hard delivering the next level of [150:46] capabilities. [150:47] I think that the way to think about the [150:49] moment, right? That we're going through [150:50] is something like GPT-4, that that was [150:53] the that was that was a model that was [150:54] great for chat because it was the first [150:56] time that you had this interactivity. It [150:57] was worth reading the output of the [150:59] model when it was something that's very [151:00] crafted for you. Whereas in the current [151:02] moment, we have these models that can [151:04] really do useful work, right? That they [151:05] actually are able to accomplish things, [151:06] they're able to use tools. And so, you [151:08] want those to be brought into your [151:10] workflows, right? Software engineering [151:11] is the first place that I think really [151:13] felt it, right? Of wow, if you actually [151:15] utilize these models, it goes from 20% [151:17] of the work of the you know, sort of [151:19] grind work being done by the models to [151:21] 80%. Suddenly, it changes the tools [151:22] you're going to use. And I think we're [151:24] going through that moment right now for [151:25] workers everywhere, right? For any kind [151:27] of slides, presentation, PowerPoint, [151:29] like all these things um that you do or [151:31] just connecting your data between [151:33] different systems. And all of that is [151:35] just going to get much, much better. And [151:36] so, the way I look at it is like we're [151:38] going to keep having step function [151:39] better models that are more useful to [151:41] you across anything that you need to do [151:43] in your business. Um we're we're [151:45] figuring out how to make these models [151:46] broadly accessible. We think that's a [151:47] part of our mission is to bring these [151:48] models to the world in a safe way. Um [151:51] and that that is something that we're [151:52] doing kind of in partnership with many [151:53] companies and uh and bring bring it to [151:56] bear. So, uh that pace of progress, the [151:58] you know, sort of model release cadence, [152:00] those are all things that we intend to [152:01] keep up. [152:02] >> Yeah, fantastic. So, you know, this [152:03] conference is the Data plus AI [152:05] conference. And so, you have folks that [152:07] are uh data practitioners, folks that [152:09] are focused on AI and and the synergies [152:11] and combinations between them. Uh one [152:13] thing that I think is sort of [152:14] underappreciated about uh OpenAI's work [152:17] is the degree to which you use data in a [152:19] way to inform the improvement of quality [152:21] in models and in your own products. And [152:23] it's been um it's also been a a a [152:25] privilege to be a a part of that. You [152:28] know, Databricks is uh publicly used by [152:30] OpenAI for certain things. And I I love [152:32] seeing when a when you or Sam tweet [152:34] something and I see a graph, I'm [152:36] thinking, "Oh, I know that's a [152:36] Databricks graph." I'm always texting [152:38] you about that. Um but yeah, I'd love to [152:41] understand a bit um [152:42] what role does data play even just for [152:45] your own development of products and [152:46] models at OpenAI. And uh and yeah, any [152:49] anything about also how Databricks or [152:50] other pieces of infrastructure are [152:51] important in that. [152:53] >> Well, fundamentally [152:55] I think that [152:56] data is almost the core ingredient upon [153:00] which [153:01] I don't know if you want to think of our [153:02] models as like a cake or something, [153:03] right? You know, it's like flour. [153:05] Exactly. [153:05] >> I think the data's probably in the [153:06] bottom. [153:07] >> Exactly. It's something like that. Uh [153:08] and that this is true both for how we [153:11] actually produce the models. And the [153:12] amount of return on just like actually [153:14] making sure your data is clean is like [153:15] not to be understated. Um but also more [153:17] broadly, understanding how are people [153:19] using our products? Right? Really [153:21] understanding, you know, chat chat GPT [153:23] is the most broadly deployed AI chatbot. [153:26] And that there's a huge diversity of [153:28] things that people do with it, right? [153:30] And that a lot of what I really try to [153:33] that I really care about is thinking [153:35] about what are the use cases, right? And [153:36] there's a lot that we dig into where [153:38] people are using our models in like [153:40] these surprising and amazing ways we [153:41] never would have anticipated. And [153:43] sometimes, you know, it's applications [153:44] like in healthcare, people using using [153:47] AI to get information [153:48] about, you know, you have a medical [153:49] report that you upload, you get an [153:51] insight from that that you wouldn't have [153:52] gotten any other way. You don't have [153:53] access to a doctor. And just being able [153:54] to find and understand the different [153:57] ways that people are getting value from [153:58] these models is something we spend a lot [153:59] of time thinking about and doing. [154:01] On the agentic side, I think that [154:03] [music] that there just like even just [154:05] really looking through [154:07] um [154:08] just understanding how is an AI actually [154:10] solving a problem? It becomes much [154:12] harder to just read a trace, right? You [154:14] need some better way of looking at these [154:16] things analytically. But the amazing [154:18] thing about where we are is we have this [154:20] technology that's able to sort of [154:22] ingesting information at a much larger [154:23] scale and rate than than humans alone [154:25] could. And so sometimes we can actually [154:27] use our own models in order to [154:28] understand how the models are are [154:30] operating and behaving. And this is true [154:32] on the insight side, but it's also true [154:34] on the safety side, right? For example, [154:36] one of the, you know, sort of most [154:37] painful parts of using Codex has always [154:39] been approvals. And now we have an AI [154:41] layer that's able to do those approvals [154:42] for you. [154:43] >> Yeah, fantastic. Any Databricks feature [154:45] requests? You know, I I can try to make [154:47] it happen if you've got [154:48] >> for us it's always scalable. [154:51] We want it faster. We want we want [154:53] better better better latency. So I think [154:55] it's just in general the big thing we [154:56] want is just like we we have such major [154:59] workloads and everything is scaling and [155:00] and and falling over constantly. And so [155:01] just like the the kind of basics of [155:04] infrastructure reliability is always [155:05] going to be our top one. [155:06] >> Better, faster, stronger. [155:07] >> I mean, the the usual. [155:08] >> [laughter] [155:09] [gasps] [155:10] >> Um uh great. So I'd love to talk also, [155:12] you know, uh Databricks and OpenAI last [155:14] year announced a partnership together. [155:16] Um that partnership was a $100 million [155:18] partnership initially focused on uh [155:20] allowing our customers to have access to [155:22] the GPT family of models to combine with [155:24] their enterprise data. Um, and in you [155:26] know, the last 6 months as as you've uh [155:28] continued to innovate around Codex, [155:30] we've added support for that in our AI [155:32] gateway and and we've also worked [155:33] together on uh allowing Genie uh our AI [155:36] to be used natively from inside of [155:37] Codex. So, lots of exciting stuff [155:39] happening there. [155:40] I'd love to hear how OpenAI thinks about [155:43] first partnerships in general. What what [155:45] roles do partners play as you kind of [155:47] are expanding your presence and then [155:49] anything specifically, you know, we have [155:50] thousands of joint customers. A lot of [155:52] them are in this room. You know, what [155:53] what do you want them to hear about our [155:55] future work together? [155:56] >> Well, number one is that we view what [155:59] we're doing is that we're kind of a a [156:01] spark in the overall engine of of the [156:04] economy of what you know, this AI [156:05] revolution that we're all building [156:06] together. And so, partners are critical [156:08] part of that that we really want to be [156:10] building with the ecosystem. You know, [156:12] there's there's certain things that that [156:14] we're able to do in-house and we'll [156:15] double down on, but I think that the [156:17] only way that we get there is through [156:18] the leverage of what everyone is doing [156:20] and and working together with companies [156:21] like Databricks is absolutely essential [156:22] for us. And so, it's been really amazing [156:24] to see how people are using [156:26] systems like Genie powered by our models [156:28] in all these different contexts. Uh you [156:30] know, people using it internally within [156:32] their own companies. And so, how many [156:33] people in this room have have tried [156:35] Codex? [156:38] So, I think that everyone in this room [156:41] should absolutely try it because it's a [156:44] what we have been evolving what Codex is [156:46] in a very significant way. Like, I think [156:48] that that I First of all, the Codex app [156:51] is something is very differentiated. [156:52] It's something that is worth trying. [156:54] It's like there's nothing like it right [156:55] now. It feels like a very different way [156:56] to use a computer. Um, and and Patrick, [156:58] maybe you could talk about your own use [157:00] of Codex. Like, yeah, what [157:01] >> Yeah, I mean, we've like rolled it out. [157:02] I would consider us early adopters at [157:04] Databricks. And uh and not only are we [157:07] seeing a huge productivity gains for our [157:09] developers, but just the pace at which [157:11] it's improving is remarkable. I mean, I [157:13] come back, you know, either a new model [157:14] release or an update to the desktop [157:16] software and like everything is just [157:18] getting better very, very fast. So, you [157:20] know, we're always encouraging our own [157:21] customers as we do to look at multiple [157:24] AI solutions, AI vendors, kind of [157:27] empower your engineers with the best [157:28] models and tools that they that they can [157:30] find and, you know, certainly it's [157:31] become a very critical part of our of [157:33] our development stack. [157:34] >> Yes. Yeah, and I think it's it's been [157:35] very interesting to watch kind of the [157:36] competitive dynamics where, um, you [157:39] know, I think we were relatively late to [157:40] the game in some ways for building the, [157:43] you know, Codex type form factor, but we [157:45] have focused extremely hard and the rate [157:47] of improvement is so high. And so, we're [157:49] bringing this technology to make it more [157:50] accessible, available to everyone. Um, [157:53] one thing people actually under [157:54] appreciate is Codex itself is open [157:55] source, right? And that that is like in [157:57] our DNA is how do we actually make this [157:59] be something people can hack on and use [158:01] in in all the contexts that they want. [158:02] >> Yeah. So, it sounds like your message is [158:04] try Codex out. And my message is you can [158:07] do it with Databricks. It's super easy, [158:09] especially, you know, if you're already, [158:10] uh, working with Open AI and Databricks [158:11] together. So, uh, so yeah, we'd love to [158:14] to have everyone try it out. Uh, there's [158:15] also many talks at this conference about [158:17] different folks through our partnership [158:19] building cool applications and internal [158:21] tools, uh, with Databricks plus Open AI. [158:23] So, a lot of exciting stuff, uh, for [158:25] folks here. [158:25] >> Yeah, it's never been a better time to [158:27] be a builder. [158:28] >> Yes. [158:28] >> Like I think that [158:30] that to me is the most amazing thing [158:31] about this moment is that creativity is [158:34] being unleashed, that anything you want [158:36] to do, suddenly it's possible. If you [158:38] can imagine it, you can build it. [158:40] >> So, we're almost out of time. I think [158:42] for my last question, I will ask, uh, [158:44] you, you know, earlier this morning Ali [158:46] was talking about AGI and in his mind [158:49] AGI is here in the sense that the types [158:51] of tasks that, you know, our our [158:53] partners, our customers care about are [158:54] almost already perfectly executed by, [158:57] uh, by AIs. Now, AGI means different [158:59] things to different people. What's your [159:01] take on, you know, AGI? Do we have AGI? [159:03] Is it very far away? Is it nearby? Is it [159:05] undefined? You know, how do you Where do [159:07] you sit in that one? [159:08] >> I I I do think it's kind of wild that [159:10] AGI is this like very personal thing. [159:12] It's almost like AGI is a feeling, not a [159:14] defined [clears throat] thing. [159:15] >> It's the friends we had all along. [159:16] >> Exactly. But I do think that there is [159:18] something significant that is worth [159:20] thinking about. And I think maybe one of [159:21] the category errors that that I think [159:23] about is that AGI is almost a spectrum, [159:26] not a not a not a moment, right? Or it's [159:28] like AI progress is not a thing that has [159:30] an end point, right? It's going to [159:31] continue. And I think if the AGI moment [159:33] is something we haven't hit yet. Like [159:35] the way that I think about it is when [159:36] you really do have an [159:39] you know, sort of assistant that is able [159:41] to [159:42] really autonomously go and and work on [159:44] any task for you and it accelerate you [159:46] and and do that at the level of of the [159:48] top humans. And we have this jagged [159:49] intelligence, right? It's something that [159:51] actually really accelerates people in [159:53] certain ways, but it's also like very [159:54] limited and stunted in other ways. And I [159:56] think this is a great opportunity [159:57] because one thing we really stand for at [159:59] OpenAI is keeping humans at the center, [160:01] right? We really want humans to be [160:02] setting the goals, being in control, and [160:04] to make sure that these these systems [160:06] are in benefit of humanity. And so that [160:08] that means that we have this opportunity [160:09] to sort of leverage these systems in [160:11] cases that we really want, but also have [160:13] a lot of time to think about where do we [160:15] want the humans to fundamentally remain [160:17] and to remain the drivers of of [160:18] everything that happens from here. And [160:20] so um I think we're not I think I think [160:22] we're we're not near at the end of this [160:23] exponential. It's going to continue. And [160:25] again, everyone in this room I think has [160:27] something to contribute to to that [160:28] change. [160:29] >> So in a sense, is it fair to say that [160:31] you know, the framing of AGI versus not [160:32] AGI is like not a helpful framing at [160:34] this point or or how do you feel about [160:36] that? [160:36] >> because everyone has a different [160:37] definition. I think it's not the right [160:38] framing. I think that the right framing [160:40] is what are the capabilities, what do we [160:42] want these AIs to be doing, how can we [160:45] benefit from them, how can they help us, [160:47] right? And I think that keeping that at [160:48] the center and thinking about what we're [160:51] building together, to me that is number [160:52] one. [160:53] >> Awesome. Um well, I think our part [160:55] parting recommendation is try out Codex. [160:58] Uh if you're Databricks customer, you [160:59] can use it with the AI coding gateway [161:00] extremely easily. It takes about 1 [161:02] minute to set up. Um I think Ali will [161:05] come on next to uh keep the program [161:06] going, but Greg, thank you so much for [161:08] being here. And uh everyone, please join [161:10] me uh in a round of applause to thank [161:11] Greg. [161:13] >> [applause] [161:16] >> Thank you for having me. [161:17] >> Yep. [161:28] >> All right. Okay. That's awesome. I would [161:31] love partnering with Open AI. They're [161:32] awesome. Uh you should check out Codex. [161:35] Uh they're also using Databricks. Uh you [161:36] know, they have all these, you know, [161:38] almost billion users, and they put all [161:39] that data in Databricks and they [161:41] analyze to make sure that, you know, [161:43] nobody's jailbreaking or nothing you [161:45] know, weird is happening. Um so, let's [161:47] bring us back to the data realm. We have [161:48] the most uh exciting actually [161:50] announcement still left. Uh so, you [161:52] don't want to miss that. Okay, so what [161:54] does the data realm look like? We [161:55] already talked about the right-hand [161:56] side. This is where you're asking your [161:58] analytical questions. Data warehouse [162:00] saying, data science, and then Reynold [162:02] showed us that now we can get super low [162:03] latency, world's fastest engine uh with [162:06] Raiden and Lake House RT. [162:08] Um and then what we did is we went over [162:11] to the OLTP side, and Nikita, who's [162:13] really our expert on OLTP databases, [162:15] started focusing on how can we actually [162:17] get all of that to be fused so that you [162:19] can do everything Postgres on top of the [162:22] lake really fast, really cheap with [162:24] disaster recovery across the clouds. So, [162:27] how do we deal with this onslaught and [162:29] the storm of AI and studies coming our [162:30] way? We still have these waters that we [162:32] have to shuffle things between OLTP and [162:35] OLAP. So, let's welcome back Reynold of [162:38] Raiden fame back to stage. [162:46] Thank you again. [162:50] So, [162:51] you heard from Balau simplify data [162:52] engineering from myself about how we're [162:54] going to unify analytics, and uh Nikita [162:56] from um modernize our OLTP. But, [162:59] honestly, one of the biggest problems [163:01] there's still one big thing remain, [163:03] which is we have the giant continent [163:05] OLTP databases where applications start [163:08] putting their data in first. And if your [163:10] application becomes successful enough, [163:12] you want to reason on that data and [163:13] analyze on your data. And so we end up [163:16] actually building CDC pipelines that [163:18] ship the data from OLTP databases over [163:20] to your analytics continent. And how [163:23] many Can I get a show of hands How many [163:25] of you would love your CDC pipelines? [163:28] All right. Oh, there's some. Like, I [163:29] would love to learn from you how you [163:31] actually maintain them. [163:33] All right, CDC pipelines are basically [163:35] change data capture pipelines. It reads [163:37] the bin log of OLTP databases and it [163:40] gets all the delta little delta changes [163:42] and ships the delta over to analytics [163:44] systems and reconstruct the state of [163:46] OLTP databases in them. They're super [163:48] fragile. They're very annoying to [163:50] maintain. Many data engineers been woken [163:52] up at 3:00 a.m. Um actually because it [163:54] caused the data corruption. As a matter [163:56] of fact, some of people at Databricks [163:58] actually joke about CDC doesn't stand [163:59] for change data capture. It really [164:01] stands for continuous data corruption. [164:04] All right. [164:05] And to solve this problem, the industry [164:08] started introducing um zero CDC, zero [164:11] ETL, mirroring. And all of this are just [164:13] fancy terms of let's do a managed CDC by [164:16] hiding the pipeline so it automatically [164:18] happens. But in practice, there's still [164:20] a pipeline under the hood. It suffers [164:21] from the exactly the same issue CDC has. [164:24] All right. [164:25] So, some of you might have heard of [164:27] another term called HTAP, hybrid [164:29] transactional analytical system um or [164:32] processing. And HTAP is sort of the holy [164:34] grail of database engineering. The idea [164:36] is you create a single database system [164:38] that's capable of handling both of OLTP [164:41] workloads and your analytics workloads [164:43] in one system. [164:44] Um but HTAP has I think most of you [164:47] probably haven't actually used HTAP [164:48] system in practice and the reason is [164:50] HTAP is a category that largely failed. [164:52] There's very little adoption of it, and [164:55] all of the HTAP systems out there are [164:57] kind of very proprietary. They don't [164:58] have a big ecosystem, and by building a [165:01] single system, they started compromising [165:03] on both the performance for OLTP and the [165:05] performance for analytics. [165:08] So, the question is, how do we truly [165:10] unify the two? [165:12] Right? If HTAP doesn't work and CDC is [165:14] very annoying and causes corruption all [165:16] the time, how do we do it? [165:18] And the solution to this actually goes [165:21] back to the very fundamentals problem or [165:24] dichotomy of row-oriented storage and [165:26] column-oriented storage. As Ali and [165:28] Nikita both alluded to earlier, [165:31] um OLTP systems needs row-oriented [165:33] storage because it needs to do a needle [165:34] in a haystack lookup. It needs to [165:36] actually be doing uh updates on the rows [165:38] rows very quickly. Whereas analytics [165:40] system does a lot of scans of large [165:42] amount of data and it really benefit [165:44] from column-oriented storage. [165:46] If you look at architectural diagram [165:48] that Nikita already showed you, where [165:49] I've safe keepers and page servers that [165:51] write data in an actual row-oriented [165:54] format to the data lakes, [165:56] um the solution actually lies right [165:58] there. [165:59] When we looked at and profiled the [166:00] systems of safe keeper and page server, [166:02] those are storage services, it turned [166:04] out they're very underutilized from a [166:07] CPU perspective, but very IO bound. [166:09] Because all they're doing is local disk [166:10] reads, local disk writes, and network [166:12] reads and network writes. [166:14] So, it actually give us the opportunity [166:16] to leverage those idle CPUs on those [166:19] storage services to do a transcoding at [166:22] the very moment to turn the data from [166:24] row-oriented format to a column-oriented [166:27] format. [166:28] Right? And it turned out once you [166:29] convert the data from row-oriented [166:31] format to column-oriented format, [166:34] it doesn't actually increase the CPU [166:36] utilization that much, but actually [166:37] dramatically shrink the data volume [166:39] because column-oriented format has a [166:41] better compression ratio. It's often [166:43] somewhere between 10:1 to 100:1. [166:46] And because of all the services are IO [166:48] bound, by having a better compression [166:50] ratio, we can now actually benefit from [166:53] this transcoding instead of hurting the [166:55] performance. [166:57] And with that, [166:58] we can actually write the OLTP data into [167:02] the data lake in column-oriented format. [167:06] And once they're in the column-oriented [167:07] format, Delta and Iceberg, you can [167:09] actually apply any analytic compute [167:11] directly against that data. [167:13] So, how does this work in practice? [167:15] Imagine inserting a new row into the [167:17] lake base. And the lake base compute [167:19] eventually sends that row to the [167:21] underlying storage system. And the [167:23] storage system will do the transcoding [167:24] right there and actually write that rows [167:27] directly into the columnar format in [167:28] Delta and Iceberg. [167:30] And now you can actually fire up a query [167:32] against your lake house and the lake [167:34] house will read directly the Delta and [167:36] Iceberg format and return your query [167:38] on the freshest copy of the data. [167:42] So, we think this is actually [167:43] revolutionary. probably as at least as [167:45] big as the uh Raven talk I was talking [167:47] about. And the whole concept of lake [167:49] house plus lake base, we can actually [167:50] have unified storage for both of them. [167:53] And we're giving this term a [167:54] uh the storage technology name called [167:56] Altab. And Altab stands for lake [167:58] transactional analytical processing. We [168:00] think Altab is HTAP done right and [168:02] accomplished the goals of HTAP without [168:04] actually having a single query engine [168:06] for it, but we able to actually unify [168:08] the storage, which is by far the most [168:09] important part. And now you have one [168:11] copy of data for your OLTP databases and [168:14] one [168:14] copy of data to govern. [168:18] So, thank [168:20] >> [applause] [168:23] >> Altab can truly unify your data [168:25] infrastructure. You start with the OLTP [168:27] database for applications, but you don't [168:30] have to copy your data. There's no [168:32] pipelines to maintain and most [168:33] importantly, there's actually no [168:34] compromise in performance at all for [168:36] both your OLTP systems and your analytic [168:39] systems. And all of this are built on [168:41] open inter-operable formats. Postgres, [168:43] Delta Lake, Iceberg. [168:46] So, with Altap, we can finally actually [168:49] combine the separate continents of OLTP [168:52] and analytics into a single giant [168:54] continent. [168:57] But, we won't We won't be doing this [168:58] alone. So, we'll be open-sourcing [169:00] actually a very fundamental library that [169:02] allows converting Postgres data directly [169:05] into columnar format in parquet. The [169:06] logo is actually It's kind of a cute [169:09] logo that has a parquet elephant. Um so, [169:11] it indicates what they do. And this [169:13] should be coming uh very soon. [169:16] So, [169:19] >> [applause] [169:22] >> Again, same question. What do you need [169:24] to do differently if you want to [169:25] leverage Altap? Actually, nothing. Just [169:27] start using Lakehouse. [169:30] We'll be rolling out the Altap [169:31] capability in coming weeks. And once [169:33] it's rolled out, every single one of [169:35] your Lakehouse table will automatically [169:37] appear in your Lakehouse. There's [169:39] nothing you need to do. And we can We [169:41] only do that because there's no pipeline [169:42] to maintain under the hood. It's [169:44] available for every single table. All [169:46] right. [169:47] And in order to uh sort of demonstrate [169:49] to you, this is the last demo of the [169:50] day. Um we invite Holly onto stage to [169:53] actually show you a demo of Lakehouse [169:55] combined with Altap. [170:04] >> [music] [170:07] >> I'm going to give you a tour of both [170:09] Lakehouse and Altap and why you need to [170:12] combine your analytical and operational [170:14] data. I have business banks across the [170:17] Americas creating billions of [170:18] transactions a day, and I want an agent [170:21] that spots potential VIP customers um [170:24] and matches them to an advisor based on [170:26] their qualifications and local [170:27] regulations, all in time for teller to [170:30] tell them that information before you've [170:33] left the door. Previously, this would [170:35] have been impossible. Uh the delay would [170:37] have been too slow and the intervention [170:39] would have come too late, but not [170:41] anymore. So, let me show you Databricks [170:44] and let's go ahead and start with Lake [170:46] Base. [170:48] So, Lake Base is open source Postgres [170:50] fully managed by Databricks with [170:52] separated storage and compute and it's [170:54] really easy to get started. I'm going to [170:57] create a new project. I'm going to name [171:00] it and I'm going to stick with the [171:01] defaults and that's it. It's made. [171:04] The compute size isn't fixed, so uh I [171:07] can scale that up and down within the UI [171:09] and set a minimum and maximum size. And [171:12] on the left you can see we've got [171:13] branches here um and I can click in it [171:15] and I can go and create a new branch. Uh [171:17] we can name it. We can have it auto [171:19] delete. We can make a branch off [171:21] production. We can make a branch from a [171:23] point back in time, but we can also [171:25] branch from a branch. And it's created [171:27] almost instantly because there's no need [171:30] to copy data. [171:31] Speaking of back in time, we also have [171:34] backup and restore. This is configured [171:36] to retain 7 days of history by default, [171:39] but you can do more or less if you like [171:41] and pick any time window to restore to. [171:44] Or if you wanted, uh you could restore [171:46] from a snapshot which is scheduled on a [171:49] regular basis. [171:51] Now, all of this is empty. Here's what I [171:53] set up earlier that's processing all of [171:55] our data and we can go into the tables [171:58] and I can see all of my transactional [172:00] data here and I'm going to need a query [172:02] to scan across all of these historic [172:04] balances to calculate who is an outlier [172:07] and potentially a VIP. [172:09] And for those queries we care about some [172:11] metrics. So, first is TPS. So, this is [172:14] the line chart in green that is we're [172:16] going to care about on the right-hand [172:18] side. If this number drops, our business [172:21] is impacted and if it stops, our branch [172:24] staff are going to have an outage. The [172:26] second metric we care about with our [172:27] queries is how long our analytical [172:29] queries take and this is going to be the [172:31] top left number in blue here. And [172:34] finally, how fresh or stale our data is. [172:37] And this is measured in seconds, and it [172:39] will be the top right number in purple. [172:41] Defines how up-to-date our data is. And [172:43] if I'm querying in real time, I better [172:45] get real-time answers. So, earlier [172:48] Reynold said there were three ways of [172:49] running a query like this. And the first [172:51] is what you might an agent might try by [172:53] default, and that's to submit the [172:55] complex analytics query to existing [172:57] lake-based compute. I'm running this [172:59] from the Databricks SQL editor. It's [173:01] possible to connect to lake-based from [173:03] there. And when I'm running this query, [173:05] I can see that this is not quick to come [173:08] back. Uh this is not good of a one-off [173:10] analytics query, but it's terrible for [173:12] our agents who are going to need answers [173:14] a lot faster. And we can also see that [173:17] it's starting to impact our TPS. We can [173:20] see that this green line is going down, [173:22] and it shows a bit of a worrying trend [173:24] because realistically, it's not just [173:26] going to be one dip here, it's going to [173:28] be thousands of them. You can barely do [173:31] analytics like this, uh let alone flag [173:34] in real time who would be a VIP [173:37] customer. [173:42] And it's still going. [173:51] Okay. Um [173:53] well, you will know my name. Uh it looks [173:56] like we have time for everyone to go [173:57] around the room and introduce [173:58] themselves. We'll start at the front, [174:00] maybe. [174:02] And [174:07] Oh, it's done. Okay, finally. Maybe [174:10] we'll do that later. And in blue, we can [174:12] see that this took nearly oh over a [174:14] minute to complete. Uh we can say that [174:17] the staleness is also quite bad as well. [174:20] And we risk taking down production. [174:22] Okay, so that clearly didn't work. So, [174:24] what if instead we took a copy of this [174:26] data using CDC. [174:36] I'm using the lake base change data feed [174:38] feature, which is meant to be for audit [174:40] workloads, not up-to-date analytics. Um [174:44] and with this, I know that there's going [174:45] to be a bit of staleness with at least [174:47] 15 to 30 seconds of data, but this is [174:50] within the Databricks ecosystem. If I [174:52] was using a different Postgres offering [174:54] um or a different CDF this could be into [174:56] multiple minutes, not to mention the [174:59] additional bill and the additional team [175:00] to maintain it. We can see that the TPS [175:03] is flat um [175:05] but it's finished running and it took 13 [175:07] seconds to run, but it was out of date [175:09] by 25 seconds. [175:11] So finally, the ELTaP way of doing this. [175:14] Um with ELTaP, these tables are [175:15] available in Unity Catalog. And here it [175:18] is looking like a managed table, but we [175:20] can see from the icons that yes, it's a [175:22] lake base table, but it's also available [175:25] to both Delta and Iceberg readers. I [175:28] didn't have to set up any additional [175:29] pipelines to make this happen. I just [175:32] specified a catalog and it appeared. And [175:35] you might notice a slight difference in [175:37] our UI here, and that's that we have an [175:38] S3 path. And if I go to this S3 path, [175:43] um I could be able to see all of my [175:45] tables in here. This is great for [175:47] openness. I can query this with open [175:50] source Spark if I wanted to. [175:53] Uh that's a bit of a tangent. Uh let's [175:55] head back to ELTaP. [175:58] So here is our query again, and I'll be [176:00] querying it with a new type of [176:01] warehouse, a real-time warehouse. And so [176:04] now, I click run and it's done. It ran [176:09] in milliseconds. [176:12] Um we can see that this is very [176:14] real-time data without any staleness and [176:16] it had no impact on our TPS. So we can [176:19] throw as many agents at it as we like. [176:22] And because this ran almost [176:23] instantaneously, it also means it's [176:25] significantly cheaper in compute cost, [176:27] too. It is also cheaper to maintain [176:30] because there were no intermediary tools [176:31] to manage. [176:33] For all of us, this is huge. We have a [176:36] single system of record where your data [176:38] can be used for both your operational [176:40] and analytical systems without having to [176:42] compromise on performance. All of this [176:45] happened automatically. I didn't have to [176:48] set up any steps in the background. And [176:50] finally, we no longer a divided view of [176:53] the world between operations and [176:54] analytics. We no longer have to choose. [176:57] Thank you. [177:00] >> [applause] [177:01] [cheering] [177:08] >> All right. [177:09] So, that's it. We were able to unify all [177:12] these different islands, remove all of [177:14] the data movement into just this one big [177:16] Pangea. All right? So, we have the with [177:20] the L L tap or lake tap, you can just, [177:22] you know, untap, get your lake data out. [177:24] The underlying data is just stored in [177:26] Iceberg. Okay? So, it's just open source [177:28] Iceberg Delta. And you can do [177:30] transaction processing, and you can do [177:31] analytics. You can do real-time on it. [177:33] You can do data science. So, that's a [177:35] game changer. Okay. So, just to recap, [177:37] we already unified data engineering and [177:39] data science with Spark. Then we had [177:41] data warehousing with the lake house [177:43] that we unified there. And then we were [177:44] able to with the Raiden engine or lake [177:46] house RT unify all of these into one. [177:49] Uh OLTP databases, and then that got us [177:53] uh ELTP. [177:54] Okay. So, that's what that looks like. [177:56] So, to zoom out and recap today quickly, [178:00] uh we have our agents, but they're [178:01] lacking context. We want to give them [178:03] all that context on the platform. So, we [178:05] did a bunch of announcements today. So, [178:08] uh we introduced the genetic data [178:09] foundation. We had the lake house RT [178:12] that's super super fast. We had the lake [178:14] base with disaster recovery across the [178:16] different clouds, ELTP, Lake Flow with [178:18] all the connectors. So, super exciting. [178:21] Then we went to the context layer. Their [178:23] Genie ontology is extremely important. [178:25] That's your context graph. [178:27] Uh we have the Unity AI gateway for all [178:29] your control, cost governance of all [178:31] your models and agents, [178:33] um open sharing, and then we had [178:35] Omnigent, which you're going to hear [178:37] more about tomorrow. [178:39] And then finally, we have the Gentic [178:41] DevWork. This is all the Genie agents. [178:43] There's three main ones. There is Genie [178:45] 1, [178:46] Genie Code, and Genie Agents. Those are [178:49] the three main ones. Uh and then [178:50] finally, we talked about the apps that [178:52] we're getting into. We're very excited [178:53] about those. Lake Watch for your SIM and [178:55] Customer Lake uh for CDP. So, that's a [178:58] wrap uh for the keynote session today. [179:01] 11:30, we have the sessions. And also [179:03] tomorrow morning, don't miss uh Matei's [179:05] keynote on Omnigent. [179:07] Uh have a great show for you. Thanks for [179:09] coming. [179:11] >> Here we go. [179:12] I'll do my hair. [179:14] I'll get my nails. Baby, how you feel?