DeepMind Scales AI to Power Grids as Zero-Shot Models Transform Ads
Ai4
Automated verification loops and emergent zero-shot architectures are rapidly eliminating the need for custom model fine-tuning across industrial and commercial workflows.
## Background
Published on YouTube by Ai4 on August 28, 2025, this presentation features Praneet Dutta, former Research Engineer Tech Lead at Google DeepMind. According to the page description, the talk covers his group's published research spanning AI weather models for renewable energy forecasting to real-world applications of Generative AI in domains such as advertising.
## TL;DR
Praneet Dutta provides a detailed overview of deploying applied AI across both physical infrastructure and commercial generative media. He explains how hierarchical reinforcement learning in the Industrial Task Suite achieved double-digit HVAC energy reductions and how the GraphCast weather model was fine-tuned to resolve complex mountainous forecasting challenges for Chile's renewable electrical grid. Shifting to generative AI, he explores how automated OCR evaluation loops solved text rendering in Google Imagen and details synthetic data pipelines used for high-fidelity e-commerce product recontextualization. He concludes by highlighting the swift transition from bespoke LoRA fine-tuning to emergent zero-shot capabilities powering enterprise advertising and massive installations at the Las Vegas Sphere.
## Atmosphere & style
The presentation carries a reflective, authoritative, yet approachable conference keynote tone as Dutta shares milestones from his seven-and-a-half-year career at Google DeepMind immediately following his departure. His delivery combines rigorous engineering insight with practical war stories of physical deployments, balancing complex technical diagrams with playful personal anecdotes and prompt tests. It feels like an insider look into the bleeding edge of applied AI, capturing both the gritty challenges of real-world physics and the dizzying pace of modern generative breakthroughs.
## Key takeaways
- [02:03](https://www.youtube.com/watch?v=08PUBSewNkQ&t=123s) Controlling physical industrial facilities like HVAC systems presents massive challenges due to unpredictable human occupancy and multi-scale weather variations.
- [02:03](https://www.youtube.com/watch?v=08PUBSewNkQ&t=123s) DeepMind developed reinforcement learning controllers for industrial HVAC units, achieving double-digit percentage energy savings across full-scale US production facilities.
- [04:41](https://www.youtube.com/watch?v=08PUBSewNkQ&t=281s) To safely test RL agents without risking physical infrastructure damage, DeepMind built the Industrial Task Suite, a specialized thermodynamics simulation benchmark.
- [04:41](https://www.youtube.com/watch?v=08PUBSewNkQ&t=281s) A hierarchical RL architecture—combining a low-frequency manager for chiller cycling with a high-frequency worker for setpoint tuning—prevents mechanical wear-and-tear while optimizing power.
- [08:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=484s) DeepMind's GraphCast architecture uses graph neural networks and multi-mesh message passing to outperform traditional physics-based numerical weather models like ECMWF's HRES on a single TPU.
- [13:10](https://www.youtube.com/watch?v=08PUBSewNkQ&t=790s) Collaborating with Chile's grid operator (CEN), DeepMind applied GraphCast to forecast surface wind vectors to balance steep renewable volatility and solar duck curve dynamics.
- [15:18](https://www.youtube.com/watch?v=08PUBSewNkQ&t=918s) Pre-trained GraphCast initially suffered high error rates in complex mountainous orography like the Andes and Himalayas, requiring localized fine-tuning, spatial priors, and debiasing to beat operational baselines.
- [17:49](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1069s) In commercial advertising generation, text rendering requires zero-tolerance typographic accuracy, which DeepMind solved by integrating automated OCR extraction and verification loops into Imagen training.
- [19:33](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1173s) Product recontextualization for e-commerce was framed as a synthetic data problem, combining image-to-video simulated camera rotations, negative examples, LoRA tuning, and automated vision evaluators.
- [22:46](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1366s) Rapid generative model advancements between 2024 and 2025 are shifting the commercial paradigm from bespoke per-product LoRA fine-tuning toward native zero-shot generation.
- [25:55](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1555s) Generative media has expanded beyond standard digital ads into large-scale immersive physical venues, such as collaborative installations for the Las Vegas Sphere.
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=4s) Introduction and Setting the Agenda
Praneet Dutta opens his morning presentation at Ai4 by welcoming the audience and acknowledging their stamina after three intensive conference days. He introduces his background, noting that as of the previous Friday he had wrapped up his tenure at Google DeepMind to embark on a new adventure after more than seven and a half years working across the applied AI domain. His goal for the session is to share major milestones from his work deploying AI in physical, real-world systems as well as developing generative AI models and practical commercial applications.
Before diving into the core technical material, Praneet shares a LinkedIn QR code and invites attendees—especially those based in the San Francisco Bay Area—to connect for coffee and exchange notes. He delivers the standard industry disclaimer that all remarks reflect his personal views rather than those of any big tech employer and do not constitute legal advice. Setting expectations for the talk, he outlines a roadmap starting with physical AI deployments in industrial facilities, transitioning to generative media—sharing a playful example of an image of a man and his pet on Mars generated with a text-to-image model inspired by the *Postcards from Earth* experience at the Las Vegas Sphere—and concluding with audience Q&A.
[02:03](https://www.youtube.com/watch?v=08PUBSewNkQ&t=123s) HVAC Optimization: Challenges of Real-World Industrial Control
Deploying AI systems to autonomously operate physical industrial infrastructure represents a notoriously difficult challenge. Unlike digital sandboxes, large-scale commercial facilities and data centers are subject to massive, unpredictable variations in human occupancy and thermal payloads. A convention venue like the MGM Grand swings between packed conference days and empty off-peak periods, creating fluctuating internal heat loads. At the same time, external weather patterns introduce multi-scale dynamic shifts, ranging from high-frequency diurnal day-to-night temperature changes to broad seasonal transitions and long-term climate warming trends.
```mermaid
flowchart TD
accTitle: Real-World HVAC Complexity Factors
subgraph Payloads["Dynamic Thermal Payloads"]
A["Human Occupancy<br/>(Conventions vs Off-Peak)"]
B["Hardware Compute Heat<br/>(CPUs & GPUs)"]
end
subgraph Weather["Multi-Scale Weather Drivers"]
C["Diurnal Shifts<br/>(Day / Night Transitions)"]
D["Seasonal & Climate Trends<br/>(Long-term patterns)"]
end
subgraph Controller["DeepMind RL Control Agent"]
E["Industrial HVAC & Air Handlers"]
end
Payloads --> E
Weather --> E
```
To capture substantial headroom in operational expenditure and energy efficiency, the DeepMind team developed reinforcement learning controllers for industrial heating, ventilation, and air conditioning (HVAC) systems and air handling units (AHUs). When deployed across full-scale production facilities in the United States, these RL algorithms achieved double-digit percentage energy savings. Praneet highlights that realizing such efficiencies safely in physical infrastructure requires strict validation methodologies rather than unconstrained live experimentation.
[04:41](https://www.youtube.com/watch?v=08PUBSewNkQ&t=281s) Hierarchical Reinforcement Learning and the Industrial Task Suite
Directly handing control of critical infrastructure to an unconstrained reinforcement learning agent carries the unacceptable risk of physical damage or system failure. To address this, DeepMind developed the Industrial Task Suite—a specialized simulation benchmark analogous to OpenAI Gym but tailored specifically to complex industrial thermodynamics. The suite integrates configurable environment modules, noise generators, domain-specific scenarios, and constraint evaluators, allowing researchers to parameterize the simulation to mimic an airport, a shopping mall, or the MGM Grand at the push of a button before transferring policies to physical hardware.
A major challenge addressed in the simulation suite is chiller plant modeling. Large industrial facilities deploy multiple parallel chillers to handle dynamic cooling loads, but cycling these massive mechanical units rapidly causes severe efficiency penalties and premature mechanical wear-and-tear. To smooth out switching transients while minimizing power consumption, the team implemented hierarchical reinforcement learning:
- **High-Level Manager:** Operates at a low frequency (e.g., hourly), making macro structural decisions such as turning chiller units online or offline as occupancy shifts.
- **Low-Level Worker:** Executes high-frequency closed-loop control (e.g., every 5 minutes), fine-tuning continuous temperature setpoints in response to transient sensor readings.
- **Validation:** Benchmarked in published NeurIPS papers against standard industry baselines and human-designed heuristic controllers, demonstrating superior efficiency, hardware longevity, and operational stability.
[08:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=484s) AI Weather Modeling: Beyond Numerical PDEs to GraphCast
Accurate atmospheric forecasting is critical for optimizing facility cooling, assisting disaster management and wildfire mitigation, informing government policy, and helping agricultural operators manage crop yields. Traditional numerical weather prediction models—such as the high-resolution forecast (HRES) from the European Centre for Medium-Range Weather Forecasts (ECMWF)—rely on solving systems of partial differential equations (PDEs). These numerical methods require thousands of supercomputer nodes, execute only a few times per day, and critically cannot learn directly from the vast archives of historical atmospheric data collected over decades.
```mermaid
flowchart LR
accTitle: GraphCast Architecture Overview
subgraph Input["Global Spatial Input"]
Grid["Discrete 3D Grid Points<br/>(Lat, Lon, Pressure Levels)"]
end
subgraph Processing["GNN Processor"]
MM["Multi-Mesh Message Passing<br/>(7 Spatial Resolutions)"]
end
subgraph Output["Autoregressive Forecast"]
Pred["6-Hour Increment Rollouts<br/>(t=0 up to 240+ Hours)"]
end
Grid --> MM
MM --> Pred
```
To overcome these constraints, DeepMind introduced GraphCast, a machine learning weather forecasting architecture published in *Nature*. Built on graph neural networks (GNNs), GraphCast processes global atmospheric variables across three-dimensional spatial grids (incorporating latitude, longitude, elevation, pressure, temperature, and humidity) using a multi-mesh message-passing architecture spanning seven spatial resolutions. This enables the model to resolve both localized short-term dynamics and continental-scale teleconnections. GraphCast delivers state-of-the-art accuracy across standard benchmarks, runs autoregressive 6-hour interval rollouts up to 240 hours (10 days) near-instantaneously on a single TPU, and accurately predicted the landfall trajectory of Hurricane Maria far closer to ground truth than ECMWF's HRES.
[13:10](https://www.youtube.com/watch?v=08PUBSewNkQ&t=790s) Renewable Energy Forecasting with Coordinador Eléctrico Nacional (CEN)
Applying GraphCast's core predictive capabilities to global decarbonization, Praneet collaborated on an initiative between Google DeepMind, Google X, and Chile's independent grid operator, the Coordinador Eléctrico Nacional (CEN). Chile’s electrical grid has undergone an aggressive transition toward renewable energy, significantly displacing thermal generation with massive solar and wind capacity. This shift introduces steep supply volatility and the classic "duck curve" phenomenon, where daytime solar surges are followed by evening drops that require wind and thermal balancing.
To maintain grid stability without over-relying on carbon-intensive thermal reserves, grid operators require highly precise spatial wind predictions. Praneet highlights GraphCast's performance on 10-meter eastward and northward surface wind components (10U/10V variables in meters per second). Evaluating global forecast rollouts at 12-hour, 2-day, and 10-day lead times demonstrates that GraphCast systematically outperforms traditional numerical baselines across extensive geographic territories, providing grid operators with the actionable foresight needed to maximize renewable dispatch.
[15:18](https://www.youtube.com/watch?v=08PUBSewNkQ&t=918s) Overcoming GraphCast's Orographic Challenges in Chile
While GraphCast achieved landmark success globally, granular evaluation revealed significant regional blind spots. In geographies characterized by complex, mountainous orography—such as the Himalayas across India, Tajikistan, and China, as well as the Andes along Chile's western coastline—GraphCast underperformed baseline numerical weather prediction models like ECMWF's HRES at short lead times (e.g., 12-hour prediction intervals). At longer 10-day lead times, these errors diminished, but the short-term deficiency posed a critical operational issue for electrical grid operators trying to forecast immediate wind energy dispatch.
```mermaid
flowchart LR
accTitle: Chile Weather Fine-Tuning Pipeline
subgraph Input["Input Data"]
A["Sparse Regional Weather & Priors"]
end
subgraph Optimization["Fine-Tuning & Correction"]
B["GraphCast Pre-trained Model"]
C["Chile-Specific Loss & Wind Variables"]
D["Output Debiasing"]
end
subgraph Impact["Downstream Deployment"]
E["Surpasses HRES Baseline"]
F["Chilean Government Renewable Forecasting"]
end
A --> B
B --> C
C --> D
D --> E
E --> F
```
To resolve this limitation, the team applied post-training techniques specifically tailored to Chile's terrain. They fine-tuned GraphCast to focus intensely on wind speeds and strongly correlated meteorological variables, supplied the model with localized spatial priors, and engineered mechanisms to debias its raw predictions. This domain-specific tuning brought the model's error curve comfortably below the HRES benchmark across short-term forecast intervals. The breakthrough culminated in a published paper and a real-world deployment with the Chilean government to power their downstream renewable energy grid management.
- **Orographic Weakness**: Pre-trained global AI weather models struggled with short-term (12-hour) forecasts in rugged terrain, showing pronounced red error clusters along the Chilean coast and the Himalayas.
- **Tuning Focus**: Optimization targeted wind vectors and correlated atmospheric variables alongside regional priors and debiasing layers.
- **Operational Outcome**: Delivered superior short-range accuracy compared to the operational physics-based baseline, directly aiding Chilean grid planning.
---
[17:49](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1069s) Transitioning to Generative Media and Text Rendering in Imagen
Shifting gears to generative media, the speaker reflected on his recent two years at Google DeepMind working on core text-to-image systems, specifically Google Imagen. While large language models and diffusion architectures are reshaping modern creative industries, early image generation models suffered from notorious functional flaws—most notably their inability to render legible, accurate alphanumeric text within generated scenes.
```mermaid
flowchart TD
accTitle: Text Generation Verification Loop
subgraph Generation["Image Synthesis"]
A["Text Prompt"] --> B["Imagen Diffusion Model"]
B --> C["Rendered Image Asset"]
end
subgraph Verification["Automated Evaluation Loop"]
C --> D["Text Extraction / OCR Model"]
D --> E{"Matches Ground Truth?"}
E -- "Yes" --> F["Approved Asset / Downstream Ad"]
E -- "No" --> G["Loss Penalization & Retraining"]
G --> B
end
```
The DeepMind team placed heavy research focus on solving text rendering for commercial viability. In commercial graphics, advertising campaigns, and multi-panel comics, a single misspelled word renders an entire asset unusable regardless of how visually stunning the rest of the image is: "even if it gets like, you know, if you have like 10 words, it gets one word wrong, you can't say that, oh, you know, it's 90% there, that's good. You'll still discard it." By building automated text extraction and OCR-driven verification loops directly into the training process, the team achieved near-perfect typographic generation within complex media formats.
- **Historical Flaw**: Early diffusion models frequently garbled text, producing illegible glyphs rather than readable characters.
- **Zero-Tolerance Standard**: Commercial advertising requires 100% typographic fidelity, where a single character error invalidates an entire generated campaign.
- **Verification Advantage**: Unlike subjective artistic metrics, text accuracy is programmatically verifiable and measurable via automated OCR extractors during model post-training.
---
[19:33](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1173s) Product Recontextualization and the Synthetic Data Pipeline
In e-commerce advertising, brands frequently face a severe asset shortage: they may possess only one or two isolated catalogue photos of a specific SKU in a single colorway. When deploying multi-channel marketing campaigns across platforms like Amazon, brands must place that product into rich, diverse settings—whether placed on an aesthetic backdrop, showcased in dynamic lighting, or held by a model. However, generative tools frequently warped critical product contours, thicknesses, and brand markers, violating strict commercial "product fidelity" standards.
```mermaid
flowchart TD
accTitle: Product Recontextualization Pipeline
subgraph DataAug["Synthetic Data Augmentation"]
A["Sparse Product Image"] --> B["Image-to-Video Camera Rotation"]
A --> C["Subject Masking & In/Outpainting"]
A --> D["Hard Negative Generation"]
end
subgraph Training["Custom Product Tuning"]
B & C & D --> E["LoRA Fine-Tuning Layer"]
end
subgraph Filtering["Automated Quality Gates"]
E --> F["Generated Candidate Assets"]
F --> G["CLIP / DINO / SegCLIP Evaluators"]
G --> H["High-Fidelity Campaign Output"]
end
```
To solve this for advertisers without waiting for foundational base models to evolve, the team framed product recontextualization as a synthetic data problem. Building on DreamBooth and InstructPix2Pix paradigms, they introduced a three-part pipeline:
1. **Synthetic Data Augmentation**: Generating novel viewing angles by feeding a single frame into an image-to-video diffusion model to simulate orbital camera rotations, extracting masks to outpaint varied backgrounds, and deliberately compiling negative examples of competing products to teach the network what *not* to generate.
2. **LoRA Fine-Tuning**: Applying Low-Rank Adaptation (LoRA) weights to specialize diffusion models on specific product geometries.
3. **Automated Evaluation Gates**: Constructing automated evaluator suites utilizing CLIP, DINO, and SegCLIP to measure cross-image consistency, prompt alignment, and semantic segmentation before assets ever reach human advertisers.
---
[22:46](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1366s) Fine-Tuning Benchmarks vs. The Rise of Zero-Shot Capabilities
The resulting synthetic data and LoRA pipeline—published at ICLR—demonstrated substantial empirical gains over standard DreamBooth baselines on internal furniture datasets, dramatically elevating both image-level acceptance rates and overall product pass rates among professional advertisers. While external multi-domain benchmarks yielded more balanced results across automated raters like CLIP, the methodology established a reproducible standard for enterprise-grade product consistency.
Yet the field moved at breakneck speed between 2024 and 2025. What previously demanded multi-stage data augmentation pipelines and per-product fine-tuning began shifting toward native zero-shot generation within modern foundational models. Demonstrating this shift, the speaker shared an informal test conducted the night before the presentation: feeding a single headshot into a modern model with the prompt to depict him piloting a Cessna airplane produced a convincing, zero-shot composition without any dedicated fine-tuning. A subsequent generation attempting to depict him enjoying the AI4 conference introduced minor facial artifacts ("baby fat"), illustrating that while pixel-perfect identity editing remains an active research frontier, emergent zero-shot capabilities are rapidly replacing compute-intensive tuning workflows.
- **ICLR Findings**: Demonstrable lift in commercial pass rates over baseline DreamBooth using synthetic positives, hard negatives, and auto-filtering.
- **Paradigm Shift**: The heavy 2024 requirement for bespoke LoRA fine-tuning per SKU is being rapidly superseded by 2025 zero-shot multi-modal conditioning.
- **Remaining Frontier**: Pixel-perfect facial and product identity preservation under zero-shot conditioning is nearing production quality but still encounters minor distortion artifacts.
---
[25:55](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1555s) Commercial Frontiers: Scale, Verification, and New Creative Mediums
With foundational capabilities like precise typographic rendering, pixel-level manipulation, and high-fidelity recontextualization maturing, generative models are transitioning into scalable commercial production systems. The speaker highlighted public coverage surrounding Google's Imagen releases, noting how verifiable text generation enables programmatic marketing platforms to generate and deploy enterprise ad variants out-of-the-box across Google's advertising ecosystem.
Beyond standard digital interfaces, generative AI pipelines are expanding into monumental physical installations. The speaker referenced recent collaborative efforts by Google teams to power immersive, large-scale entertainment experiences, including generative media created for *The Wizard of Oz* at the Las Vegas Sphere. Closing his presentation, he emphasized that generative media has moved past raw technical feasibility, leaving user imagination as the primary limiting factor for real-world creative applications.
- **Imagen Ecosystem**: Scaled generative infrastructure enables automated creation and verification of ad campaigns directly within enterprise advertising workflows.
- **Immersive Venues**: Emerging collaborations span large-format venue experiences, including generative installations deployed at the Sphere in Las Vegas.
- **Concluding Outlook**: As base models solve structural consistency and verified rendering, the bottleneck has transitioned from model capability to human creative implementation.
## Notable quotes
> [02:03](https://www.youtube.com/watch?v=08PUBSewNkQ&t=123s) Deploying AI systems to autonomously operate physical industrial infrastructure represents a notoriously difficult challenge. — Praneet Dutta
> [04:41](https://www.youtube.com/watch?v=08PUBSewNkQ&t=281s) Directly handing control of critical infrastructure to an unconstrained reinforcement learning agent carries the unacceptable risk of physical damage or system failure. — Praneet Dutta
> [08:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=484s) These numerical methods require thousands of supercomputer nodes, execute only a few times per day, and critically cannot learn directly from the vast archives of historical atmospheric data collected over decades. — Praneet Dutta
> [13:10](https://www.youtube.com/watch?v=08PUBSewNkQ&t=790s) Chile’s electrical grid has undergone an aggressive transition toward renewable energy, significantly displacing thermal generation with massive solar and wind capacity. — Praneet Dutta
> [15:18](https://www.youtube.com/watch?v=08PUBSewNkQ&t=918s) In geographies characterized by complex, mountainous orography—such as the Himalayas across India, Tajikistan, and China, as well as the Andes along Chile's western coastline—GraphCast underperformed baseline numerical weather prediction models like ECMWF's HRES at short lead times — Praneet Dutta
> [17:49](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1069s) even if it gets like, you know, if you have like 10 words, it gets one word wrong, you can't say that, oh, you know, it's 90% there, that's good. You'll still discard it. — Praneet Dutta
> [19:33](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1173s) In e-commerce advertising, brands frequently face a severe asset shortage: they may possess only one or two isolated catalogue photos of a specific SKU in a single colorway. — Praneet Dutta
> [22:46](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1366s) What previously demanded multi-stage data augmentation pipelines and per-product fine-tuning began shifting toward native zero-shot generation within modern foundational models. — Praneet Dutta
> [22:46](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1366s) The heavy 2024 requirement for bespoke LoRA fine-tuning per SKU is being rapidly superseded by 2025 zero-shot multi-modal conditioning. — Praneet Dutta
> [25:55](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1555s) As base models solve structural consistency and verified rendering, the bottleneck has transitioned from model capability to human creative implementation. — Praneet Dutta
## Mentioned
Praneet Dutta, Google DeepMind, Google X, Ai4, Las Vegas Sphere, Postcards from Earth, MGM Grand, Industrial Task Suite, OpenAI Gym, NeurIPS, GraphCast, European Centre for Medium-Range Weather Forecasts (ECMWF), HRES, Hurricane Maria, Coordinador Eléctrico Nacional (CEN), Google Imagen, DreamBooth, InstructPix2Pix, LoRA, CLIP, DINO, SegCLIP, ICLR, The Wizard of Oz
## Video description
Praneet Dutta, Research Engineer Tech Lead, Google DeepMind
Abstracts
In this talk, Praneet will describe his groups published work from AI weather models for renewables forecasting to the applications of Generative AI models for domains such as advertising .
Former Google DeepMind practitioner Praneet Dutta argues that deploying AI into physical infrastructure and commercial generative pipelines requires specialized simulation, domain-specific tuning, and verifiable automated feedback loops rather than unconstrained base models.
Safe RL for Industrial HVAC Infrastructure (2:03)
Operating critical industrial infrastructure presents severe thermodynamic volatility due to unpredictable human occupancy and multiscale weather fluctuations. Direct deployment of unconstrained reinforcement learning risks physical asset damage and premature mechanical wear from rapid equipment cycling. To validate policies before physical transfer, DeepMind developed the Industrial Task Suite, a simulation environment modeling industrial thermodynamics across venues such as airports, shopping malls, and convention centers.
To manage chiller plants safely, the team introduced a hierarchical reinforcement learning architecture. A high-level manager operates at low frequencies, such as hourly intervals, to decide macro actions like switching chillers on or off. A low-level worker operates every five minutes to fine-tune continuous temperature setpoints. Across production facilities in the United States, this architecture achieved double-digit percentage energy savings and improved mechanical longevity compared to standard heuristic baselines.
GraphCast Global Forecasting and Orographic Tuning (8:04)
Traditional numerical weather prediction models like ECMWF's HRES rely on partial differential equations running on thousands of supercomputer nodes, limiting update frequency and ignoring decades of historical atmospheric records. DeepMind's GraphCast architecture uses graph neural networks with multi-mesh message passing across seven spatial resolutions. Running autoregressive 6-hour interval rollouts up to 240 hours near-instantaneously on a single TPU, GraphCast demonstrated state-of-the-art accuracy globally, outperforming HRES on Hurricane Maria's landfall trajectory and delivering superior 10-meter surface wind forecasts across lead times from 12 hours to 10 days.
However, granular evaluations revealed regional deficiencies in complex mountainous orography, including the Himalayas and the Chilean Andes, where GraphCast initially lagged HRES on short-term 12-hour forecasts. Collaborating with Chile's Coordinador Eléctrico Nacional (CEN) to manage renewable energy volatility from solar and wind, the team resolved this through regional fine-tuning. By supplying localized spatial priors, targeting Chile-specific wind loss variables, and implementing output debiasing, the tuned model reduced short-range forecast errors below operational baselines to assist grid dispatch.
Commercial Product Recontextualization and Text Verification (17:49)
Early diffusion models frequently failed to render legible text or preserve commercial product geometry, creating zero-tolerance failures in digital advertising where a single typographical error or altered product contour invalidates an asset. DeepMind integrated automated text extraction and OCR evaluation loops directly into Imagen's post-training, penalizing losses to achieve typographic accuracy suitable for automated enterprise ad variants.
To address SKU asset shortages in e-commerce, the team treated product recontextualization as a synthetic data problem. They created multi-angle variations using image-to-video models to simulate orbital camera sweeps, applied inpainting and outpainting, gathered negative competitive examples, and trained Low-Rank Adaptation (LoRA) weights evaluated by CLIP, DINO, and SegCLIP gates. This synthetic augmentation pipeline, published at ICLR, significantly improved image- and product-level pass rates over standard DreamBooth baselines.
The Transition Toward Native Zero-Shot Capabilities (22:46)
While bespoke data augmentation and LoRA fine-tuning established commercial consistency standards, rapid model improvements between 2024 and 2025 shifted workflows toward native zero-shot generation. Modern multi-modal foundation models can increasingly insert identities into novel contexts—such as placing a single headshot into a flight cockpit—without requiring custom per-product weight adaptations.
Despite this progress, precise identity retention remains an ongoing engineering challenge, as unconstrained zero-shot conditioning can still produce minor facial distortions or geometric artifacts. Nevertheless, as foundational models resolve typographic and structural fidelity, deployment is expanding beyond standard digital interfaces to monumental installations, including generative media created for the Las Vegas Sphere.
Dutta concludes that as base models overcome foundational hurdles in consistency and typographic rendering, the primary operational constraint has shifted from raw algorithmic capability to human creative execution, though pixel-perfect identity preservation without dedicated fine-tuning remains an active research frontier.
## Background
本内容来自频道 Ai4 于 2025 年 8 月 28 日在 YouTube 平台发布的演讲视频《GenAI in the Real World With Google DeepMind》。视频简介指出，前 Google DeepMind 研究工程师技术主管 Praneet Dutta 在本次演讲中分享了其团队已发表的各项前沿成果，涵盖从用于可再生能源预测的 AI 气象模型到生成式 AI 在广告等商业领域的落地应用。
## TL;DR
Praneet Dutta 在演讲中系统回顾了其在 Google DeepMind 期间推动 AI 技术走向物理与商业现实的两大主线。首先，他深入介绍了利用分层强化学习（HRL）实现复杂暖通空调（HVAC）两位数能耗削减，以及研发 GraphCast 气象模型并在智利国家电网落地短期风速预测的工业实践。随后，他剖析了生成式 AI 在电商广告与营销创意中的技术演进，详解了攻克商品保真度（Product Fidelity）的数据增强管线、模型轻量化微调及闭环文字渲染（Text Rendering）技术。最后，他展现了技术从微调迈向零样本（Zero-Shot）生成的趋势，展望了生成式 AI 赋能沉浸式娱乐与广告生态的广阔未来。
## Atmosphere & style
整场演讲呈现出典型硅谷顶尖技术专家严谨、务实又松弛随和的分享风格。Praneet 语言风趣幽默，开场贴心地调侃会议日程并分享个人近况，拉近了与现场观众的距离；在切入技术细节时，他逻辑严密、层层递进，既有对物理控制与图神经网络架构的深度拆解，也有对商业痛点的敏锐洞察。现场穿插的个人生图趣事与行业前沿案例，让整场讲座兼具前瞻性的科技震撼力与接地气的工业落地质感。
## Key takeaways
- [00:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=4s) Praneet Dutta 总结了在 Google DeepMind 的应用 AI 实践经验，演讲主要聚焦于工业实体物理环境控制以及生成式媒体商业落地两大板块。
- [02:03](https://www.youtube.com/watch?v=08PUBSewNkQ&t=123s) 大型建筑与工业暖通空调（HVAC）面临动态人流负载与气象扰动的双重挑战，DeepMind 借助强化学习实现闭环优化并达成两位数百分比的能耗削减。
- [04:12](https://www.youtube.com/watch?v=08PUBSewNkQ&t=252s) 团队开发了 Industrial Task Suite 工业仿真平台以避免实机探索风险，并重点对冷水机组频繁启停导致的能效衰减与机械磨损进行针对性建模。
- [06:28](https://www.youtube.com/watch?v=08PUBSewNkQ&t=388s) 采用分层强化学习（HRL）架构解耦控制逻辑：Manager 负责低频宏观启停决策，Worker 负责高频温度微调，有效平滑了设备运行曲线。
- [09:00](https://www.youtube.com/watch?v=08PUBSewNkQ&t=540s) 传统数值天气预报依赖超算求解偏微分方程且无法持续学习历史数据，DeepMind 基于图神经网络打造的 GraphCast 实现了单块 TPU 上的高效高精度预报。
- [10:25](https://www.youtube.com/watch?v=08PUBSewNkQ&t=625s) GraphCast 引入 7 级多尺度图网格消息传递机制融合局部与全球特征，在飓风“玛丽亚”等重大灾害的路径预测上显著优于欧洲中期预报中心基准。
- [13:10](https://www.youtube.com/watch?v=08PUBSewNkQ&t=790s) 在智利电网可再生能源调度的实际落地中，团队发现原始 GraphCast 在复杂山地地形的 12 小时超短期风速预测上存在性能劣化。
- [16:20](https://www.youtube.com/watch?v=08PUBSewNkQ&t=980s) 通过区域特征加权、关联气象变量联合优化与物理先验去偏后训练微调，大幅提升了山地短期风速预测精度并正式交付智利电力调度中心。
- [18:20](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1100s) 攻克图像中的文字渲染（Text Rendering）是生成式 AI 迈向商业化广告与营销创意的核心前提。
- [19:09](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1149s) 电商营销中的“商品重构上下文”面临商品保真度（Product Fidelity）痛点，基础生成模型极易出现轮廓拉伸与细节变形。
- [20:48](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1248s) DeepMind 构建了结合图像转视频多视角采样、背景 Outpainting 与负样本的数据增强流水线，配合 LoRA 微调与 CLIP/DINO 自动评估器提升生成质量。
- [24:38](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1478s) 生成式技术正迅速从繁琐的单 SKU 微调转向零样本（Zero-Shot）生成，仅凭单张参考图与提示词即可实现主体在全新场景中的高质量重构。
- [25:55](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1555s) 商业广告对文案错误采取一票否决制，团队利用 OCR 文本提取模型构建闭环校验损失，在工业级大模型训练中实现精准的文字排版渲染。
- [27:08](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1628s) 生成式 AI 正从沉浸式场馆娱乐延伸至广告自动化投放等多元产业，其商业与技术落地的未来空间主要取决于人类的想象力。
## Detailed walkthrough
### [00:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=4s) 开场介绍与议程概览
Praneet Dutta 首先向在场的参会者致以早安问候。他在开场时幽默地表示，虽然会议已经进行了漫长充实的三天，但非常感谢大家依然来参加他这场“虽不算大清早但也是上午”的演讲。Praneet 简要介绍了自己的履历：在应用人工智能（Applied AI）领域深耕已超过七年半的时间，参与了诸多极具挑战性的前沿项目；就在上周五，他刚刚结束在 Google DeepMind 的工作，准备踏上职业生涯的下一段全新冒险旅程。他在大屏幕上附上了个人的 LinkedIn 二维码，热忱欢迎现场参会人员在会期内交流，或者会后在旧金山湾区相约喝咖啡探讨技术心得。在现场观众纷纷举起手机拍照时，他还贴心地给出了标准的行业免责声明——本次演讲内容仅代表个人观点，不代表任何大型科技公司的立场，亦不构成法律建议。
随后，Praneet 梳理了整场演讲的时间分配与核心大纲。整场分享预计持续 25 至 30 分钟，最后预留 5 至 10 分钟供现场问答与交流。演讲内容主要分为两大板块：
1. **工业实体环境中的 AI 落地**：分享将强化学习（Reinforcement Learning）和 AI 模型真正部署到复杂工业物理设施中的实践经验与技术突破。
2. **生成式媒体（Generative Media）与商业应用**：探讨生成式 AI 在广告创意等商业领域的落地应用。Praneet 还分享了一个有趣的生活细节——他前一天在拉斯维加斯参观了著名的沉浸式球体剧场 The Sphere 播放的《来自地球的明信片》（*Postcards from Earth*），并深受启发，随即使用文生图（Text-to-Image）模型生成了一幅“一名男子带着宠物在火星漫步”的未来主义图像。
---
### [02:03](https://www.youtube.com/watch?v=08PUBSewNkQ&t=123s) 真实工业场景的 AI 挑战与暖通空调（HVAC）节能控制
演讲的第一项核心内容聚焦于 Google DeepMind 早期的重大突破之一：将 AI 系统实际部署到大型工业设施中进行控制。Praneet 坦言，在现实工业物理系统中部署控制算法是极其困难且充满挑战的，以暖通空调系统（HVAC）和大型建筑温控为例，系统面临着高度复杂的双重扰动源：
- **动态多变的人员负载**：以大型会议中心或酒店大楼为例，人流进出频繁且极不规则。在展会高峰期大楼内部热负荷极高，而在淡季或夜间则人数骤减，系统必须时刻计算并适应这种不可预测的人员热负荷波动。
- **跨尺度的气象系统扰动**：外部天气模式极为复杂，既包含以昼夜交替为代表的高频天气变化（如拉斯维加斯白天的炎热与夜间的降温），也包含跨越季节的中频模式，甚至还伴随全球变暖带来的低频长期气候变化。
为了在满足人类舒适度与设备安全运行的前提下实现节能减排，DeepMind 团队开发了一套基于强化学习（RL）的暖通空调智能控制系统。该系统针对大型工业设施中极其复杂的空气处理机组（Air Handling Unit, AHU）与制冷管网进行闭环优化。在实际部署于全美多处大型工业设施后，该算法成功实现了**两位数百分比的能耗与运营成本削减**，成为强化学习走向现实物理控制的经典范例。
---
### [04:12](https://www.youtube.com/watch?v=08PUBSewNkQ&t=252s) 工业仿真测试平台（Industrial Task Suite）与冷水机组建模
在工业控制中，直接将未经充分验证的 AI Agent 接入大型数据中心或高危物理设施进行实机探索是不可接受的，因为任何错误的控制策略都有可能导致严重的设备损坏、系统过载甚至灾难性后果。因此，高保真度的模拟仿真环境成为了算法开发与假设验证的必经之路。
```mermaid
flowchart TD
accTitle: Industrial Task Suite Simulation Flow
accDescription: Architecture showing how the simulation environment tests AI agents under various industrial conditions before real-world deployment.
subgraph SimulationSuite["工业仿真测试套件 (Industrial Task Suite)"]
Noise["噪声生成器<br/>(Noise Generator)"] --> Env["物理仿真环境<br/>(Environment: 酒店/机场/商场)"]
Scenario["场景调度器<br/>(Scenarios Modifiers)"] --> Env
Constraint["约束评估器<br/>(Constraint Evaluators)"] --> Env
end
subgraph RLControl["强化学习智能体"]
Agent["RL Agent"]
end
Env -- "状态观测 (Observations)" --> Agent
Agent -- "控制动作 (Actions: 设定点/开关)" --> Env
```
DeepMind 专门构建了一套名为 **Industrial Task Suite（工业任务套件）** 的仿真系统，类似于工业领域的 OpenAI Gym。该平台集成了环境建模、噪声生成器（Noise Generator）、场景配置器（Scenarios）以及物理约束评估器（Constraint Evaluators）。工程师只需轻点按钮，即可将待测 Agent 一键置入米高梅大酒店（MGM Grand）、国际机场或超大型购物中心等截然不同的动态物理场景中进行全方位评估与鲁棒性验证。
在具体建模案例中，冷水机组（Chillers / 制冷主机）的负荷调度尤为典型。在会议室坐满参会者时，制冷系统必须全功率全开运转；而在寒冷且无人的夜晚，则可以全部关闭。然而，现实中冷机负载的频繁切换和剧烈升降（Transient load switching）会带来两大严重隐患：
1. **热工能效损失**：冷水机组属于重型机械设备，其升温和降温过程均需要缓冲时间，过于频繁的启停切换会导致能效急剧衰减。
2. **机械疲劳磨损**：频繁切换会加剧昂贵机械部件的磨损，降低设备寿命，大幅增加建筑业主的长期维护与重置成本。
---
### [06:28](https://www.youtube.com/watch?v=08PUBSewNkQ&t=388s) 分层强化学习（HRL）架构与实际应用效果
为了在保持温度精确控制的同时消除不必要的冷机启停震荡，团队提出了基于**分层强化学习（Hierarchical Reinforcement Learning, HRL）**的解题框架。该方案将高维度的复杂控制任务解耦为“顶层管理者（Manager）”与“底层执行者（Worker）”的协同机制：
```mermaid
flowchart LR
accTitle: Hierarchical Reinforcement Learning Architecture
accDescription: Manager-Worker decomposition for HVAC chiller and temperature control.
subgraph MacroLevel["宏观管理层 (Manager - 低频控制)"]
M["Manager Agent<br/>(每小时评估一次)"]
M_Decision["宏观启停决策<br/>(如: 开启/关闭指定冷水机组)"]
M --> M_Decision
end
subgraph MicroLevel["微观执行层 (Worker - 高频控制)"]
W["Worker Agent<br/>(每 5 分钟微调一次)"]
W_Decision["精细设定点调整<br/>(如: 微调出水温度/送风设定值)"]
W --> W_Decision
end
Facility["受控物理建筑设施"] -- "环境传感器读数" --> M
Facility -- "实时温度/湿度反馈" --> W
M_Decision -. "下发策略约束".-> W
W_Decision --> Facility
```
- **Manager（顶层管理者）**：负责低频但影响重大的全局决策（例如以 1 小时为周期进行评估）。当系统整体稳定时保持现状；仅在人流彻底退去或负荷发生阶跃式变化时，下达关闭或启动某台冷水机组的宏观指令。
- **Worker（底层执行者）**：负责高频、精细的局部控制（例如以 5 分钟为周期进行调节）。依据现场人员的实时微观反馈，小幅度微调温度设定点（Temperature Setpoints），确保室内体感始终保持适宜。
这一分层架构不仅彻底消除了冷水机组的频繁骤升骤降，更显著平滑了设备的运行曲线。对比传统的专家启发式规则（Human Heuristic-based Baselines），该算法在各项基准测试中均展现出压倒性的能耗节约优势，相关科研成果也先后发表于 NeurIPS 等顶级人工智能学术会议。
---
### [08:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=484s) 从建筑温控延伸到 AI 气象建模与预测
在总结暖通能耗控制的关键影响因子时，Praneet 指出，影响建筑能耗预测的除了内部热源（如会议室人群、数据中心内全负荷运转散发巨大热量的 CPU 与 GPU）之外，最不可或缺的输入变量便是**外部天气模式**。
如果外部环境处于寒冷干燥的气候，建筑运维系统完全无需启动高耗能的冷水机组进行机械制冷，只需引入室外冷风进行自然通风（Economizer cycle / 外循环冷却）即可满足降温需求。因此，高精度气象预测对于暖通节能、可再生能源调度乃至现代社会的运行都至关重要：
- **日常生活场景**：指导公众何时涂抹防晒霜、何时携带雨具出门。
- **宏观治理与应急响应**：为政府制定环境政策、应对极端野火灾害与突发性自然灾害防御提供关键的决策时间窗口。
由此，Praneet 自然地引申出团队的另一项核心研究主线：借助生成式 AI 与深度学习的强大建模能力，推进现代 AI 气象大模型（AI Weather Modeling）的研发与突破。
### [09:00](https://www.youtube.com/watch?v=08PUBSewNkQ&t=540s) 传统气象预报的局限与 GraphCast 模型的突破
天气预测对于社会具有极其深远的影响，它不仅关系到灾害减缓和数据中心的能耗调度优化，也能帮助农民合理规划降水周期与农作物产量管理。传统的数值天气预报方法主要依赖偏微分方程（PDEs）进行物理建模求解。这些方法结构高度复杂，通常必须在数千个节点的超级计算机集群上高负荷运行，以欧洲中期天气预报中心（ECMWF）的基准高分辨率预报系统 HRES 为例，该模型每天只能运行四次。更关键的瓶颈在于，传统物理模型无法从历史数据中进行持续学习，而人类在过去数十年中其实积累了极其庞大的气象观测数据。这正是引入机器学习来构建更优天气模型的根本创新契机。
DeepMind 团队为此开发了基于图神经网络（GNN）架构的 AI 天气预测模型——GraphCast。该模型不仅登上了《自然》（Nature）期刊，更实现了质的飞跃：它能够在一块 TPU 上近乎瞬时地运行完成，同时在诸多气象预报基准测试中超越了现有最顶尖的系统。
```mermaid
flowchart LR
accTitle: 传统天气模型与 GraphCast 对比
subgraph Traditional["传统数值预报 (如 ECMWF HRES)"]
A["偏微分方程求解"] --> B["数千节点超级计算机"]
B --> C["每日仅运行 4 次 / 无法从历史数据学习"]
end
subgraph GraphCast["Google DeepMind GraphCast"]
D["图神经网络 (GNN)"] --> E["单块 TPU 硬件"]
E --> F["近乎瞬时推理 / 持续学习数十年历史数据"]
end
```
### [10:25](https://www.youtube.com/watch?v=08PUBSewNkQ&t=625s) GraphCast 的多尺度网格架构与飓风路径预测
GraphCast 将地球表面及其上方的大气层离散化为三维空间点，覆盖不同的经纬度以及垂直高度。模型对所有采样点预测各类关键气象变量，包括气压、气温、湿度等。虽然参数量在数千万级别（虽然不及动辄千亿的大语言模型，但在气象领域已属庞大体量），其内部结构却极其精细。模型由编码器、处理器和解码器构成，处理器采用图神经网络的消息传递（Message Passing）机制：以 $t = 0$ 时刻的全球气象状态为输入，按照 6 小时的步长逐步自回归预测未来 6 小时、12 小时、18 小时直至 240 小时（10 天）的天气状态，最终由解码器映射回各时间步的具体物理变量。
```mermaid
flowchart TD
accTitle: GraphCast 预测与多尺度消息传递流程
subgraph Pipeline["端到端架构"]
IN["输入: t=0 气象状态<br/>(气压/气温/湿度等)"] --> ENC["编码器 Encoder"]
ENC --> GNN["GNN 处理器<br/>(7 级多尺度网格消息传递)"]
GNN --> DEC["解码器 Decoder"]
DEC --> OUT["自回归预测<br/>(6h, 12h, 18h... 240h)"]
end
```
为了兼顾局部短时与全球长期的天气演变规律，GraphCast 创新性地引入了“多尺度网格”（Multi-mesh）结构。气象现象具有高度的地理耦合性：某个地点的即时天气主要受周边邻近区域影响，但在中长期尺度下，另一大洲发生的天气系统波动同样会逐步传导并影响全球。GraphCast 包含了 7 种不同分辨率的图网格并在其上执行消息传递，将多尺度特征深度融合。
在应对重大自然灾害方面，GraphCast 展现出了惊人的现实价值。以重创美国沿海的飓风“玛丽亚”（Hurricane Maria）为例，DeepMind 将 HRES 基准与 GraphCast 针对气旋中心（低压区）的移动轨迹预测进行了横向对比：
- 黑色实线代表飓风的真实路径。
- 红色轨迹代表模型预测路径。
- 对比结果显示，GraphCast 的预测轨迹显著更贴近实际飓风路线，而 HRES 的偏差较大。这种更高的准确性可以直接转化为实实在在的社会收益，例如提前开展人口疏散和部署救灾资源。
### [13:10](https://www.youtube.com/watch?v=08PUBSewNkQ&t=790s) 智利国家电网的痛点：山地地形对 AI 短期风速预测的挑战
Praneet Dutta 分享了他更直接参与的合作项目——可再生能源出力预测。该项目由 DeepMind 与 Google X 实验室以及智利国家电力调度中心（Coordinador Eléctrico Nacional，简称 SEN）紧密合作。智利制定了雄心勃勃的电网脱碳目标，希望最大化利用可再生能源。随着煤电等传统火电比例逐年压缩，光伏和风电占比迅猛上升；但光伏发电白天激增、夜间归零导致电网面临经典的“鸭子曲线”（Duck curve），夜间必须高度依赖风力发电与部分火电托底。因此，高精度的风能短期预测成为了电网负荷调度不可或缺的生命线。
```mermaid
flowchart TD
accTitle: 智利电网脱碳与风速预测挑战
subgraph Grid["智利电网现实考量"]
A["脱碳转型: 火电缩减 / 风光激增"] --> B["鸭子曲线: 夜间极端依赖风力"]
B --> C["高精度地面风速预测需求 (10m 高度 U 分量)"]
end
subgraph Paradox["GraphCast 的地形短板"]
C --> D["短预测步长 (12 小时)"]
D --> E["复杂山地地形 (安第斯山脉 / 喜马拉雅)"]
E --> F["AI 预测误差显著高于传统 HRES 基准"]
end
```
电网运营方希望直接利用 GraphCast 预测距地面 10 米高度的纬向风速（10m U-wind component，单位为米/秒）。然而，原始 GraphCast 论文中的对比图表暴露出了一个关键缺陷：
- 在全球性能对比图上，蓝色区域代表 GraphCast 胜过 HRES 基准，红色区域代表 GraphCast 表现更差。
- 在 12 小时这种超短期预报区间，全球复杂山地地形区（如南亚及中亚的喜马拉雅山脉、以及南美西海岸智利沿线的安第斯山脉）呈现出大片刺眼的红色，GraphCast 的表现明显落后于传统物理模型。
- 随着预测步长拉长到 2 天乃至 10 天，这些红色区域才逐渐消退。
这种因复杂山地地形（High Orography）导致的“短期预报性能劣化”，恰好击中了智利电网最看重的核心诉求——因为调度中心最急需的就是极准的短期风电预测。
### [16:20](https://www.youtube.com/watch?v=08PUBSewNkQ&t=980s) 针对区域地形的模型微调与转向生成式 AI 领域
针对短周期在复杂地形预测失准的难题，Dutta 带领团队利用其在模型后训练（Post-training）方面的专业经验进行了深度定制调优。团队通过多维度的针对性微调技术对 GraphCast 进行了优化改造：
- 强制模型重点关注智利本地特定的地理区域特征；
- 引导模型重点优化风速变量以及与之紧密相关的关联气象指标；
- 引入关键的先验物理信息，并对模型的输出进行去偏（Debiasing）处理。
调优后的评估曲线显示，无论是在绝对误差还是归一化误差指标上，经过本地优化与去偏后的 GraphCast 误差全线被压制在 HRES 基准线（图表中的黑色参考线）之下，一举逆转了此前在智利山地短期预测落后的局面。该研究成果不仅形成了学术论文，更最终落地部署交付给智利政府相关机构，直接赋能其下游可再生能源电力调度系统。
分享完气候预报的研究后，Dutta 看了看剩余时间（大约还剩 20 分钟），在台下观众拍摄幻灯片后，自然地将话题切入到其过去两年深入专注的领域——生成式 AI 与生成媒体（Generative Media），并引出了由他与团队深度参与合作打造的核心模型：Google 旗下的文本到图像生成模型 Imagen。
### [18:20](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1100s) 生成式 AI 与图像文字渲染（Text Rendering）的突破
生成式 AI、大语言模型（LLM）与扩散模型（Diffusion Models）正在深度重塑各行各业。Praneet 现场展示了一幅完全由模型通过单条提示词生成的漫画示例。在过去，图像生成模型在渲染文字时往往漏洞百出、拼写混乱，而如今模型已经能够生成带有完美或近乎完美文字排版的漫画作品。
在 Google DeepMind 期间，Praneet 的核心研究方向之一正是攻克图像中的文字渲染（Text Rendering）。文字渲染在商业应用中具有决定性意义：广告创意和营销素材对文字的准确性有着零容忍的要求。看到模型能够呈现清晰无误的文字，极大地增强了团队对商业落地的信心，也为生成式视觉模型开辟了广阔的应用场景。
```mermaid
flowchart TD
accTitle: 图像文字渲染与商品重构上下文的核心商业应用
subgraph CoreTech["核心技术突破"]
T["文字渲染<br/>(Text Rendering)"]
F["商品保真度<br/>(Product Fidelity)"]
end
subgraph Workflow["生成与评估流程"]
P["单图 / 提示词输入"] --> D["扩散模型生成"]
D --> A["自动评估与过滤<br/>(CLIP / DINO)"]
end
subgraph Commerce["商业落地场景"]
M["营销广告与电商素材"]
C["故事漫画与品牌宣传"]
end
CoreTech --> Workflow
Workflow --> Commerce
```
---
### [19:09](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1149s) 电商广告痛点：商品重构上下文与真实度（Product Fidelity）
在生成式 AI 领域，一年前的技术现状与今天相比宛如隔世。Praneet 重点分享了团队在“商品重构上下文”（Product Recontextualization）方面的研究经历。对于品牌方和电商卖家而言，通常某个特定的产品 SKU 只有一到两张单一角度、单调背景的标准白底证件照。当品牌希望在 Amazon 等平台投放广告时，需要将该产品置于各种生动的场景中——例如炫酷的背景光效、由名人和模特手持展示，或融入华丽的营销视觉大片中。
传统实拍成本高昂，而利用生成式 AI 生成营销图面临的最大障碍是“商品保真度”（Product Fidelity）。真实拍摄虽然费时费力但绝不会改变产品外观；相反，生成式模型在二次创作时极易破坏细节，比如改变手机边框的厚度、拉伸轮廓或歪曲外观特征，而这对品牌方而言是不可接受的致命伤。当时现有的开源基线模型（如 DreamBooth 和 InstructPix2Pix）在保真度上远远达不到广告主的严格标准。
- **核心商业诉求**：从仅有的 1-2 张单色商品照生成多场景、多光影的高质量广告图。
- **保真度难点**：AI 生成常导致商品轮廓走样、厚度变形或细节丢失，直接阻碍广告转化。
- **现有方案缺陷**：DreamBooth 与 InstructPix2Pix 无法满足广告主对于工业级精度的严格门槛。
---
### [20:48](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1248s) DeepMind 的解决方案：合成数据增强、LoRA 微调与自动评估器
面对基础模型保真度不足的问题，DeepMind 团队并未等待底层大模型的自然迭代，而是将其定义为一个短期的数据工程难题。DreamBooth 这类微调技术极度依赖大量人工清洗的数据，团队索性将问题反转：“何不通过合成手段自动构建大量高保真数据，明确教会模型什么是这个产品、什么不是这个产品？”
团队由此构建了一套可扩展的数据增强与微调管线，并在 ICLR 发表了相关研究论文。整个系统主要依托三大支柱：
1. **多模态数据增强管线**：利用图像到视频（Image-to-Video）扩散模型，让单张商品图旋转生成多视角并截取帧；使用产品 Mask 配合 Outpainting 算法自动替换多样化背景；同时大量引入负样本（Negative Examples），让模型明确识别非目标商品，学会“不该生成什么”。
2. **基于 LoRA 的轻量化微调**：对每个特定商品采用低秩适应（Low-Rank Adaptation, LoRA）进行专属高效微调。
3. **多维度自动评估器（Auto-raters）**：引入 CLIP、DINO 和 SegCLIP 等自监督与语义评估模型，对生成的商品图像进行图文一致性、跨样本一致性与语义保真度层层质检，在最终呈献给广告主之前完成自动化初筛。
```mermaid
flowchart LR
accTitle: DeepMind 商品重构上下文数据增强与微调架构
subgraph DataAug["数据增强流水线"]
Raw["稀疏商品照 (1-2张)"] --> I2V["Image-to-Video<br/>多视角旋转截图"]
Raw --> Out["Mask + Outpainting<br/>背景多样化替换"]
Raw --> Neg["负样本生成<br/>(明确排除特征)"]
end
subgraph Train["模型微调"]
I2V & Out & Neg --> LoRA["LoRA 轻量化微调<br/>(每商品定制)"]
end
subgraph Quality["质检与交付"]
LoRA --> Gen["扩散模型生成候选图"]
Gen --> Auto["自动评估器过滤<br/>(CLIP / DINO / SegCLIP)"]
Auto --> Pass["交付广告主"]
end
```
---
### [23:07](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1387s) 实验评估结果与广告主采纳标准
在内部家具数据集的评估实验中，DeepMind 提出的方法相比传统的 DreamBooth 基线展现出了压倒性优势。团队评估了两个核心维度：图像级别的通过率（Image-level Pass Rate）以及综合产品级别的通过率（Product Pass Rate），即广告主在严苛标准下认可并愿意采纳的生成图像比例。
在外部公开数据集的评测中，尽管在个别自动评估指标（如特定 CLIP 分数）上略有起伏，但整体表现依然保持了极高水准并具备显著竞争力。该项研究成果在 ICLR 上发布后引发了广泛反响。然而，这套在 2024 年堪称前沿的方案也带来了一个现实困惑：广告主是否必须为每一个商品投入繁重的微调计算成本？
- **内部家具数据集**：在图像通过率与整体产品通过率上大幅超越 DreamBooth 基线。
- **外部数据集验证**：在多项自动化质检指标上维持了极具竞争力的均衡表现。
- **行业反馈与痛点**：高保真效果虽获认可，但每个 SKU 均需微调的成本结构仍显沉重。
---
### [24:38](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1478s) 迈向 2025：从微调走向零样本（Zero-Shot）生成与面部重构挑战
技术演进的速度颠覆了去年的工作模式。进入 2025 年后，生成式 AI 的研究重点转向了模型的涌现能力（Emergent Capabilities）与零样本（Zero-Shot）泛化。如今，用户只需向基础大模型提供单张参考图与一段提示词，模型即可直接在全新的上下文中生成目标主体，完全省去了繁琐的 LoRA 微调与数据增强流水线。
Praneet 现场展示了自己在演讲前一天晚上做的趣味测试：他将自己的一张个人头像喂给模型，并输入提示词要求生成“置身于全新场景中开飞机”的画面。模型在未经任何微调的情况下，直接生成了一张他坐在塞斯纳（Cessna）单引擎飞机驾驶舱内的高质量图像，整体神态还原度惊人。随后他又尝试生成了自己参加 AI4 大会的画面，画面虽好，但模型却擅自给他的面部加上了一些“婴儿肥”（Baby fat）。这表明像素级精准编辑（Pixel-perfect editing）依然存在挑战，但零样本泛化能力的发展速度已经超出了所有人的预期。
> "This is the only context a model gets, right? Just a photo of me, a single prompt, and puts me in a Cessna plane. Pretty good. No fine-tuning, nothing else required."
> —— Praneet Dutta
---
### [25:55](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1555s) 工业级文字渲染的商业化落地与未来展望
Praneet 深入探讨了他在 Google 参与的工业级图像生成系统（如在 Google I/O 上公布的 Imagen 系列图像生成模型）中的核心攻坚——高精度文字渲染。在真实的商业广告场景中，广告主对文字错误采取“一票否决制”：如果广告文案总共 10 个单词，哪怕模型拼对了 9 个只错漏了 1 个，整张素材也必须直接报废，不存在所谓的“90% 达标”。
文字渲染的优势在于其高度的可验证性与可量化性。在训练流程中，团队利用成熟的文本识别与提取模型对生成的图像进行 OCR 校验，与输入的 Ground Truth 进行比对形成闭环损失，进而规模化地反哺扩散模型训练。Praneet 总结道，随着文本准确渲染、像素级保真编辑以及零样本场景重构等底层基建的成熟，AI 视觉生成的商业化版图正在以前所未有的速度展开。
- **商业广告零容忍**：文字排版要求 100% 准确，单字拼写错误即导致整图作废。
- **闭环自动化训练**：利用文字提取模型构建自动化 Ground Truth 校验管线，在 Google 级别的大规模集群上持续强化训练。
- **技术成熟矩阵**：文字精准渲染、商品像素级编辑与零样本重构正在共同构建商业生成式 AI 的核心底座。
### [27:08](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1628s) 生成式 AI 的落地应用与未来展望
在展示了多项前沿研究后，Praneet Dutta 进一步探讨了生成式 AI 在现实世界各个领域的实际落地案例。他提到了 Google 团队与活动组织方在拉斯维加斯 Sphere 场馆打造的《绿野仙踪》（Wizard of Oz）沉浸式体验项目，其中便深度融入了生成式 AI 技术。此外，在广告营销这一 Praneet 在 Google 亲自深耕的业务领域，生成式 AI 的微调模型（Fine-tuned models）正在发挥关键作用。借助这些针对特定业务微调的模型，广告主能够实现“开箱即用”（out of the box），轻松在 Google 的广告系统乃至更广泛的生态中快速构建并部署营销活动。
Praneet 总结道，无论是大型沉浸式娱乐还是高度商业化的广告投放，都仅仅是生成式 AI 无限潜力的冰山一角。他直言目前技术的边界主要受限于人类自身的想象力（"at this point we're quite limited by imagination"），并对这项技术未来将把人类带向何方表达了极其充沛的期待与兴奋。演讲最后，Praneet 再次展示了自己的 LinkedIn 二维码，感谢主办方与现场听众，并在全场掌声中宣布进入问答环节（Q&A）。
- **Sphere 沉浸式项目**：Google 团队与活动主办方合作，将生成式 AI 技术应用于 Sphere 的《绿野仙踪》（Wizard of Oz）互动体验中。
- **广告业务赋能**：通过专为广告场景微调的 GenAI 模型，帮助广告客户实现开箱即用的自动化广告活动（Ad Campaigns）配置与高效上线部署。
- **技术未来愿景**：生成式 AI 的应用前景广阔，当前的探索仍处于起步阶段，未来的可能性将取决于人们如何发挥想象力去定义和构建实际应用。
## Notable quotes
> [00:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=4s) 本次演讲内容仅代表个人观点，不代表任何大型科技公司的立场，亦不构成法律建议。 — Praneet Dutta
> [04:12](https://www.youtube.com/watch?v=08PUBSewNkQ&t=252s) 在工业控制中，直接将未经充分验证的 AI Agent 接入大型数据中心或高危物理设施进行实机探索是不可接受的，因为任何错误的控制策略都有可能导致严重的设备损坏、系统过载甚至灾难性后果。 — Praneet Dutta
> [08:04](https://www.youtube.com/watch?v=08PUBSewNkQ&t=484s) 如果外部环境处于寒冷干燥的气候，建筑运维系统完全无需启动高耗能的冷水机组进行机械制冷，只需引入室外冷风进行自然通风即可满足降温需求。 — Praneet Dutta
> [09:00](https://www.youtube.com/watch?v=08PUBSewNkQ&t=540s) 更关键的瓶颈在于，传统物理模型无法从历史数据中进行持续学习，而人类在过去数十年中其实积累了极其庞大的气象观测数据。这正是引入机器学习来构建更优天气模型的根本创新契机。 — Praneet Dutta
> [19:09](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1149s) 传统实拍虽然费时费力但绝不会改变产品外观；相反，生成式模型在二次创作时极易破坏细节，比如改变手机边框的厚度、拉伸轮廓或歪曲外观特征，而这对品牌方而言是不可接受的致命伤。 — Praneet Dutta
> [20:48](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1248s) 何不通过合成手段自动构建大量高保真数据，明确教会模型什么是这个产品、什么不是这个产品？ — Praneet Dutta
> [24:38](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1478s) This is the only context a model gets, right? Just a photo of me, a single prompt, and puts me in a Cessna plane. Pretty good. No fine-tuning, nothing else required. — Praneet Dutta
> [25:55](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1555s) 如果广告文案总共 10 个单词，哪怕模型拼对了 9 个只错漏了 1 个，整张素材也必须直接报废，不存在所谓的“90% 达标”。 — Praneet Dutta
> [27:08](https://www.youtube.com/watch?v=08PUBSewNkQ&t=1628s) at this point we're quite limited by imagination — Praneet Dutta
## Mentioned
Praneet Dutta, Google DeepMind, Google, Google X, Ai4, The Sphere (拉斯维加斯沉浸式球体剧场), 《来自地球的明信片》（Postcards from Earth）, 《绿野仙踪》（Wizard of Oz）, Industrial Task Suite, MGM Grand (米高梅大酒店), GraphCast, ECMWF (欧洲中期天气预报中心), HRES, Nature 期刊, 智利国家电力调度中心 (SEN / Coordinador Eléctrico Nacional), Imagen, DreamBooth, InstructPix2Pix, LoRA, CLIP, DINO, SegCLIP, ICLR, NeurIPS, Cessna (塞斯纳单引擎飞机), Amazon
## Video description
Praneet Dutta, Research Engineer Tech Lead, Google DeepMind
Abstracts
In this talk, Praneet will describe his groups published work from AI weather models for renewables forecasting to the applications of Generative AI models for domains such as advertising .
[Source](https://www.youtube.com/watch?v=08PUBSewNkQ)
Source quotes
[0:04] Praneet Dutta: 本次演讲内容仅代表个人观点，不代表任何大型科技公司的立场，亦不构成法律建议。
[4:12] Praneet Dutta: 在工业控制中，直接将未经充分验证的 AI Agent 接入大型数据中心或高危物理设施进行实机探索是不可接受的，因为任何错误的控制策略都有可能导致严重的设备损坏、系统过载甚至灾难性后果。
[8:04] Praneet Dutta: 如果外部环境处于寒冷干燥的气候，建筑运维系统完全无需启动高耗能的冷水机组进行机械制冷，只需引入室外冷风进行自然通风即可满足降温需求。
[9:00] Praneet Dutta: 更关键的瓶颈在于，传统物理模型无法从历史数据中进行持续学习，而人类在过去数十年中其实积累了极其庞大的气象观测数据。这正是引入机器学习来构建更优天气模型的根本创新契机。
[19:09] Praneet Dutta: 传统实拍虽然费时费力但绝不会改变产品外观；相反，生成式模型在二次创作时极易破坏细节，比如改变手机边框的厚度、拉伸轮廓或歪曲外观特征，而这对品牌方而言是不可接受的致命伤。
[20:48] Praneet Dutta: 何不通过合成手段自动构建大量高保真数据，明确教会模型什么是这个产品、什么不是这个产品？
[24:38] Praneet Dutta: This is the only context a model gets, right? Just a photo of me, a single prompt, and puts me in a Cessna plane. Pretty good. No fine-tuning, nothing else required.
[25:55] Praneet Dutta: 如果广告文案总共 10 个单词，哪怕模型拼对了 9 个只错漏了 1 个，整张素材也必须直接报废，不存在所谓的“90% 达标”。
[27:08] Praneet Dutta: at this point we're quite limited by imagination
Source quotes
[2:03] Praneet Dutta: Deploying AI systems to autonomously operate physical industrial infrastructure represents a notoriously difficult challenge.
[4:41] Praneet Dutta: Directly handing control of critical infrastructure to an unconstrained reinforcement learning agent carries the unacceptable risk of physical damage or system failure.
[8:04] Praneet Dutta: These numerical methods require thousands of supercomputer nodes, execute only a few times per day, and critically cannot learn directly from the vast archives of historical atmospheric data collected over decades.
[13:10] Praneet Dutta: Chile’s electrical grid has undergone an aggressive transition toward renewable energy, significantly displacing thermal generation with massive solar and wind capacity.
[15:18] Praneet Dutta: In geographies characterized by complex, mountainous orography—such as the Himalayas across India, Tajikistan, and China, as well as the Andes along Chile's western coastline—GraphCast underperformed baseline numerical weather prediction models like ECMWF's HRES at short lead times
[17:49] Praneet Dutta: even if it gets like, you know, if you have like 10 words, it gets one word wrong, you can't say that, oh, you know, it's 90% there, that's good. You'll still discard it.
[19:33] Praneet Dutta: In e-commerce advertising, brands frequently face a severe asset shortage: they may possess only one or two isolated catalogue photos of a specific SKU in a single colorway.
[22:46] Praneet Dutta: What previously demanded multi-stage data augmentation pipelines and per-product fine-tuning began shifting toward native zero-shot generation within modern foundational models.
[22:46] Praneet Dutta: The heavy 2024 requirement for bespoke LoRA fine-tuning per SKU is being rapidly superseded by 2025 zero-shot multi-modal conditioning.
[25:55] Praneet Dutta: As base models solve structural consistency and verified rendering, the bottleneck has transitioned from model capability to human creative implementation.
Original transcript
[0:04] Good morning everyone. Um it's great to
[0:06] be here at EI4. I know it's been three
[0:08] long days. So thank you for joining in
[0:09] for my uh I won't say early morning but
[0:12] morning talk. Really appreciate that. Um
[0:14] so quick introduction about me. Um I'm
[0:16] Prit. Um well as of last Friday I was
[0:20] with Google Deep Mind. I decided to
[0:22] embark on a new adventure in my career.
[0:24] But I've been in the applied AI space
[0:26] for more than 7 and a half years working
[0:29] on a bunch of interesting projects
[0:30] according to me. And I'm really excited
[0:32] to be sharing a few of the milestones
[0:35] that we've achieved both with AI in the
[0:37] real world as well as with generative AI
[0:39] models and their applications. Right. Um
[0:42] I have a tiny QR code here uh for my
[0:44] LinkedIn. Um I would love to meet as
[0:47] many of you as possible during the
[0:48] conference. But if not also happy to
[0:50] connect externally and share notes on
[0:52] what I've been doing. Right. Oh, I'm
[0:53] based in San Francisco Bay Area as well.
[0:55] So any of you are from the region also
[0:57] happy to grab coffee while in the space.
[1:01] Yeah, I see people taking photos. I'll
[1:02] wait a second. Okay. Um so mandary
[1:05] disclaimer while you grab your snaps. Um
[1:07] everything I'm speaking is my personal
[1:09] opinion. None of it represents the view
[1:11] of any big tech company and is not legal
[1:13] advice. So standard disclaimer there.
[1:17] Cool. So um we have like roughly less
[1:20] than 35 minutes. So I hope to spend
[1:22] around 25 to 30 minutes um covering some
[1:25] of my material. Um I'm going to be first
[1:27] talking about some of exciting use cases
[1:29] of deploying AI models in the real world
[1:32] and then speak more about um how we use
[1:34] generative media. Um have most of you
[1:37] seen the sphere the postcards from
[1:39] Earth, right? Um I was actually inspired
[1:41] by that and I used the text image model
[1:43] to create like a futuristic looking
[1:45] image of a man and his pet on the on
[1:47] Mars. So not bad. So um yeah I hope to
[1:50] kind of cover some of working generative
[1:51] media as well as some commercial
[1:53] applications which you've had with that
[1:55] and then hopefully have 5 10 minutes at
[1:57] the end for questions but if not I'm
[1:58] also free after my talk if anybody wants
[2:00] to exchange notes.
[2:03] Cool. Um so I'll first start with some
[2:06] of the early work we did. Um and I think
[2:08] it's quite exciting because it's one of
[2:09] the few use cases where we have actually
[2:11] deployed AI systems in industrial
[2:13] facilities which is quite hard. Um when
[2:16] you think about like deploying AI models
[2:18] in the real world like controlling HVAC
[2:20] systems or better models it's it's quite
[2:22] challenging in its nature for a couple
[2:24] of factors. Two primary reasons is like
[2:27] when you think of a building like this
[2:28] right people are coming in and out. It's
[2:30] maybe busy during peak convection
[2:33] convention months but probably less busy
[2:35] during other months. So a lot of human
[2:37] load that needs to be factored in,
[2:39] right? Um but also weather systems as
[2:42] you can imagine like we have seasons.
[2:44] It's quite warm here in Vegas. It gets
[2:45] slightly warmer as we cross
[2:47] Thanksgiving. So we have both. Can you
[2:50] hear me? Okay. Yeah. We have both high
[2:52] frequency like day and nightly weather
[2:54] patterns as well as you know seasonal
[2:55] patterns and even lower frequency
[2:57] long-term patterns. Global warming is a
[2:59] thing. So, you know, modeling all this
[3:01] complexity with human movement in
[3:03] buildings and, you know, not just
[3:04] humans, but different sort of payloads
[3:07] as well as like weather patterns makes
[3:09] this an especially challenging field to
[3:10] be kind of deploying real world systems
[3:13] to accurately um control certain tasks.
[3:16] So, I want to give a quick summary of
[3:18] some of our work. This was a few years
[3:20] ago, but I think the principles are
[3:21] quite interesting to share. Um, where we
[3:24] designed a reinforcement learning system
[3:26] to control the HWAC system. So, as you
[3:28] notice, we have, you know, this lovely
[3:30] air conditioner here, an air handling
[3:32] unit. There's a lot of complexity behind
[3:34] the scenes that, you know, go into
[3:36] keeping a building nice and cool, so
[3:38] everyone's happy, right? Um, and these
[3:41] systems, as you can imagine, they're
[3:43] quite um there's a lot of headroom for
[3:45] improvement in terms of both cost as
[3:46] well as energy saving. So that's
[3:48] something which our team at DeepMind
[3:49] focused on working a few years ago on
[3:51] how could we deploy reinforcement
[3:53] learning systems to actually control the
[3:56] cooling systems in large industrial
[3:57] buildings such as facilities. Um for
[4:00] this specific case we achieved like a
[4:02] double-digit percentage saving in large
[4:04] facilities in the United States and um
[4:06] it's a quite exciting project. I'd like
[4:08] to share that how we use reinforcement
[4:10] learning in the real world.
[4:12] Um so specifically what we did for this
[4:14] is as you can imagine um when you think
[4:16] of a data center large building you
[4:18] can't just give an AI agent like the
[4:19] controls and say go ahead you know make
[4:21] use of it cuz you can imagine
[4:22] catastrophically something could
[4:24] potentially blow up or go wrong right so
[4:27] um for systems such as these um
[4:30] simulation is a great way to kind of
[4:32] test out AI agents see how well one
[4:34] performs versus the other and then once
[4:36] you have some sort of hypothesis baked
[4:38] in use that insight to build it into the
[4:40] real
[4:41] So at DeepMind we built something called
[4:42] an industrial task suite. Think of it of
[4:45] something like the open AI gym but very
[4:47] specific for industrial systems. So in
[4:49] this case you see this in green. We have
[4:51] an environment and we have stuff like
[4:52] noise generator scenarios constraint
[4:55] evaluators. All of these are different
[4:57] knobs and parameters that we can tweak.
[4:59] So we can have a simulation acting like
[5:01] the MGM Grand or an airport or a large
[5:04] shopping mall and we can easily with
[5:06] just a click of a button have an agent
[5:08] being evaluated in completely different
[5:10] context. So this is something really
[5:12] cool we built out. Um one specific case
[5:15] that we were able to I like sharing this
[5:17] example is uh chiller modeling. So if
[5:20] you think of chillers as air
[5:22] conditioners right in this case you see
[5:24] three chillers here. Um air conditioners
[5:27] have different load. So in the example
[5:29] of right now, there's so many of us in
[5:30] the room. I'm sure the air conditioner
[5:32] running on at full blast, right? But
[5:35] let's say it's like a cool night and no
[5:36] one's in the room. You probably have all
[5:38] of them turned off, right? So situations
[5:42] are not static. They're quite dynamic
[5:43] and they keep changing with time. And as
[5:45] you can see in the left, typically if
[5:48] you kind of change the number of
[5:49] chillers you're running or change the
[5:50] load amongst chillers, it's quite fast
[5:52] and transient. Um, and we want to make
[5:54] them more smooth. Why do we want to do
[5:57] that? um for two primary reasons. Um
[5:59] these are mechanical devices, so they
[6:02] take time to ramp up and down. So the
[6:04] faster you switch amongst them, they're
[6:05] more inefficiencies. Um but they're also
[6:08] mechanical devices. So you know uh
[6:10] there's a lot of wear and tear. So you
[6:11] also want to be mindful of that. And
[6:12] there's also the costsaving angle for um
[6:15] you know building operators in terms of
[6:16] how long you want to prolong your
[6:17] equipment. So um we kind of used our
[6:20] simulation suite to kind of see how we
[6:22] can take care of this sort of chiller um
[6:26] switching right um we wanted to verify a
[6:28] hypothesis that could be use something
[6:30] called hierarchical reinforcement
[6:31] learning that's basically a technique
[6:33] where you know you kind of decompose the
[6:35] problem into a top level manager worker
[6:37] architecture where the manager takes
[6:39] some of the higher level low frequency
[6:41] control objectives such as should I turn
[6:43] a chiller off or not which is a bigger
[6:45] decision and the worker kind of does the
[6:47] fine grain adjustments such as changing
[6:49] the temperature, making sure everything
[6:51] else is okay and attacks more
[6:52] frequently.
[6:54] So yeah, this is kind of like the layout
[6:55] of what we did. So you know you we in
[6:57] the simulation we kind of simulated a
[6:59] facility such as this. Um we kept seeing
[7:02] observations from the facility, how well
[7:04] the sensors, how well the building's
[7:05] operating, are the sensors all okay the
[7:07] manager says that okay you know I won't
[7:09] act for an hour everything looks good
[7:11] but the low-level worker it does the
[7:13] fine grain temperature set point. So
[7:15] let's say that we all say it's getting
[7:16] too warm in here. It does fine grained
[7:18] temperature set changes every 5 minutes
[7:20] and then maybe at the end of the day
[7:22] when we all leave the manager says now
[7:23] I'll act and now I'll make um I'll maybe
[7:25] turn all the chillers off right um so
[7:28] for the purpose of today's presentation
[7:29] I don't have the full results but we
[7:31] were able to verify this algorithm
[7:33] versus many baselines such as human
[7:35] heristic based algorithms and we able to
[7:37] see how you know better performed
[7:40] so I for this project I'll pause right
[7:42] here but you know if you're interested
[7:44] we had like a bunch of different papers
[7:45] at new rips and other venues um on what
[7:47] we did. So if you're in the energy space
[7:49] or just generally interested in
[7:51] reinforcement learning in the real world
[7:52] um happy to share um happy to share more
[7:54] context after the talk or you can read
[7:56] these papers. Let's do a quick time
[7:58] check. Perfect. Okay. Um cool. So the
[8:04] second project I want to chat a bit more
[8:06] about was if you remember in the data
[8:08] center cooling project what I mentioned
[8:10] is two major factors that kind of
[8:12] influence how much um load or how much
[8:15] energy we need for cooling is humans or
[8:17] different objects that you know like
[8:19] data centers where you have CPUs and
[8:20] GPUs emitting heat but also weather
[8:23] patterns right so if it's a very cold
[8:25] day externally you probably don't need
[8:26] to be running your chillers at full
[8:28] blast you can just have like you know uh
[8:30] ventilation from outside right So
[8:33] predicting weather is important for a
[8:35] number of different reasons. This being
[8:37] one of them, right? And uh we very as
[8:40] with generative AI being very useful in
[8:42] many different fields, it's also become
[8:44] quite important in the AI weather
[8:46] modeling space.
[8:48] Um so as I kind of mentioned, there are
[8:50] many different practical use cases.
[8:52] Apart from us knowing whether we need to
[8:53] wear sunscreen or carry an umbrella
[8:55] outside, it's important for government
[8:57] to kind of like enforce policy, help
[8:59] with wild, help with disaster
[9:00] mitigation. It's helpful for us for data
[9:02] center optimization as I mentioned and
[9:04] also you know like helping farmers
[9:06] realize helping farmers better plan for
[9:08] for rain and crop yield management.
[9:12] Um so traditional method methods of
[9:15] weather forecasting relied on closed on
[9:17] partial error on partial error equations
[9:19] PTEES right they are typically quite
[9:22] complex they run on supercomputers you
[9:24] know so you imagine like thousands of
[9:26] nodes they run four times a day and by
[9:29] this I mean specifically a model from
[9:31] the European Meteorological Center they
[9:33] have a model called HRES right that's a
[9:35] baseline I'm going to share with you in
[9:36] a second what we compare AI weather
[9:38] models against um so they're quite
[9:41] limited But the last part is quite key
[9:43] that they cannot learn from historical
[9:45] data and we have tons and tons of
[9:47] weather data over multiple decades right
[9:49] and that's where the innovation core
[9:51] innovation came in on using machine
[9:52] learning on you know learning better
[9:54] weather models
[9:56] so uh a group at deep mind they
[9:58] developed a model it's called graphcast
[10:01] right it's based on the graph neural
[10:03] network architecture um it can fit in a
[10:05] single TPU right and it can run near
[10:08] instantaneously
[10:09] um with state-of-the-art performance
[10:11] ments on weather forecasting or recent
[10:12] benchmarks. There's a full nature paper
[10:15] if you want to take a read at it, but um
[10:17] it's quite powerful and there's a bunch
[10:19] of other weather related um AI models
[10:22] that have recently been introduced.
[10:25] So I probably won't get into um too much
[10:29] detail on how graphcast works but we do
[10:31] have some time so I can maybe go slower
[10:34] but basically you know it kind of if you
[10:36] think of every point on the earth both
[10:38] like on a longitude latitude um you know
[10:40] at a single point but also a certain
[10:41] height it takes a certain it takes
[10:44] certain discrete um you know points at
[10:46] different locations of the earth right
[10:48] it models different weather inputs such
[10:50] as you know the pressure air temperature
[10:53] humidity etc. right um for all of these
[10:56] points. So it's actually a quite massive
[10:57] model. It's in the order of I think tens
[10:59] of millions of parameters which is quite
[11:01] large right um not like LLMs but like
[11:04] large in in weather modeling at least.
[11:06] Um so it kind of takes all of these
[11:07] inputs right it passes that into the
[11:10] processor which is basically the graph
[11:12] neural network kind of um structure. It
[11:14] does it then does something called
[11:16] message passing where it kind of tries
[11:17] to predict that you know if this is the
[11:19] weather at time t is equal to zero what
[11:21] will the weather look at 6 hours 12
[11:23] hours 24 18 hours and in you know 6
[11:25] hours increments all the way till I
[11:27] think 240 hours right um it tries to
[11:30] predict the weather and then it goes
[11:32] back it once it you know understands the
[11:34] best embedding um it goes back to the
[11:36] actual um the decoder kind of maps it
[11:38] down to the actual variables at a
[11:40] certain step right um it has something
[11:42] called a multi mesh So basically that
[11:45] you know um when you try to understand
[11:47] what the weather is at certain point you
[11:49] care about you know you're more
[11:50] immediately impacted by the weather at a
[11:52] point near you right but more long-term
[11:55] you're also impacted by the weather at I
[11:56] don't know a continent away if something
[11:58] happens some other continent that
[11:59] influences the weather phenomenon all
[12:00] across the world it's quite interesting
[12:02] so they had seven different resolutions
[12:04] of message passing and they kind of you
[12:06] know combine all those in for predicting
[12:08] future weather states
[12:10] um there are many interesting
[12:11] applications um I'll talk more about
[12:13] renewable renewable forecasting in a
[12:15] second, but one of them from societal
[12:17] perspective is quite important. Um there
[12:19] was hurricane Maria, I think off the
[12:21] coast of um the US, right? Um quite
[12:24] catastrophic and quite sad. Um so this
[12:27] happened a few years ago. So what some
[12:29] of the weather organizations tried to do
[12:31] was they they took so we have two plots
[12:33] here. The plot on the left is hres which
[12:36] is a baseline from ECMWF and the plot on
[12:38] the right is graphcast right. Uh the red
[12:41] is the predicted path by each of these
[12:43] models and the one in black is the
[12:46] actual path. So as you can see graph
[12:49] cast is much closer to the truth right
[12:52] on or what the actual center of the
[12:54] cyclone is like a low pressure region.
[12:56] Um and that can transform into you know
[12:58] societal savings like we can evacuate
[13:00] people faster get disaster mitigation in
[13:03] and um I'm really excited by personally
[13:05] excited by this area of research and
[13:06] hopefully it can help us all in the
[13:08] future.
[13:10] Um so the project which I was a bit more
[13:12] directly involved with was in renewables
[13:14] energy forecasting. Um this was
[13:16] collaboration during my time at DeepMind
[13:18] with Google X and an organization might
[13:21] pronounce it correctly but Sen uh
[13:23] electric coordinator Electrio National
[13:26] um in Chile right um they were quite
[13:29] ambitious in being able to um kind of
[13:31] like decarbonize the grid and use as
[13:34] much renewable resources as possible
[13:35] right so they were quite interested in
[13:37] working with us on a state-of-the-art
[13:38] model for this
[13:41] um an interesting phenomenon for SEN was
[13:43] or Australia in general is if you look
[13:45] at a different source sources of energy.
[13:47] So solar, wind and thermal, right? Um
[13:49] over the years you can see that um so we
[13:52] have these plots for different years.
[13:54] You can see that the amount of thermal
[13:55] used decreases year on year, right? Um
[13:58] versus that for solar and wind
[14:00] increases. Obviously there's a bit of a
[14:02] ramp up and there's a duck curve
[14:03] behavior because you know we have solar
[14:05] in the day and at at night time we have
[14:07] to depend on other sources such as wind
[14:09] as well as thermal. But the high level
[14:11] trend is we seeing a larger demand for
[14:13] renewable sources of energy. So being
[14:15] able to predict them better is is very
[14:18] very important to have a good um you
[14:20] know load forecasting model.
[14:23] Um so naturally this kind of turned to
[14:25] models such as graphcast and us that you
[14:27] know can we very accurately predict how
[14:29] well the wind forecasting uh how how the
[14:32] wind speed would be so we can we
[14:34] forecast them better. So this could
[14:35] directly be used by the grid operators
[14:37] right. Um we have a very interesting
[14:39] plot here. This is from the original
[14:42] graphcast paper right? Um the first plot
[14:45] basically predicts the wind speed. So
[14:47] this 10 U that you see here basically
[14:49] means the 10 m above the ground east
[14:51] west wind speed right in meters/s. Um so
[14:54] we're kind of looking at this variable.
[14:56] Um there are three plots. The top plot
[14:58] is a prediction at 12 hours right like
[15:01] even your we right now at 12 hours
[15:02] beyond us. What does it think the wind's
[15:04] going to be? The the one after that is
[15:06] at 2 days and after that is at 10 days.
[15:08] Right now in this map of the earth you
[15:11] see this color schema. Blue represents
[15:13] where graph cast is performing better
[15:15] than the baseline hres and red is where
[15:18] it's performing worse. Right? And for
[15:21] some of you who have been paying
[15:22] attention, you can notice there are
[15:24] certain regions or geographies in the
[15:25] world where graphcast is arguably a lot
[15:27] worse. So you see the mountaneous
[15:28] Himalayas, you know, above like India,
[15:31] Tajakhan, China, all these countries
[15:32] over there and um it's probably also
[15:34] geography le. If you see Chile on the
[15:37] west coast of South America, you see a
[15:38] lot of red at 12 hours. I'm not sure if
[15:40] it's very visible from presentation, but
[15:43] those are regions where graphcast sort
[15:45] of underperforms the baseline models.
[15:47] Um, at short lead times by lead times, I
[15:50] mean the prediction interval. At longer
[15:52] lead times, at 10 days, if you look at
[15:54] the bottom plot, the red sort of
[15:56] disappears, right? So, AI based weather
[15:58] models or graphcast specifically kind of
[16:01] suffers at shorter lead times in geog in
[16:03] geographies with high, you know,
[16:05] orgraphy. They have mountainous
[16:06] orographies, right? But kind of gets
[16:08] better. And that was a problem for these
[16:11] um you know the weather forecast the
[16:13] grid cuz they wanted to have a better
[16:14] sense of like how well can they predict
[16:16] how well how much wind energy will they
[16:18] get in the short term.
[16:20] So what we try to do is you know we've
[16:22] my background is mostly on post training
[16:24] models right whether it's better models
[16:25] or gen media models. So you know me and
[16:27] the team we came in. So we have two
[16:29] plots here. Um I guess for the purposes
[16:32] of both of them represent the same
[16:33] thing. The right is just a normalized
[16:35] version of the left plot right. So um
[16:37] the black line is basically the baseline
[16:40] hres right the x-axis is the time
[16:43] interval which you're forecasting. So
[16:44] the longer you forecast the more error
[16:46] you have and um the y-axis on the left
[16:49] plot is the error right and on the right
[16:52] plot is a normalized error. So you can
[16:55] see that we want to stay under the black
[16:57] line over here right which is the
[16:59] baseline HS model and we experiment with
[17:01] very different techniques of tuning
[17:03] graph cast. We told it to very
[17:05] specifically focus on Chile. We told it
[17:07] to focus on wind and other correlated
[17:09] weather um you know weather variables
[17:12] and we also tried feeding it different
[17:14] prior other important information that
[17:16] we think could help the model. And you
[17:17] can see that with this experimentation
[17:19] like the plot where we kind of like you
[17:21] know kind of gave graphcast a greater
[17:23] focus on Chile but also told it that you
[17:25] know we'll kind of debias your outputs
[17:28] it performs a lot better than the
[17:29] baseline models. So that was great for
[17:32] us. It was great real world impact and
[17:35] eventually yeah we got a paper out but
[17:37] that's nice but eventually we were able
[17:39] to release this um you know with the
[17:40] Chilean government um and they could use
[17:43] some of our work for the downstream
[17:45] renewable forecasting
[17:47] just doing a quick time check. So yeah I
[17:49] think we have a healthy 20 minutes. So
[17:51] um um
[17:54] go yeah taking photos. Cool. Okay, so
[17:59] now I'll talk about stuff which I guess
[18:00] most of you are interested about more
[18:02] generative AI and the stuff which I've
[18:03] been working on more recently over the
[18:05] last two years in the generative media
[18:07] space right and this is a photo I
[18:09] borrowed from Google uh the model I
[18:11] worked on with my amazing collaborators
[18:13] Google imagin which is their uh textto
[18:15] image model it's quite nice um so you
[18:20] know in general the high level takeaway
[18:21] I think most of you'll agree with I
[18:23] guess this is why we at the conference
[18:24] is generative AI LLMs diffusion models
[18:26] they have been transforming industries
[18:28] and they're going to be, you know,
[18:29] changing our lives um in many different
[18:31] ways, right? Um I'm going to be talking
[18:34] more about this in a second, but this
[18:37] this this example on the right, it's a
[18:39] comic that was generated completely by
[18:41] just prompting a model, right? And uh
[18:43] what I focused on at DeepMind was better
[18:45] text rendering. So when you ask a model
[18:47] to generate an image or something, it
[18:49] usually or at least in the past used to
[18:51] get a text wrong most of the time. I'm
[18:53] sure it still does as well. So that's a
[18:55] very important commercial use case. So
[18:57] seeing you know instances like this
[18:58] where can produce you know comics with
[19:00] perfect text or near perfect text is
[19:02] very very encouraging and it's kind of
[19:04] great for great to see the commercial
[19:06] applications that open up with this.
[19:09] Um so I'll talk about you know the story
[19:12] I kind of want to mention here is some
[19:14] things were very very hard to do like
[19:16] even a year ago which is like a long
[19:18] time in generative AI and I'll I'll
[19:20] share like one specific use case on
[19:21] product reconextualization which we
[19:23] tried last year and the amount of effort
[19:25] and engineering and research we do to
[19:27] make it work and I won't give a spoiler
[19:29] I'll maybe just show it. So um as I
[19:33] mentioned um there is a use case called
[19:35] product recontextualization very
[19:36] important in advertising and e-commerce.
[19:38] So if you're a brand and let's say you
[19:40] have a product such as a phone and you
[19:41] might have just like one or two photos
[19:43] of it you know like maybe you know you
[19:44] have your SKUs and it's there in just
[19:46] one color and you just have two photos
[19:48] in a single color right so when you want
[19:50] to advertise on Amazon or different
[19:52] other um you know platforms you want to
[19:54] show your phone in many different kind
[19:56] of lights so I don't know with like a
[19:57] cool background or with you know some
[20:00] celebr or someone holding it or with
[20:02] some glitzy kind of like marketing um um
[20:05] kind of um portfolio right So
[20:10] generative AI has kind of enabled that
[20:11] and now it's probably a lot easier to do
[20:13] it but back then it was quite hard right
[20:16] um there's a term called product
[20:18] fidelity so for those of you who own
[20:20] brands and sell products um you know
[20:22] that you know when you advertise a
[20:24] product you want to make sure that the
[20:25] image of it is very very high fidelity
[20:28] to what it actually is which is not a
[20:30] problem when you actually take a photo
[20:31] of it but when you use generative AI to
[20:33] kind of like create a campaign sometimes
[20:36] it kind of gets something wrong so maybe
[20:37] the countours is off or I don't know the
[20:39] thickness is off and that's a big deal
[20:41] for brands right so this was a use case
[20:43] that we were looking at um quite a bit
[20:45] at deep mind last year there's existing
[20:48] baseline such as dream booth and
[20:49] instruct pixtopix that does some sort of
[20:51] fine-tuning product reality but what we
[20:54] noticed was that this is probably not
[20:55] good enough for like the bar advertisers
[20:57] care about right so the way we thought
[21:00] about it last year is we knew the models
[21:01] want to get better so we thought of this
[21:03] more as a short-term data problem right
[21:06] booth um which was what we built on
[21:09] basically requires a lot of like curated
[21:10] data to fine-tune the model on a certain
[21:12] product. So we kind of flip the problem
[21:14] around and say that what if we kind of
[21:16] like just synthetically create a bunch
[21:18] of highfidelity data what the product is
[21:21] and what the product is not. So we can
[21:22] use that for training a diffusion model
[21:24] so it can generate it in in new context
[21:27] much better right. Um so we tried this
[21:29] last year. Uh we have a paper at Icleair
[21:32] if you want to take a look at it that
[21:33] goes into details but we invested a lot
[21:36] on creating a scalable data pipeline so
[21:38] we can see how well um we can uh improve
[21:41] this sort of product refundexualization
[21:43] um and we did three major things right
[21:45] as I mentioned data pipeline we then did
[21:48] something called LORA based finetuning
[21:49] which is low rank approximation of
[21:50] finetuning per product right and then we
[21:53] created a sense of u let me just check
[21:55] saying the right thing but yeah we
[21:56] created a bunch of different autoators
[21:58] for further filtering once you train the
[22:00] model to kind of like generate
[22:02] highfidelity examples of a certain
[22:03] product, you want to use an autoator so
[22:05] it can further filter it down before you
[22:07] show it to an advertiser, right? So an
[22:10] example of data augmentation. So you can
[22:12] see here we have like this desk. You can
[22:14] use like a diffusion, you know, image to
[22:16] video model where you give it a starting
[22:18] frame and you ask it ask to rotate
[22:20] around it. So you get a bunch of
[22:21] different other novel views where you
[22:22] can take screenshots and use your
[22:23] training data. You can take a mask of a
[22:26] certain product, right? And then you can
[22:29] outpaint the background, replace it with
[22:30] different other backgrounds. So it gives
[22:31] more diversity. And then you can also,
[22:34] and this is the easiest bit, take
[22:36] negative. So you tell it what the
[22:37] product is not, right? So you can say
[22:38] that, oh, you know, this is a different
[22:40] chair from my chair. And so I want to
[22:42] feed this into the model as well. So it
[22:43] learns what not to produce.
[22:46] Um, so yeah, uh, we generated a bunch of
[22:49] different autoators as well. So I'm not
[22:51] sure how familiar all of you are with
[22:52] clip dino sec clip but these are used
[22:54] for me measuring different types of
[22:56] consistency whether the images are
[22:58] consistent amongst themselves are
[22:59] generated or is it consistent with the
[23:00] prompt as well as more semantic um kind
[23:03] of autoerators right um and what we saw
[23:07] after kind of combining all this
[23:08] together where we had this pipeline
[23:10] where we just have like very sparse uh
[23:12] product lines from advertisers right we
[23:15] went to our entire data augmentation
[23:17] pipeline on creating synthetic positives
[23:19] negatives background replacement,
[23:21] outpainting, inpainting and using that
[23:23] for post- training our diffusion based
[23:25] model right uh we saw tons of
[23:28] improvement in the performance and I can
[23:30] maybe just show this result right here
[23:33] um so we have two plots we've run this
[23:35] on our internal data set right so in the
[23:38] table on the top we run it on a data set
[23:40] of furniture um so the first row is what
[23:43] we run at our technique the second is
[23:44] with the baseline dream booth and we
[23:46] look at two different metrics how much
[23:48] is a pass rate how much do advertisers
[23:50] say they like it versus not at both the
[23:52] image level as well as like the overall
[23:55] product pass rates. So they have certain
[23:56] bar on how many of these images work for
[23:58] them, right? Um and you can see that
[24:00] with our data augmentation and other
[24:02] contributions, we were significantly
[24:04] better on existing dream booth based
[24:05] techniques, right? Um and then we ran
[24:08] this on a few other external data sets.
[24:10] Um the results were a bit more balanced
[24:12] in this sense, right? Um we were a bit
[24:14] lower on certain of these autoerators
[24:16] such as clip, right? uh but overall it
[24:18] was still quite promising. So uh and we
[24:21] were also not too far off in in in this
[24:23] sense. So very promising results. We
[24:25] were able to get a lot of excitement
[24:27] around this and we clear. So this was
[24:30] great and you know if you're an
[24:31] advertiser you'll get excited saying
[24:32] that oh this is awesome but then do I
[24:34] need to fine-tune my models a lot to get
[24:36] the same level of results?
[24:38] This was 2024.
[24:40] life is a lot easier right now, right?
[24:42] In in 2025, a lot of effort has gone
[24:45] into kind of zero shorting models to
[24:46] have these, you know, inherent
[24:48] capabilities or emergent capabilities,
[24:49] if you will, where that if you give a
[24:52] model a certain image of a product and
[24:54] you ask it to create new advertising
[24:57] campaigns for it in completely new
[24:59] context, right? Um, it can do a zero
[25:01] shot. So, I give an example of myself. I
[25:03] just did this last night, right? So, I
[25:05] took a photo of my head shot from uh
[25:07] somewhere and then um I said, "Oh, take
[25:10] a photo of me in a completely new
[25:11] context and flying a plane." And I would
[25:14] argue I it's pretty good, right? It kind
[25:16] of gets it gets me reasonably well,
[25:18] right? And this is the only context a
[25:20] model gets, right? Just a photo of me, a
[25:22] single prompt, and puts me in a Cessna
[25:24] plane. Pretty good. No fine-tuning,
[25:26] nothing else required.
[25:29] It's not exactly perfect though, right?
[25:31] Um I I tried another example immediately
[25:34] after that. So saying that, oh, I enjoy
[25:35] AI4. Um, I guess most of you don't know
[25:38] me, but that's not exactly my face,
[25:40] right? It's kind of added some baby fat
[25:41] over here. Um, so pixel perfect editing
[25:44] is still a challenge, but I'm quite
[25:46] confident that this is something that
[25:48] will get better with time and we can see
[25:50] a lot of other emergent use cases um
[25:52] with with these.
[25:55] Um, so that's as of today, right? I just
[25:57] ran this last night. Um I'm very excited
[26:01] about the future and all the
[26:02] opportunities will open up. Uh as I
[26:04] mentioned I worked on improved text
[26:06] rendering for uh image generation at
[26:08] Google. So we had a Verge article when
[26:10] we launched Imagin 4 last year uh sorry
[26:12] this year um at Google IO and a similar
[26:14] article for some of the other players um
[26:16] out there. So text rendering is a very
[26:19] very important use case. You can imagine
[26:21] that if you're an advertiser and you
[26:22] want to generate a campaign with
[26:23] generative AI models, even if it gets
[26:25] like, you know, if you have like 10
[26:27] words, it gets one word wrong, you can't
[26:29] say that, oh, you know, it's 90% there,
[26:31] that's good. You'll still discard it,
[26:32] right? So, um, it's something that has a
[26:36] pretty high bar, but it's also quite
[26:37] verifiable and easily measurable. You
[26:39] can use different like text models to
[26:41] kind of see what um, extract the text
[26:43] from a certain image, use that to kind
[26:45] of like enforce the ground truth was
[26:47] generated and use that to keep training
[26:49] models. So it was a lot of fun working
[26:50] on text rendering at Google scale. Um
[26:54] and as you start seeing these base
[26:55] capabilities, text rendering is one of
[26:57] them. Pixel perfect editing, you know,
[26:59] recontextualization. Um it's really
[27:01] really amazing to see the commercial
[27:02] opportunities open up. I mentioned
[27:04] Spear. So I just got to know recently
[27:06] that there was a collaboration with
[27:08] folks at Google um and the organizers
[27:10] here on creating the Wizard of Oz
[27:11] experience at Sphere. So there's some
[27:12] amount of generative AI involved is what
[27:14] I've learned, right? But even in areas
[27:17] such as advertising right uh we've been
[27:20] able to see um you know this is what I
[27:22] work with also at Google we've been able
[27:24] to use a generative AI tuned models so
[27:26] that advertisers can come in and out of
[27:28] the box be able to kind of deploy their
[27:30] campaigns um on Google system but I'm
[27:32] also sure more widely as well right um
[27:34] these are just two small examples of
[27:36] what's possible I think we at this point
[27:38] we're quite limited by imagination so I
[27:41] am very very excited for the future and
[27:43] where this technology helps us Right. Um
[27:46] so thank you so much for the talk um for
[27:48] allowing me to give the talk. Um putting
[27:50] my LinkedIn QR code again but I'd like
[27:52] to spend this time for uh any questions.
[27:55] So thank you.
[27:57] [Applause]
