Google DeepMind Deploys Text Diffusion on Edge Devices to Slash Latency

Text Diffusion — Brendan O’Donoghue, Google DeepMind — AI Engineer

AI Engineer

While text diffusion models cut latency tenfold by reducing memory bandwidth transfers, high serving costs at large batch sizes restrict their deployment to single-user edge applications like robotics.

AI-assisted summary of Text Diffusion — Brendan O’Donoghue, Google DeepMind — original source. Check the original for full context.

Source published:

Key ideas

  • Google DeepMind's Gemini Diffusion preview demonstrates that text diffusion can reach end-to-end client generation speeds of roughly 2,000 tokens per second by cutting memory transfers tenfold during batch-size-one inference. (6:13)
  • Unlike autoregressive models that get trapped by early token commitments, text diffusion uses bidirectional attention across its canvas to iteratively revise answers and intermediate steps as downstream reasoning resolves. (8:47)
  • Despite achieving up to 10x lower latency in single-stream environments, text diffusion remains expensive for centralized web-scale chatbots because its poor throughput at large batch sizes drives up aggregate serving costs. (4:02)
  • Alphabet has already deployed text diffusion models into production for edge and local workloads—such as robotics and on-device smartphone tasks—where batch size is naturally one and low latency is paramount. (25:00)

Detailed notes

## Background This talk was delivered by Brendan O’Donoghue, a research scientist at Google DeepMind, and published on YouTube by the AI Engineer channel on June 4, 2026. According to the page description, the presentation details why text diffusion provides a fast alternative to autoregressive models—requiring ~10x fewer memory transfers at 256 tokens—and explores trade-offs, bidirectional self-correction, and applications unlocked by 2,000 tokens-per-second generation.

## TL;DR O’Donoghue introduces discrete text diffusion, contrasting its parallel denoising passes across a full token canvas against the left-to-right next-token prediction of standard autoregressive architectures. He explains that text diffusion achieves up to 2,000 tokens per second at batch size one because it drastically reduces memory-bandwidth-bound weight streaming from HBM, though its compute intensity makes it expensive to serve under high-concurrency batching. The talk highlights unique architectural advantages including bidirectional reasoning, self-correction, dynamic and adaptive computation budgets, and surgical in-place text editing. Finally, O’Donoghue showcases interactive demos—such as real-time generated Wikipedia pages, a dynamic operating system, and voice-coded web applications—before clarifying in Q&A how blockwise generation, edge deployments like robotics and phones, and reinforcement learning apply to diffusion models.

## Atmosphere & style The presentation carries the engaging, grounded tone of an engineer speaking directly to peers, punctuated by pragmatic insights and dry humor about modern AI timelines. O'Donoghue presents complex architectural trade-offs between memory bandwidth and throughput with clarity and candid transparency, openly noting the serving costs that prevent large labs from replacing autoregressive systems at web scale. The atmosphere transitions smoothly from technical accelerator diagrams to dynamic live demos that elicit genuine curiosity, creating the feeling of a pragmatic peek behind the curtain at alternative paradigms actively evolving within frontier research labs.

## Bulletin - [06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) Google DeepMind's Gemini Diffusion preview demonstrates that text diffusion can reach end-to-end client generation speeds of roughly 2,000 tokens per second by cutting memory transfers tenfold during batch-size-one inference. - [08:47](https://www.youtube.com/watch?v=r305-aQTaU0&t=527s) Unlike autoregressive models that get trapped by early token commitments, text diffusion uses bidirectional attention across its canvas to iteratively revise answers and intermediate steps as downstream reasoning resolves. - [04:02](https://www.youtube.com/watch?v=r305-aQTaU0&t=242s) Despite achieving up to 10x lower latency in single-stream environments, text diffusion remains expensive for centralized web-scale chatbots because its poor throughput at large batch sizes drives up aggregate serving costs. - [25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Alphabet has already deployed text diffusion models into production for edge and local workloads—such as robotics and on-device smartphone tasks—where batch size is naturally one and low latency is paramount.

## Key takeaways - [00:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=0s) Text diffusion adapts continuous diffusion principles to language by applying discrete corruption to clean token sequences during training and iteratively refining noise canvases during inference. - [02:06](https://www.youtube.com/watch?v=r305-aQTaU0&t=126s) The Gemini Diffusion research preview was branched from Gemini 2.0 Flash to replace next-token prediction with diffusion, matching general task quality while substantially cutting latency. - [04:02](https://www.youtube.com/watch?v=r305-aQTaU0&t=242s) The primary barrier preventing large-scale commercial deployment of text diffusion is its poor serving throughput under heavy batch concurrency compared to amortized autoregressive models. - [06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) In single-stream autoregressive serving, generating 256 tokens requires 256 memory bus transfers from HBM to Tensor Cores, whereas text diffusion generates the same canvas in about 24 denoising passes. - [06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) Reducing memory bandwidth round trips by roughly 10x enables text diffusion to deliver up to 2,000 tokens per second into client browsers. - [08:47](https://www.youtube.com/watch?v=r305-aQTaU0&t=527s) Bidirectional attention eliminates causal masking, allowing diffusion models to retroactively correct mistakes in preceding tokens after finishing downstream calculations. - [10:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=600s) When solving multi-step arithmetic, Gemini Diffusion initially guessed 60, revised to 49, and finalized on 39 once it calculated 36 + 3, whereas GPT-4o and Gemini 2.5 Flash hallucinated or had to apologize for early missteps. - [12:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=720s) Dynamic computation provides predictable scaling at inference time, where allocating additional forward passes consistently improves performance across coding and reasoning benchmarks. - [12:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=720s) Adaptive computation allows the model to autonomously exit denoising early on simple or memorized sequences (like generating digits of pi in 4 steps) while devoting more steps to complex topics. - [14:26](https://www.youtube.com/watch?v=r305-aQTaU0&t=866s) Native in-place text editing allows targeted bug fixes, comment insertions, and paragraph bridging without needing to rewrite or regenerate surrounding context. - [16:09](https://www.youtube.com/watch?v=r305-aQTaU0&t=969s) Radical latency reductions enable dynamic software paradigms, including on-the-fly generated encyclopedias, fully simulated forum threads, dynamic operating systems, and voice-coded web apps generated in 15 seconds. - [20:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1200s) Text diffusion models train on standard pre-training corpora using adapted corruption schedules, and scaling model parameters decreases the total number of denoising passes required per generation. - [21:40](https://www.youtube.com/watch?v=r305-aQTaU0&t=1300s) Generating arbitrary output lengths relies on blockwise autoregressive generation, which denoises fixed windows (e.g., 512 tokens) and locks them into context for subsequent blocks. - [23:44](https://www.youtube.com/watch?v=r305-aQTaU0&t=1424s) Production architectures rely on discrete token diffusion with classification logits rather than continuous latent autoencoders. - [25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Standard post-training techniques like reinforcement learning transfer directly to text diffusion once policy updates are adjusted for multi-step denoising trajectories.

## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=0s) How Text Diffusion Works (Training and Inference)

Brendan O'Donoghue, a research scientist at Google DeepMind, opens his presentation as audience members are still trickling into the room, noting that the opening slides are primarily introductory. He frames text diffusion as a forward-looking research domain within DeepMind. While the broader community is already well-acquainted with image, video, and audio diffusion as the modern state-of-the-art—where a model is trained to reverse noise added to ground-truth media and then generates outputs at inference time by gradually refining a canvas initialized from pure random noise—the exact same underlying principle translates directly to natural language.

In text diffusion, training begins with an uncorrupted sequence of clean tokens, such as a well-formed sentence. Noise is systematically introduced through various corruption schemes. Although this corruption can theoretically occur continuously or discretely, discrete corruption is the standard mental model: tokens are either randomized or swapped for other random discrete vocabulary tokens across a spectrum of noise levels. A neural network is then trained to repair these corrupted sequences, predicting missing pieces and correcting errors across the entire context window.

During inference, generation flips this corruption on its head. Instead of writing text linearly from left to right, the entire sequence canvas is initialized with pure discrete noise—random tokens sampled directly from the vocabulary. Over a set of successive refinement iterations, the model simultaneously updates and denoises the canvas in whatever order it determines is optimal, progressively filling in information until a clean, fluent text output emerges. This dynamic visually mirrors image diffusion: starting from complete visual static, coherent sentences crystallize after several denoising steps.

```mermaid flowchart LR accTitle: Text Diffusion Training and Inference Workflow accDescr: Diagram showing how text diffusion corrupts data during training to learn denoising, and iteratively cleans random noise into coherent sentences during inference.

subgraph Training["Training Phase"] A["Clean Token Sequence"] -->|"Discrete corruption / noise"| B["Corrupted Token Canvas"] B -->|"Supervised objective"| C["Neural Network Predicts Corrections"] end

subgraph Inference["Inference Phase"] D["Pure Random Discrete Noise"] -->|"Iterative refinement step 1"| E["Partially Denoised Canvas"] E -->|"Iterative refinement step N"| F["Clean Generated Sentence"] end ```

---

[02:06](https://www.youtube.com/watch?v=r305-aQTaU0&t=126s) Gemini Diffusion Research Preview vs. Autoregressive Models

Reflecting on past milestones, O'Donoghue highlights the Gemini Diffusion research preview released roughly a year prior. It was branched off the Gemini 2.0 Flash architecture as an experimental variant designed to swap standard autoregressive next-token prediction for text diffusion, and it was opened to approximately 100,000 users. Benchmarking at the time indicated that Gemini Diffusion matched Gemini 2.0 Flash across general tasks, showing a slight advantage in coding evaluations and minor deficits in other areas, while delivering dramatically lower latency. However, O'Donoghue cautions against over-indexing on those specific benchmarks, observing wryly that a single year in modern machine learning represents “prehistoric times.”

To clarify the paradigm shift, O'Donoghue contrasts the operational mechanics of autoregressive architectures against diffusion models: - **Autoregressive models (e.g., standard Gemini, Gemma, GPT):** Text generation proceeds strictly token by token. Conditioned on an incoming context prompt, the model samples the first token, appends it to the context, and uses that new sequence to sample the second token, continuing sequentially until reaching a stopping criterion. - **Text diffusion models:** When context arrives, the system allocates an entire target canvas—spanning hundreds or thousands of tokens depending on the configuration—and initializes it with random discrete noise. The network runs multiple refinement iterations over the entire block in parallel, allowing tokens across the whole sequence to attend to one another bidirectionally rather than strictly gazing backward.

---

[04:02](https://www.youtube.com/watch?v=r305-aQTaU0&t=242s) Pros, Cons, and the Throughput Bottleneck

Text diffusion offers distinct architectural advantages alongside a critical operational disadvantage that currently limits its broad commercial deployment. On the positive side, the standout benefit is vastly superior inference speed; diffusion delivers far higher tokens-per-second generation by capitalizing on GPU and TPU compute characteristics. Additionally, diffusion operates without causal masking, unlocking bidirectional attention across the whole token canvas. This architectural freedom enables self-correcting generation (a model can draft an answer, realize down the page that its working reasoning contradicts the answer, and alter its prior text) and dynamic or adaptive computation, where difficult prompts receive more denoising iterations while easy prompts exit early. Diffusion models also support direct in-place editing, such as fixing trailing tokens and prompting the model to infill or regenerate the matching prefix.

The central roadblock holding text diffusion back from replacing industry standard autoregressive systems is its throughput profile under massive concurrency. Autoregressive models are slow for a single stream, but they amortize hardware costs effectively: large batches of incoming queries can be packed together across tensor cores, maximizing overall throughput and making cost per query viable at web scale. Because diffusion requires multiple complete forward passes over the exact same canvas data just to generate a single response, it hits compute thresholds prematurely. Consequently, text diffusion exhibits remarkably low latency for an individual user but poor aggregate throughput for high-concurrency environments, driving up serving costs. O'Donoghue points directly to contemporary platform struggles to illustrate why labs are cautious: “And right now, if you've played with Claude recently, you'll know that they have some throughput concerns, so people really care about throughput right now. And so no one's landing text diffusion into any of these big models, primarily because of that disadvantage. It's just too expensive to serve, even if it is much lower latency.”

---

[06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) Hardware Efficiency: Why Text Diffusion is Faster

Explaining why text diffusion achieves radically lower latency requires examining modern accelerator topology across GPUs and TPUs. Accelerators contain high-performance execution blocks like Tensor Cores designed for dense matrix multiplications, providing massive floating-point operations per second (FLOPS). However, model parameters, dynamic activations, and Key-Value (KV) caches reside inside High Bandwidth Memory (HBM). To execute a forward pass, weights and data must be streamed from HBM into the compute cores across a physical bandwidth bus. In modern hardware, adding raw FLOPS to silicon is relatively cheap, but scaling memory bus bandwidth is difficult and costly. As a result, accelerator architectures present an acute imbalance: abundant compute capacity constrained by a tight memory bandwidth channel.

```mermaid flowchart TD accTitle: Accelerator Memory Bandwidth Bottleneck accDescr: Flow diagram showing how weights and KV cache transfer from HBM over a constrained bus to tensor cores for computation.

subgraph Accelerator["GPU / TPU Architecture"] HBM["High Bandwidth Memory (HBM)<br>• Model Weights<br>• Dynamic Activations<br>• KV Cache"] Bus["Tight Memory Bus Channel<br>(Primary Bottleneck)"] Compute["Tensor Cores<br>(Abundant Compute / High FLOPS)"]

HBM -->|"Must stream data repeatedly"| Bus Bus -->|"Constrained transfer rate"| Compute end ```

When running single-stream (batch size one) autoregressive generation, serving is severely memory-bound. Generating each successive token demands streaming the entire model's weights and cumulative KV cache out of HBM into the cores. Generating 256 tokens autoregressively therefore requires 256 individual read cycles over the entire model parameter footprint. In contrast, a text diffusion model generating a 256-token canvas might only require 24 iterative denoising passes. While each forward pass streams the model weights, performing 24 passes rather than 256 yields roughly a tenfold reduction in overall memory transfers:

| Architectural Metric | Autoregressive Next-Token Model | Text Diffusion Model | |:--- |:--- |:--- | | **Tokens Generated** | 256 tokens | 256 tokens | | **Model Forward Passes** | 256 passes (1 token per pass) | 24 passes (canvas denoised in parallel) | | **Full Memory Weight Transfers** | 256 streaming transfers from HBM | 24 streaming transfers from HBM | | **Memory Bandwidth Efficiency** | Heavily memory-bandwidth bottlenecked | ~10x fewer data roundtrips over the memory bus | | **Observed Latency Profile** | High per-token latency at batch size 1 | Up to ~10x lower latency when memory bound |

Under genuine memory-bound operating regimes, this ~10x drop in bus traversals translates directly into an equivalent speedup in response latency. O'Donoghue notes that the Gemini Diffusion preview reliably hit approximately 2,000 tokens per second for real users. This 2,000 tokens/sec figure was not an idealized synthetic metric; it measured end-to-end delivery right into the client browser, fully accounting for the initial context prefill phase. While short single-token outputs remain dominated by prefill overhead, longer output sequences allow the system to fully realize this massive generational speed advantage.

---

[08:47](https://www.youtube.com/watch?v=r305-aQTaU0&t=527s) Bidirectional Reasoning and Self-Correction

Moving past raw latency metrics, O'Donoghue turns to the unique algorithmic properties of bidirectional attention within text diffusion, highlighting capabilities that linear next-token autoregressive models fundamentally cannot replicate. To demonstrate, he references a math problem showcased at Google I/O: evaluating an arithmetic expression with nested fractions and operations, specifically:

$$\sqrt{81} \times \left(\frac{2}{3}\right)^2 + \dots = 39$$

When standard autoregressive models tackle such prompts, they are forced to commit causally to every preceding token. If their early calculations drift off track, they become trapped by their own context history, frequently hallucinating or doubling down on erroneous reasoning steps to remain consistent with their previous tokens.

Gemini Diffusion behaves entirely differently due to its bidirectional canvas. After just a single initial forward pass over the noisy sequence, the model stakes out a rough overall structure: it writes `answer = 60`, which is incorrect, and immediately starts drafting out an explanatory derivation beneath it (`solution: calculate the square root of 81...`). By the second forward pass, as the intermediate calculations on the canvas begin to take shape, the model revises its top-level prediction from `60` down to `49`, updating its forward-looking draft on the fly as it works through the problem.

- In conventional causal generation, early mistakes propagate downstream because attention cannot reach ahead into future reasoning steps. - In diffusion generation, the entire response layout—both the final conclusion and the intermediate reasoning steps—co-evolves across multiple passes, allowing the network to continuously refine its answers as downstream details resolve.

[10:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=600s) Bidirectional Reasoning and Self-Correction

Brendan O’Donoghue walks through the mechanics of how the text diffusion model solves a multi-step arithmetic problem across forward passes. In an early forward pass, the model reaches five steps into the derivation ($2^2 = 4$), with several uncertain "blue" tokens still destined to flip. By the third pass, it has successfully worked through the full chain of logic to calculate that $36 + 3 = 39$. Because diffusion models enjoy bidirectional attention across the entire sequence rather than strictly predicting left-to-right, the model can observe its own downstream reasoning and go back to overwrite its earlier attempts. In this run, the initial response token had mistakenly predicted $60$, then corrected to $49$, before finally fixing the answer at the beginning of the text to $39$ once the full derivation resolved. A couple of subsequent forward passes then clean up remaining token artifacts.

```mermaid flowchart LR accTitle: Bidirectional Self-Correction in Diffusion Models accDescr: Diagram contrasting how diffusion corrects mistakes retroactively with autoregressive left-to-right compounding errors. subgraph DIFF["Diffusion Model (Bidirectional)"] direction TB D1["Pass 1: Guesses '60'"] --> D2["Pass 2: Adjusts to '49'"] D2 --> D3["Pass 3: Computes '36 + 3 = 39'"] D3 -->|"Retroactively corrects initial token"| D4["Final Output: '39'"] end subgraph AR["Autoregressive Model (Causal)"] direction TB A1["Outputs initial token: '40' or '42'"] --> A2["Generates reasoning left-to-right"] A2 --> A3["Forces error: '36 + 3 = 42'<br/>or issues awkward apology"] end ```

O’Donoghue emphasizes that this capacity for bidirectional reasoning and self-correction exposes a fundamental flaw in traditional autoregressive architectures. When fed the identical prompt, leading autoregressive models from that period—both GPT-4o and Gemini 2.5 Flash, which were vastly larger than the Gemini Diffusion research preview—stumbled: - GPT-4o guessed $40$ up front because that was its highest-probability single-token prediction without looking ahead; after generating the full chain of logic and finding $39$, it was forced to awkwardly apologize in-line ("Sorry, I made a mistake... It's 39, not 40"). - Gemini 2.5 Flash initially guessed $42$, refused to backtrack, and hallucinated its arithmetic downstream to preserve continuity, asserting that $36 + 3 = 42$. - Modern thinking models handle this via explicit scratchpads, but O'Donoghue observes: “And, you know, you can fix this with modern reasoning thinking models, but, you know, then you just kind of punt in the problem into something else.”

[12:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=720s) Dynamic and Adaptive Computation

Beyond bidirectional revision, text diffusion provides two distinct computational advantages: dynamic computation and adaptive computation. Dynamic computation allows the user to allocate an arbitrary compute budget at inference time. Supplying more denoising steps (additional forward passes) yields an almost monotonic improvement in performance across evaluation benchmarks. Even if an early draft is mostly complete, granting the model extra passes gives it the opportunity to spot edge-case errors and refine token assignments. Internal testing tracking six coding benchmarks revealed a consistent, smooth curve showing output quality climbing directly with the step count.

Adaptive computation operates in reverse: the model is trained to decide autonomously when an answer has converged and stop denoising, spending minimal compute on simple prompts and reserving deep computation for difficult ones. O'Donoghue illustrates this dynamic behavior with examples from the Gemini Diffusion model: - Memorized/simple tasks: Generating the first 100 digits of $\pi$ requires 100 tokens, yet the model completes it in only 4 denoising steps because the sequence is readily recalled from weights. An autoregressive model would have produced only 4 tokens in that same time budget. - Moderate tasks: Writing standard code like FizzBuzz required roughly 18 forward passes. - Complex tasks: Explaining quantum mechanics in a single concise paragraph required 31 denoising steps, as the model spent more iterations settling on phrasing and conceptual accuracy. - Benchmark scaling: Across evaluations, easy suites like MBPP (mostly basic Python programs) completed rapidly, whereas challenging benchmarks like GPQA Diamond required substantially longer compute budgets, fully determined by the model's self-assessed convergence.

[14:26](https://www.youtube.com/watch?v=r305-aQTaU0&t=866s) In-place Text Editing

O'Donoghue highlights fast in-place text editing as another capability native to diffusion models. In computer vision, diffusion models perform inpainting by observing the surrounding unmasked context to cleanly reconstruct cut-out sections—unlike autoregressive image generators that are forced to proceed in strict left-to-right, raster order. The identical mechanism applies directly to text diffusion. Because the network processes all sequence positions simultaneously, it can manipulate specific token spans without regenerating surrounding content from scratch.

In live demonstrations, O'Donoghue showcases several in-place workflows: - Targeted bug fixing: Presenting a snippet of code with the command "There's a bug in this code. Can you fix it?" prompts the model to denoise and rewrite strictly the buggy lines while leaving adjacent code untouched. - In-place documentation: Inserting docstrings and explanatory comments directly into existing function blocks without re-emitting the entire script. - Narrative insertion: Providing the first and third paragraphs of a story and asking the model to insert an intermediate paragraph; the model leverages full bidirectional visibility of both the preceding and following prose to craft a contextually seamless bridge.

[16:09](https://www.youtube.com/watch?v=r305-aQTaU0&t=969s) Low-Latency Applications and Demos

The most profound practical differentiator of text diffusion is ultra-low latency. Rather than simply serving familiar text interfaces slightly faster, radical speedups unlock entirely new software paradigms. O'Donoghue shares several internal prototypes and community experiments showcasing dynamic generation:

- Real-time generated Wikipedia: An interactive encyclopedia prototype where clicking any hyperlink generates the entire target article on the fly—including complete HTML structure, body copy, and styling—fast enough to feel like browsing live web servers. - Dynamic Reddit clone: A simulated forum generating synthetic community threads, bot discussions, and commentary on the fly for arbitrary user-invented topics (such as a subreddit dedicated to sharks). The interface combines text diffusion with Google's image model (Juno) to dynamically populate images alongside text, though the image model lags slightly behind the near-instant text generation. - The Generative Operating System: O'Donoghue's personal favorite demo, an entire desktop operating system where every click dynamically generates the next state: “So it looks like a real operating system, but it's all been generated by the model on the fly, responding to every click.” Clicking folders, opening a README file, or launching browser views renders the corresponding OS interface in real time. - Voice-driven "vibe coding": A community project by a developer on X (Twitter) using the Gemini Diffusion web interface. In a recorded clip, the developer speaks instructions aloud to construct a functioning task manager: > “Create a to-do app. Add 10 random to-dos. Allow to-dos to have a completed state. Mark four random to-dos as completed. Allow me to sort to-dos by name and by state... Please convert this to dark mode.” The model generated, updated, and re-skinned the interactive web application end-to-end in approximately 15 seconds.

O'Donoghue closes the presentation by reiterating to the audience of AI engineers that ultra-low latency will redefine user software experiences, inviting the developer community to explore these affordances as next-generation text diffusion models roll out from the team.

[20:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1200s) Q&A: Training Data, Distillation, and Step Scaling in Larger Models

Brendan O'Donoghue opens the floor to audience questions, first addressing how text diffusion models are trained relative to standard autoregressive architectures. He confirms that the data pipelines remain essentially identical, explaining that while the underlying algorithms and corruption objectives have to be adapted for diffusion, the team feeds the models the exact same training data. When an audience member asks about distillation techniques, O'Donoghue notes that text diffusion models can indeed be distilled, though he is not certain whether any formal papers on the topic have been published externally yet, hinting that the team plans to release work in this area soon.

The discussion shifts to how model scale affects diffusion efficiency. O'Donoghue points out an advantageous scaling property: as parameter counts grow, models actually need fewer denoising passes to reach the same final text output. Even though each individual forward pass becomes computationally heavier in terms of FLOPs, the total number of required steps drops significantly, creating what he describes as a diminishing marginal cost when serving larger models.

- Training uses standard text corpora; only the training algorithm and corruption schedule change. - Larger parameter models require fewer diffusion steps for equivalent output quality compared to smaller variants. - While per-step FLOPs rise with model size, the reduction in total forward passes offsets server evaluation costs.

[21:40](https://www.youtube.com/watch?v=r305-aQTaU0&t=1300s) Blockwise Autoregressive Generation and Context Windows

An audience member asks how text diffusion models handle arbitrary output lengths and whether they are locked to a rigid maximum token budget. O'Donoghue explains that generating text across long horizons typically relies on a hybrid setup known as blockwise autoregressive generation. Rather than trying to diffuse an infinite sequence all at once, the system fixes a specific block length—such as 32, 512, or 1,024 tokens—diffuses that block across its denoising steps, and then treats that generated block as standard context to predict the subsequent block.

When asked whether models must know the output length in advance, O'Donoghue clarifies that models can either use an auxiliary prediction head to guess the exact response length or operate within a fixed upper limit and terminate early. Addressing questions about whether newly generated windows can go back and modify previously generated blocks, he notes that while bidirectional cross-block editing is theoretically possible, in practice the generated context is typically locked in place once finalized, functioning similarly to standard autoregressive history.

```mermaid flowchart LR accTitle: Blockwise Autoregressive Generation accDescr: Text diffusion processes input context, diffuses fixed-size token blocks, and appends them to autoregressive history. subgraph Inputs ["Input Stage"] A["Context Prompt"] end subgraph DiffusionBlock ["Fixed-Window Denoising"] B["Noisy Mask Tokens"] -->|"24 Denoising Steps"| C["Clean Block (e.g. 512 tokens)"] end subgraph OutputHistory ["Autoregressive Chain"] D["Lock Output into History"] --> E["Next Block Denoising"] end A --> B C --> D ```

- Standard generation applies diffusion within fixed-size sequence windows (for instance, 32, 512, or 1,024 tokens). - The prefill phase functions identically to autoregressive models; the generation step then runs blockwise. - Models operate with a token budget limit but retain the ability to finish and stop before hitting that threshold.

[23:44](https://www.youtube.com/watch?v=r305-aQTaU0&t=1424s) Discrete Diffusion vs. Latent Space Representation

A questioner inquires whether the diffusion process takes place in a continuous latent space or directly over token vocabularies. O'Donoghue explains that their implementation relies on discrete diffusion. At every denoising step, discrete tokens are corrupted or masked, and the model directly outputs discrete vocabulary tokens via standard prediction logits at the top layer.

He acknowledges that continuous latent-space text diffusion exists in academic literature, but notes that discrete token-level diffusion remains the predominant approach for state-of-the-art text models. Because tokens remain discrete throughout the process, the architecture relies on standard classification heads rather than continuous autoencoders.

> “Well, so this is all like discrete diffusion, right? So it's always tokens in, tokens out.” — Brendan O’Donoghue

[25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Deployment Niches: Robotics, Edge Devices, and RL Reasoning

Looking ahead at the architectural battle between autoregressive and diffusion models, O'Donoghue argues that they serve entirely different deployment regimes rather than competing for the exact same workloads. The major challenge for text diffusion is serving throughput at high batch sizes, making it expensive for massive centralized server farms like those powering consumer web chatbots. However, for single-stream, on-device tasks where queries are not batched with thousands of other users, text diffusion's raw low latency gives it a massive advantage.

O'Donoghue reveals that Google is already running text diffusion models in production on edge devices within the Alphabet ecosystem, specifically highlighting robotics and phone applications where local latency is paramount and batch size is effectively one. When attendees ask whether diffusion models can match frontier reasoning benchmarks or handle reinforcement learning, O'Donoghue answers affirmatively, explaining that standard post-training techniques like RL transfer over once the optimization objective is adapted to diffusion trajectories.

Addressing an audience proposal to speed up diffusion by initializing generation with a smaller draft model rather than pure noise, O'Donoghue cautions against the added training complexity: “The problem with that is if you train it with noise, it expects noise. You'd have to train it with this smaller model as you like the output. It just adds complexity. So we don't usually do that.” After answering a final lighthearted question about generating graph structures, O'Donoghue thanks the crowd and wraps up the session.

> “Quality isn't the concern. It's the throughput for serving in the big batch setting.” — Brendan O’Donoghue

- Text diffusion is already deployed in real-world on-device applications inside Alphabet, including robotics and smartphones. - For single-batch inference, diffusion matches autoregressive quality while eliminating sequential latency overhead. - Reinforcement learning (RL) reasoning pipelines work on text diffusion architectures by adjusting the RL policy update algorithms to the multi-step denoising path.

## Notable quotes > [25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Quality isn't the concern. It's the throughput for serving in the big batch setting. — Brendan O’Donoghue

## Mentioned Brendan O'Donoghue, Google DeepMind, Gemini Diffusion, Gemini 2.0 Flash, Gemini 2.5 Flash, GPT-4o, Claude, Gemma, Juno, Google, Alphabet, Reddit, Wikipedia, MBPP, GPQA Diamond

## Video description GPT-4o answered 40. Gemini 2.5 Flash answered 42 and stuck to it even after working through the reasoning incorrectly. The Gemini Diffusion model, considerably smaller than both, answered 60 on the first forward pass, then 49, then corrected itself to 39 once it finished reasoning. Bidirectional attention means it can see future tokens and go back to fix mistakes. Autoregressive models cannot do that.

Brendon O'Donoghue covers why text diffusion is fast (24 denoising steps to generate 256 tokens means roughly 10x fewer memory transfers than autoregressive generation), what the tradeoff is (lower throughput at large batch sizes makes it expensive to serve at scale today), and what gets unlocked when latency drops to 2,000 tokens per second. The demos include a fake Wikipedia generated on the fly, a Reddit clone with AI generated comments and images, an operating system where every click generates the next screen, and a todo app built in 15 seconds by voice.

Speaker info: - https://x.com/bodonoghue85 - https://bodono.github.io/ - https://www.linkedin.com/in/bodono/

Timestamps: 0:00 Introduction to Text Diffusion 1:02 How Text Diffusion Works (Training and Inference) 2:06 Gemini Diffusion Research Preview 3:04 Difference Between Autoregressive and Diffusion Models 4:02 Pros and Cons of Text Diffusion 6:13 Hardware Efficiency: Why Text Diffusion is Faster 8:47 Bidirectional Reasoning and Self-Correction 12:00 Dynamic and Adaptive Computation 14:26 In-place Text Editing 16:09 Low Latency Applications and Demos 20:05 Q&A Session

---

Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/r305-aQTaU0-text-diffusion-brendan-o-donoghue-google-deepmind

Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):

- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney

---

[Source](https://www.youtube.com/watch?v=r305-aQTaU0)

Reading summary

Brendan O’Donoghue of Google DeepMind presents text diffusion as an alternative to autoregressive architectures, explaining that while its parallel denoising yields massive latency reductions and bidirectional self-correction, high serving costs under large-batch concurrency currently restrict its commercial deployment to single-stream and on-device environments. Mechanics of Discrete Text Diffusion (0:00) Text diffusion applies the same corruption-and-refinement principles as image diffusion directly to natural language, using discrete tokens rather than continuous latent spaces. During training, clean token sequences are corrupted with random vocabulary tokens, and the neural network learns to predict corrections across the entire context window. At inference time, the model initializes a target canvas of hundreds or thousands of tokens with pure discrete noise and refines the whole block across multiple parallel forward passes, allowing tokens to attend bidirectionally across the sequence. For extended generation beyond a single window, O'Donoghue explains that systems implement blockwise autoregressive generation. The model diffuses fixed-size sequence blocks—such as 32, 512, or 1,024 tokens—and commits each finished block into standard context history before diffusing the subsequent block. Output lengths are managed via upper limits or auxiliary length-prediction heads, and O'Donoghue notes that scaling parameter counts requires fewer denoising iterations to achieve equivalent text quality. Hardware Efficiency Versus Serving Throughput (4:02) The primary operational distinction between text diffusion and autoregressive generation lies in how they interact with accelerator memory architectures. In modern GPUs and TPUs, dense compute capability is abundant, but streaming model weights and Key-Value caches from High Bandwidth Memory (HBM) over memory buses creates a severe bottleneck at batch size one. Generating 256 tokens autoregressively requires 256 separate streaming memory passes over the entire parameter footprint. By contrast, a text diffusion model can denoise a 256-token canvas in roughly 24 forward passes, yielding an approximate tenfold reduction in memory bus transfers. In Google's Gemini Diffusion research preview, this architectural shift produced real-world browser delivery rates of roughly 2,000 tokens per second. Despite this latency advantage, O'Donoghue emphasizes that diffusion faces an acute throughput roadblock at web scale. While autoregressive models amortize weight-streaming costs across large concurrent user batches, diffusion models require multiple full forward passes per sequence, rapidly saturating compute thresholds. As platforms face severe capacity pressures, major providers avoid serving text diffusion for centralized consumer chatbots because the aggregate serving cost remains too high. Bidirectional Revision and Adaptive Compute (8:47) Because text diffusion discards causal masking, its bidirectional attention enables algorithmic capabilities unavailable to standard next-token models. When solving complex multi-step math problems, autoregressive systems often commit to incorrect early tokens and hallucinate downstream calculations to maintain sequential consistency. By contrast, Gemini Diffusion drafts intermediate derivations and retroactive conclusions concurrently, adjusting initial guesses—such as revising an opening answer token from 60 to 49 and finally to 39—as downstream logic resolves across iterations. The architecture also performs non-destructive in-place text editing, allowing users to modify isolated lines of buggy code, add narrative paragraphs between existing prose, or insert docstrings without re-emitting untouched context. Diffusion models also support dynamic and adaptive computation budgets. Users can trade latency for performance by increasing denoising steps, which internal testing showed yields near-monotonic score improvements across coding benchmarks. Conversely, models can be trained to halt generation upon self-assessed convergence. O'Donoghue noted that the Gemini Diffusion model resolved simple memorized prompts like 100 digits of pi in just 4 steps, standard tasks like FizzBuzz in 18 steps, and complex conceptual explanations in 31 steps. Production Niches in Single-Stream Systems (16:09) Rather than directly displacing centralized autoregressive models, text diffusion currently targets single-stream environments where batch size is effectively one and local latency is paramount. O'Donoghue disclosed that Google already deploys text diffusion in production on edge hardware within Alphabet, specifically citing robotics and smartphone applications where sequential latency cannot be tolerated. Standard post-training methods, including reinforcement learning, transfer to diffusion models once optimization objectives are adapted to multi-step denoising trajectories. At human-facing latencies, O'Donoghue demonstrated prototypes illustrating how ultra-low latency alters interaction design. Examples included dynamically generated encyclopedia articles where clicking hyperlinks renders formatted HTML instantly, simulated community forums generating text and Juno-based images on demand, and an experimental generative operating system that computes file system states and desktop views per click. In another demo, a voice-driven coding workflow built, updated, and re-styled an interactive task-management web application end-to-end within 15 seconds. While text diffusion demonstrates substantial latency and algorithmic advantages through bidirectional parallel refinement, its prohibitive serving costs in large-batch server environments remain an unresolved commercial obstacle. O'Donoghue suggests its near-term adoption will center on edge hardware, robotics, and interactive applications where single-stream responsiveness outweighs datacenter throughput economics.

Source quotes

[25:00] Brendan O’Donoghue: Quality isn't the concern. It's the throughput for serving in the big batch setting.

Original transcript

[0:00] Thank you. still filtering into the room. It's mostly intro stuff for the first couple of slides, so they won't miss anything. Okay, welcome everybody. My name is Brendan. I'm a research scientist at DeepMind. I'm talking today about text diffusion, which is kind of a more forward-looking research area. at DeepMind. So you're probably familiar with image and video diffusion, which is kind of state-of-the-art for these modalities right now where you You take the ground truth. say, image, you add noise to it in training, and then you train a neural network to remove that noise gradually, And then at inference time, you just initialize the picture with pure noise, and then you iteratively refine out the noise to recover back to, you know, whatever image or video or audio or whatever you're looking And the principle is essentially the same for text. for text diffusion, where you start with a clean-- sequence of tokens, like so a clean sentence or something like that. And then you gradually add noise. You corrupt it somehow. There's lots of different ways to do that. You can do it in a continuous or discreet way, but let's just say discreet for now. which would in this case just mean adding random tokens, or replacing tokens with other random tokens. And you do that for a bunch of different noise levels, and you train the neural network to fill in to try to correct. the mistakes basically in the text. And then at inference time, you initialize the sequence of tokens to just pure noise, like pure random discrete tokens from the vocabulary, and then you iteratively refine through that to fill in the information. in the order that the neural network wants to do, to recover back to say a clean sentence. And then in practice, so we--you know, I saw--we--I showed you some GIFs here of what it looks like for images. You get very similar. looking outputs for text where it kind of starts off all noisy and then gradually fills in the text, and then you get clean. relatively clean outputs at the end. Okay, so the team I'm on, we had a research demo release one year ago now called Gemini Diffusion, which was a variant of a Gemini model. which ditch text diffusion instead of autoregressive next token generation. And that was like a research preview that was open to about 100k people. And, you know, we're still, you know, keep posted for new developments in that direction, upcoming soon. And we did have some good numbers at the time, but again, it's a year ago, which is like prehistoric times in this field. Our kind of main comparator model was Gemini 2.0 Flashlight at the time, because that was the architecture we were branching from the text fusion model. And we basically had very similar quality across the board there. Mostly, you know, a little bit of advantage in code. a little bit of disadvantage in some other areas, but kind of relatively similar performance, at much better latencies. But again, this is a year ago, so I wouldn't fixate too much on these numbers. Okay, so what's the difference between autoregressive generation and diffusion? So in the standard vanilla Gemini-Gemma, know, GPT, whatever. MR. generation of text, you do this you have some context that comes in and you want to generate some response to that, and the model does it one token at a time. So it generates the first token and then condition on that generates the next token and so on. Whereas in diffusion, you have, you know, a context will come in, whatever that is, and it'll initialize, like I mentioned, like a long sequence of tokens, could be hundreds, could be thousands, could be shorter, depends on the model. to be random noise, and then it iteratively refines that canvas to remove the noise over the course of a few denoising steps. So rather than one token at a time, it does the entire block together but over a couple of iterations. not just one pass, it does multiple passes, but it gets to attend to the future tokens and so on. So it's kind of a different way of generating text. MR. So that obviously has some pros and cons. So the main pro that people really like, and probably is the biggest advantage that text diffusion models have, is that it's just faster inference. It just generates faster tokens per second. because it makes much better use of the hardware, the TPU and the GPU. And I have some slides on that to explain why. But some other advantages are I can do bidirectional attention within this canvas of tokens. So you know-- Autoregressive models can only attend to the past. They have causal attention. within their transformer, whereas a text fusion model is not restricted, though. It can attend to the future. And that has some interesting properties, like it can do self-corrected generation based on future tokens. So we could do some reasoning. see that I got the answer incorrect, and then go back and fix the reasoning and do it again. I have a demo of that. Because this process is iterative, it does a number of steps. to respond. That means the model can actually do adaptive computation. It turns out that you can train the model to spend more time on harder [5:00] problems and less time and easier problems. And like diffusion models in general, you can do things like in-place editing, where you say, like, fix the last tokens. and give me the prefix that corresponds to those tokens. But the main disadvantage it has, and the reason why it's not kind of used everywhere right now, is lower throughput for large batches. Autoregressive models, they're slow, but you can have a big batch of queries together. And then if you push that through the neural network, on the GPU, each individual user is slow, but you make good use of the TPU by doing that. And so you can serve a lot of queries. And so you keep your costs down. You can serve a lot. Whereas since TextDivision does multiple forward passes on the same data, you know, it hits a compute threshold basically earlier, And it's-- it's even though it's lower latency for any one user, it tends to be lower throughput overall, so higher cost to serve. And right now, if you've played with Claude recently, you'll know that they have some throughput concerns, so-- Thank you. People really care about throughput right now. And so no one's landing text diffusion into any of these big models, primarily because of that disadvantage. It's just too expensive to serve, even if it is much lower latency. Okay, so just leading into why does it have lower latency, in case you're not familiar with kind of the architecture of how GPUs and TPUs run today. So in a GPU, there's like a tensor core, which does these like big matrix multiplies. It's very efficient, has a lot of flops. or hops or whatever. And then the memory that sits on the TPU GPU this HBM, that's where the weights and the activations and everything are stored, and that has to transfer over. from the memory all the way to the activations in the KV cache, into the tensor core, in order to do the computation. And so it has to flow through this bandwidth channel. and that bandwidth channel is very tight, It turns out that both GPUs and TPUs have a lot of flops, and not that much bandwidth. It's quite hard-- it's expensive to put bandwidth onto these chips, and it's easy to put flops. So because of that ratio, you can If you do, the more flops you do for each streaming amount of data you put through, the better. All right. So when you were serving, an autoregressive model, these chips are memory bound. They're basically-- bottlenecked by this bandwidth. So when you do autoregressive next token generation, for each token, You're doing one token at a time, let's say batch size one. You have to stream over the entire neural network and all the KV cache and everything. to get one token and then you do it again for the next token and so on. Whereas for text division, you're generating, say, 256 tokens You still stream over everything. But, If you can do that less times, than the number of tokens, this iterative refinement process, then you'll get a speed up. So if you can do, say, 24 passes to generate 256 tokens, You'll be doing it. 10 times fewer memory transfers. than an autoregressive model. And if you are truly memory bound, then you'll be 10 times faster, something like that. So that's the real reason, that's the hardware reason why text diffusion models at much lower latency than order regressions. So we had this Gemini Diffusion demo, maybe some of you got access to it last year, and that was able to hit something like 2000 tokens a second. pretty consistently, depending on the length of the query. Obviously, it depends the longer sequence is generating, the less it's pre-fill dominated. And so you can really lean into these very long sequences of very fast tokens. But if you're only generating one token, for instance, you'll be just dominated by the cost of the pre-fill. And the tokens per second number that was reported on this web page was incorporated pre-fill and everything I've So this was 2,000 tokens secondized, genuine raw tokens that you would receive in your rib resa. OK. So that was the-- kind of a whirlwind tour of text fusion And its main advantage, which is latency, but I want to dig in a little bit into some of the other advantages that text fusion has, which are kind of a little bit less-- talked about in the literature. But I think they're pretty cool and this is why I'm excited about it. So at I/O last year, Google I/O last year, they showed this demo for the text division model which is You know, this is a really easy prompt. But, you know, lots of models actually make mistakes on it. So the prompt is -- if you go to the next slide -- What is the square root of 81 times 2/3 squared plus blah, blah, blah? And I think the answer is 39 to this problem. And so-- You pass that into the model and you ask the Gemini diffusion to respond to that. And after one forward pass, these are the tokens that's generated. So one forward pass through the model, It's starting to respond. Now, through this iterative refinement process, a 1, 4, it is not all it's going to do. But after one forward pass, it has this. So it has answer equals-- and then it says 60. It's not correct, but that's what it's guessing for now. And then it starts to do the reasoning. So a solution, calculate the square root of 81, and so on. After two forward passes, It's changed 60 to 49. And it's gotten a little bit further [10:00] So it's gotten like, you know, five steps into the reasoning. 2 squared equals 4. And some of the blue tokens are still going to change. And then after three forward passes, it's actually gotten all the way through the reasoning. So it gets the answer correct at the end. 36 plus 3 is equal to 39. And it's gone back and fixed the original response to, say, 39. So it had a mistake twice, 60 and 49. But once it finished the reasoning, it was able to return back and fix the mistake that it made at the start. Now it's going to do a couple more forward passes in order to fix, you know, some of these tokens that aren't quite right in the text, but overall that's basically the structure of the output that it'll return. And this is, you know, this is a A property that text diffusion models have, This ability to do bidirectional reasoning, so to to not only see the past but also see the future, that it's going to utter, it's going and also to use that information to do self-correction. So I made a mistake. But it was able to-- it had another forward pass. It was able to go through and fix that mistake. At the time, you know, much, much bigger models than the one we were serving. made a mistake for this problem. So both ChatGPT 4.0, which was new at the time, and Gemini 2.5 Flash, which was brand new at the time, both made an error on this exact problem. So you give them the exact same prompt, and then they would say, remember the answer is 39. THE BUSINESS. GPT-4-0 said 40. Because that's the best guess it can do at that one token. It went through the reasoning, and then it said 30. It did manage to figure out I was 39. It said, sorry, I made a mistake. It said, It's 39, not 40. Gemini 2.5 flash also made a mistake at 42, and it actually just -- stuck to its guns and never changed and said 36 plus 3 is 42. So it incorporated the error into its reasoning later. These are way bigger models than the Gemini diffusion model. It's a flaw of order of models, that the text version models don't have. And, you know, you can fix this with modern reasoning thinking models, but, you know, then you just kind of punt in the problem into something else. Anyway, okay, so that's one-- which is bidirectional reasoning, self-correction. Another one is what I hinted at before, which is dynamic computation. So you can give the model more time, that inference more forward passes, you get a bigger budget, and it can just do better. It's not exactly monotonic, but it is roughly monotonic, that the quality across every eval basically just continues to go up. Because it gets, even if the solution is almost entirely clean and correct, it gets to look at it. and see that I made a mistake, and then fix it. So you get this nice kind of-- curve where you always see-- As the number of denoising steps, which is the forward you get--overall, the quality gets higher. And these are just six coding evals that we monitor internally. Thank you. On top of that, a slightly different concept is the model can do adaptive computation, which is that you can allow the model, you train it in a way, to determine itself when it is finished, And then for easy responses, it can... use a little bit of compute, and for harder responses, it can take longer. So here's just three examples from the Gemini diffusion model. which is what are the first 100 digits of pi. This is actually 100 tokens. It looks like a short response. It's actually 100 tokens. And it only takes four steps to do that. Because the model, it's an easy response because you've just memorized. the 100 digits of pi, you just output it. Whereas an order regression model at the same time would have only done four tokens. So that's a very easy one. Slightly more challenging is to write a little bit of code, so that takes, you know, 18, Forward passes. to generate FizzBuzz, and then something more complicated is explain quantum mechanics in a single paragraph, and that took 31 denoised steps. It just took its time, just for whatever reason, decided to spend longer on those ones. And so the model naturally gets to decide, I guess, to determine when. It's going to finish. and return the response. And typically we see that harder evals take more time. So this is a year ago now, so the evals are kind of Old school, but On the harder end is GPQA diamond. which for the model size we were targeting was quite a hard eval, and that took a long time. for it to respond to those ones. Whereas on the other end, like MBPP, which is mostly basic Python programs, it was very easy to-- it took very little time for it to respond to these things. And this is entirely determined by the model itself. Just easier problems, easier prompts it could respond to quickly, and harder ones it decided itself to spend more time reasoning. Okay, so that's another property, which is the dynamic and adaptive computation. Lastly is the kind of fast in-place editing. So-- Diffusion models in general have this very nice property where you can-- so take an image. like in this example, cut something out of it or whatever, or give it a little prompt, and it'll fill it in. And you can use the context that you haven't cut out to fill in the piece you've cut out correctly. And so you can use that for like clever image editing, things like that. And the reason they can do that is because, you know, it's not autoregressive. There are autoregressive image generators. [15:00] Which go left to right, top to bottom, like raster order. generating pixels, you know. Diffusion doesn't work like that. It'll just see the entire image and then start to denoise it. And because of that, because it gets to see every pixel, it gets to see every pixel, it can fill in the missing information and do it in a way that's kind of consistent whatever prompt you're giving. So we can do something similar. So I have a couple of demos here. MR. Can you see that? Yeah. So this is just some code, and you say, "There's a bug in this code. Can you fix it?" And it'll just make the edit in the correct place. You can barely see that. of the embassy. And you can say things like, can you add documentation? It'll go in. just one by one generating all the tokens, is doing a clever editing procedure to to actually fill in the correct edits here. You can do that with more general text, Like you can take a story, And then say, add a middle paragraph. And because it can see that the first and the third paragraph, it can fill in. the paragraph in a way that's consistent with the rest of the story, for instance. And this is just in-place editing, basically. OK, so those are some of the advantages. Don't have a lot of time. The biggest advantage, like I mentioned, is this low latency. And-- You know, we really lean into that. And I just want to show you a couple of demos of some of the things that people internally have built to kind of show what the advantage of low latency can give you. So it's not just the same thing faster. Some really new applications. So in your own work, you're all AI engineers. It would be interesting to see when the next diffusion model comes out from our team, what the low latency could unlock. and kind of what new applications can be built. So here are just some demos. So this is Wikipedia. Let me just pause it, actually. This is Wikipedia, where everything is generated on the fly, even the HTML. So this is actually being-- oops-- this is actually being generated by the model on the fly. So it's a web page with the HTML and the text and everything being generated on the fly. So it looks like regular Wikipedia. And when you click on it, the latency is low enough that it can just fill in the page as if it was a real Wikipedia page. So that's Wikipedia generated on the fly by a very low latency model. We have a similar thing where we did it for Reddit. So now all the responses to your posts will be by bots. They weren't already. Yeah. And it's generating fake comments. The Gemini diffusion model was not an image-generated model, so this demo links in the -- which was our state-of-the-art image-generator model at the time. which I think was Juno, was it before Nano Banana, So it's the two of these models working together to fill in the web page. The image generation model is a little slower. MR. But you can see that it's like, you know, you can... invent any Reddit you want, sharks in this case, and it'll generate the page with the text, the images follow a minute later, And then you can interact with this website as if it was a real website with real users and so on. All the comments, all the images, all the HTML, everything is being generated entirely on the fly here. This is being generated by the model. I love this one. This is my favorite one. This is... an operating system also being entirely generated on the fly. So every click here is generating the next page of the operating system. So it looks like a real operating system, but it's all been generated by the model. on the fly, responding to every click, So every time you enter the readme, it generates the text, the web pages you can do, go back to the desktop and so on. I think this-- Okay. And then this is a demo from someone on Twitter who used the who use the Gemini diffusion API. sorry, not the API, the web page, to do some vibe coding with his voice So I really like this one. Create a to-do app. Add 10 random to-dos. Allow Todo's to have a completed state. Mark four random to-dos as completed. Allow me to sort to-dos by name and by state. Alright, let's see if this works. Sort by name, sort by state, testing, enter, it was added to the bottom. We'll try deleting a few. Let's add one. Everything's working. Please convert this to dark mode. And this was literally 15 seconds of work. Okay, so that was someone outside of our team, so he could say that. Yeah, the Vibe coding by voice. But just in general, I think that, you know, low latency models can really unlock some new experiences. for users and new products. And so we're excited to see what people will do when the next generation comes out. Okay, and on that note, thank you very much. [20:00] Questions? Thank you. Yeah, we use all the same data. I mean, the algorithms have to change a bit, but we use all the same data. Thank you. may be uh... most? You can distill them, yeah. I'm not sure if there are any published ones. . I don't think so. Maybe there's some externally. really have-- - Thank you. by the Thank you. There's a, yeah, so we're going to release something soon. Thank you. Yeah. Thank you. . Uh, Yeah, so there's a, the bigger models tend to require less Steps. for the same output. So they kind of Even if the model is getting bigger and the flops per forward pass are getting bigger, They tend to reduce the the forward passes they need. So you kind of have some sort of diminishing cost of serving even the biggest models. Have you... - Hi. Thank you. I don't know. We haven't got to that yet. I'm a research scientist. MR PRICE: Yeah. I just the answer. Yeah. You expect that I have this image and I have it out. Yeah. frame. in causing for text. So there's a few different ways. The easiest way is to just-- fixed some some window length. And then just iterate on that. So it's like autoregressive-- blockwise autoregressive. kind of a. standard way to do it. But you can have a head that will predict the length of the response and stuff like that if you want to. But you have to... in order to No, it still can generate unlimited texts. but it's just-- If you fix a window length, then it just does that window length auto aggressively if it needs to generate many, many windows. of text. Yeah. Thank you. being Yeah. Could you ahead of time limit. Yeah. for both of them. Yeah. Hold on. understand what late season. to be ahead of time. Mm-hmm. Yeah, yeah, these are all with a limit, but they just finish it with earlier than the limit. Thank you. Windows. may then go back and attend to-- previous one. Yeah. Yes. I mean, you could potentially, but for us we just -- set it in stone. Continue. Yeah. That's the question is here. Thank you. like a focus. Yeah, yeah, that's how it works. Thank you, Ray. violent. Vision mode. Oh, no, so it's... Prefill is the same, it's just that you've got some context and you prefill. And then after that, the generation step is typically in blocks of some fixed size. like 512 or 1,000 or 32 or whatever you want. And then that's auto-aggressive. Yeah. I get that. MR PALLADINO: Yeah. So you can do that. The easiest way to do it is to just have like a logits at the top, just vanilla. prediction head at the top. Well, so this is all like discrete diffusion, right? So it's always tokens in, tokens out. Like if I show you the-- For every step, yeah. So if I go back here, it's always like a discrete corruption process. And then you fill in a discrete token back in. So it's always... So you're always in a discrete space. But you can do it in latent spaces. And people have done that. But most of the text diffusion models in literature is discrete diffusion today. There might be one day. . Yeah. What's your outlook? one of the architectures or... that they have. [25:00] I think for now they have different use cases. So, you know, if you think about what a low latency model provides, that's worse throughput. So what's that trade-off? Is on-device applications. So we are in a couple of on-device applications already within the Alphabet ecosystem. So robotics, things like that, where you want to run a model on the device itself, so your phone or a robot or whatever. But you want it to be low latency. And you're not batching with thousands of other queries, like Gemini being served in the server side. So you want the lowest latency model. Quality isn't really any, they're the same quality, basically. So then you may as well pick the low latency one, because you don't have the throughput concerns. What do you mean, like, in the future, like, can we get quality up to the bar with the current, like, proxy models? Quality isn't the concern. It's the throughput for serving in the big batch setting. Yeah. Yeah. Yeah. So currently, you are getting more or less specific quality of front-term models in your research? Yeah, yeah. So what about reasoning tasks? Because we need to differentiate very well. Sorry? What about reasoning tasks? Is it possible to differentiate very well? Yeah, yeah. Yeah, you can do RL. Yeah, you just need to change the algorithm, but you can still do it. Yeah. Yeah? Is there much scope for combining diffusion and also aggressive text models Yeah you can do that Yeah Yeah Yeah Can you like initialize with a smaller model so you don't have to start with noise so you can have to do this? The problem with that is if you train it with noise, it expects noise. You'd have to train it with this more model as you like the output. It just adds complexity. So we don't usually do that. Yeah? Is there a paper, like a certain version that you have to learn more about? Well, not from our team, but there's a bunch of literature out there, yeah. Which are . Yeah. You get the idea, I think. Yeah. Yeah. Any other questions? There's a lot of questions. I've gone through them all. That's good. Oh, one more. OK. Just a random one. What happens if the graph gets generated? I don't know. I don't think we've tried to do that. Probably would work. We'll probably do something. Yeah. Cool. OK. Thanks, everybody. Thank you.