AI Engineer
While text diffusion models cut latency tenfold by reducing memory bandwidth transfers, high serving costs at large batch sizes restrict their deployment to single-user edge applications like robotics.
AI-assisted summary of Text Diffusion — Brendan O’Donoghue, Google DeepMind — original source. Check the original for full context.
Source published:
Key ideas
- Google DeepMind's Gemini Diffusion preview demonstrates that text diffusion can reach end-to-end client generation speeds of roughly 2,000 tokens per second by cutting memory transfers tenfold during batch-size-one inference. (6:13)
- Unlike autoregressive models that get trapped by early token commitments, text diffusion uses bidirectional attention across its canvas to iteratively revise answers and intermediate steps as downstream reasoning resolves. (8:47)
- Despite achieving up to 10x lower latency in single-stream environments, text diffusion remains expensive for centralized web-scale chatbots because its poor throughput at large batch sizes drives up aggregate serving costs. (4:02)
- Alphabet has already deployed text diffusion models into production for edge and local workloads—such as robotics and on-device smartphone tasks—where batch size is naturally one and low latency is paramount. (25:00)
Detailed notes
## Background This talk was delivered by Brendan O’Donoghue, a research scientist at Google DeepMind, and published on YouTube by the AI Engineer channel on June 4, 2026. According to the page description, the presentation details why text diffusion provides a fast alternative to autoregressive models—requiring ~10x fewer memory transfers at 256 tokens—and explores trade-offs, bidirectional self-correction, and applications unlocked by 2,000 tokens-per-second generation.
## TL;DR O’Donoghue introduces discrete text diffusion, contrasting its parallel denoising passes across a full token canvas against the left-to-right next-token prediction of standard autoregressive architectures. He explains that text diffusion achieves up to 2,000 tokens per second at batch size one because it drastically reduces memory-bandwidth-bound weight streaming from HBM, though its compute intensity makes it expensive to serve under high-concurrency batching. The talk highlights unique architectural advantages including bidirectional reasoning, self-correction, dynamic and adaptive computation budgets, and surgical in-place text editing. Finally, O’Donoghue showcases interactive demos—such as real-time generated Wikipedia pages, a dynamic operating system, and voice-coded web applications—before clarifying in Q&A how blockwise generation, edge deployments like robotics and phones, and reinforcement learning apply to diffusion models.
## Atmosphere & style The presentation carries the engaging, grounded tone of an engineer speaking directly to peers, punctuated by pragmatic insights and dry humor about modern AI timelines. O'Donoghue presents complex architectural trade-offs between memory bandwidth and throughput with clarity and candid transparency, openly noting the serving costs that prevent large labs from replacing autoregressive systems at web scale. The atmosphere transitions smoothly from technical accelerator diagrams to dynamic live demos that elicit genuine curiosity, creating the feeling of a pragmatic peek behind the curtain at alternative paradigms actively evolving within frontier research labs.
## Bulletin - [06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) Google DeepMind's Gemini Diffusion preview demonstrates that text diffusion can reach end-to-end client generation speeds of roughly 2,000 tokens per second by cutting memory transfers tenfold during batch-size-one inference. - [08:47](https://www.youtube.com/watch?v=r305-aQTaU0&t=527s) Unlike autoregressive models that get trapped by early token commitments, text diffusion uses bidirectional attention across its canvas to iteratively revise answers and intermediate steps as downstream reasoning resolves. - [04:02](https://www.youtube.com/watch?v=r305-aQTaU0&t=242s) Despite achieving up to 10x lower latency in single-stream environments, text diffusion remains expensive for centralized web-scale chatbots because its poor throughput at large batch sizes drives up aggregate serving costs. - [25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Alphabet has already deployed text diffusion models into production for edge and local workloads—such as robotics and on-device smartphone tasks—where batch size is naturally one and low latency is paramount.
## Key takeaways - [00:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=0s) Text diffusion adapts continuous diffusion principles to language by applying discrete corruption to clean token sequences during training and iteratively refining noise canvases during inference. - [02:06](https://www.youtube.com/watch?v=r305-aQTaU0&t=126s) The Gemini Diffusion research preview was branched from Gemini 2.0 Flash to replace next-token prediction with diffusion, matching general task quality while substantially cutting latency. - [04:02](https://www.youtube.com/watch?v=r305-aQTaU0&t=242s) The primary barrier preventing large-scale commercial deployment of text diffusion is its poor serving throughput under heavy batch concurrency compared to amortized autoregressive models. - [06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) In single-stream autoregressive serving, generating 256 tokens requires 256 memory bus transfers from HBM to Tensor Cores, whereas text diffusion generates the same canvas in about 24 denoising passes. - [06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) Reducing memory bandwidth round trips by roughly 10x enables text diffusion to deliver up to 2,000 tokens per second into client browsers. - [08:47](https://www.youtube.com/watch?v=r305-aQTaU0&t=527s) Bidirectional attention eliminates causal masking, allowing diffusion models to retroactively correct mistakes in preceding tokens after finishing downstream calculations. - [10:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=600s) When solving multi-step arithmetic, Gemini Diffusion initially guessed 60, revised to 49, and finalized on 39 once it calculated 36 + 3, whereas GPT-4o and Gemini 2.5 Flash hallucinated or had to apologize for early missteps. - [12:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=720s) Dynamic computation provides predictable scaling at inference time, where allocating additional forward passes consistently improves performance across coding and reasoning benchmarks. - [12:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=720s) Adaptive computation allows the model to autonomously exit denoising early on simple or memorized sequences (like generating digits of pi in 4 steps) while devoting more steps to complex topics. - [14:26](https://www.youtube.com/watch?v=r305-aQTaU0&t=866s) Native in-place text editing allows targeted bug fixes, comment insertions, and paragraph bridging without needing to rewrite or regenerate surrounding context. - [16:09](https://www.youtube.com/watch?v=r305-aQTaU0&t=969s) Radical latency reductions enable dynamic software paradigms, including on-the-fly generated encyclopedias, fully simulated forum threads, dynamic operating systems, and voice-coded web apps generated in 15 seconds. - [20:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1200s) Text diffusion models train on standard pre-training corpora using adapted corruption schedules, and scaling model parameters decreases the total number of denoising passes required per generation. - [21:40](https://www.youtube.com/watch?v=r305-aQTaU0&t=1300s) Generating arbitrary output lengths relies on blockwise autoregressive generation, which denoises fixed windows (e.g., 512 tokens) and locks them into context for subsequent blocks. - [23:44](https://www.youtube.com/watch?v=r305-aQTaU0&t=1424s) Production architectures rely on discrete token diffusion with classification logits rather than continuous latent autoencoders. - [25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Standard post-training techniques like reinforcement learning transfer directly to text diffusion once policy updates are adjusted for multi-step denoising trajectories.
## Detailed walkthrough ### [00:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=0s) How Text Diffusion Works (Training and Inference)
Brendan O'Donoghue, a research scientist at Google DeepMind, opens his presentation as audience members are still trickling into the room, noting that the opening slides are primarily introductory. He frames text diffusion as a forward-looking research domain within DeepMind. While the broader community is already well-acquainted with image, video, and audio diffusion as the modern state-of-the-art—where a model is trained to reverse noise added to ground-truth media and then generates outputs at inference time by gradually refining a canvas initialized from pure random noise—the exact same underlying principle translates directly to natural language.
In text diffusion, training begins with an uncorrupted sequence of clean tokens, such as a well-formed sentence. Noise is systematically introduced through various corruption schemes. Although this corruption can theoretically occur continuously or discretely, discrete corruption is the standard mental model: tokens are either randomized or swapped for other random discrete vocabulary tokens across a spectrum of noise levels. A neural network is then trained to repair these corrupted sequences, predicting missing pieces and correcting errors across the entire context window.
During inference, generation flips this corruption on its head. Instead of writing text linearly from left to right, the entire sequence canvas is initialized with pure discrete noise—random tokens sampled directly from the vocabulary. Over a set of successive refinement iterations, the model simultaneously updates and denoises the canvas in whatever order it determines is optimal, progressively filling in information until a clean, fluent text output emerges. This dynamic visually mirrors image diffusion: starting from complete visual static, coherent sentences crystallize after several denoising steps.
```mermaid flowchart LR accTitle: Text Diffusion Training and Inference Workflow accDescr: Diagram showing how text diffusion corrupts data during training to learn denoising, and iteratively cleans random noise into coherent sentences during inference.
subgraph Training["Training Phase"] A["Clean Token Sequence"] -->|"Discrete corruption / noise"| B["Corrupted Token Canvas"] B -->|"Supervised objective"| C["Neural Network Predicts Corrections"] end
subgraph Inference["Inference Phase"] D["Pure Random Discrete Noise"] -->|"Iterative refinement step 1"| E["Partially Denoised Canvas"] E -->|"Iterative refinement step N"| F["Clean Generated Sentence"] end ```
---
[02:06](https://www.youtube.com/watch?v=r305-aQTaU0&t=126s) Gemini Diffusion Research Preview vs. Autoregressive Models
Reflecting on past milestones, O'Donoghue highlights the Gemini Diffusion research preview released roughly a year prior. It was branched off the Gemini 2.0 Flash architecture as an experimental variant designed to swap standard autoregressive next-token prediction for text diffusion, and it was opened to approximately 100,000 users. Benchmarking at the time indicated that Gemini Diffusion matched Gemini 2.0 Flash across general tasks, showing a slight advantage in coding evaluations and minor deficits in other areas, while delivering dramatically lower latency. However, O'Donoghue cautions against over-indexing on those specific benchmarks, observing wryly that a single year in modern machine learning represents “prehistoric times.”
To clarify the paradigm shift, O'Donoghue contrasts the operational mechanics of autoregressive architectures against diffusion models: - **Autoregressive models (e.g., standard Gemini, Gemma, GPT):** Text generation proceeds strictly token by token. Conditioned on an incoming context prompt, the model samples the first token, appends it to the context, and uses that new sequence to sample the second token, continuing sequentially until reaching a stopping criterion. - **Text diffusion models:** When context arrives, the system allocates an entire target canvas—spanning hundreds or thousands of tokens depending on the configuration—and initializes it with random discrete noise. The network runs multiple refinement iterations over the entire block in parallel, allowing tokens across the whole sequence to attend to one another bidirectionally rather than strictly gazing backward.
---
[04:02](https://www.youtube.com/watch?v=r305-aQTaU0&t=242s) Pros, Cons, and the Throughput Bottleneck
Text diffusion offers distinct architectural advantages alongside a critical operational disadvantage that currently limits its broad commercial deployment. On the positive side, the standout benefit is vastly superior inference speed; diffusion delivers far higher tokens-per-second generation by capitalizing on GPU and TPU compute characteristics. Additionally, diffusion operates without causal masking, unlocking bidirectional attention across the whole token canvas. This architectural freedom enables self-correcting generation (a model can draft an answer, realize down the page that its working reasoning contradicts the answer, and alter its prior text) and dynamic or adaptive computation, where difficult prompts receive more denoising iterations while easy prompts exit early. Diffusion models also support direct in-place editing, such as fixing trailing tokens and prompting the model to infill or regenerate the matching prefix.
The central roadblock holding text diffusion back from replacing industry standard autoregressive systems is its throughput profile under massive concurrency. Autoregressive models are slow for a single stream, but they amortize hardware costs effectively: large batches of incoming queries can be packed together across tensor cores, maximizing overall throughput and making cost per query viable at web scale. Because diffusion requires multiple complete forward passes over the exact same canvas data just to generate a single response, it hits compute thresholds prematurely. Consequently, text diffusion exhibits remarkably low latency for an individual user but poor aggregate throughput for high-concurrency environments, driving up serving costs. O'Donoghue points directly to contemporary platform struggles to illustrate why labs are cautious: “And right now, if you've played with Claude recently, you'll know that they have some throughput concerns, so people really care about throughput right now. And so no one's landing text diffusion into any of these big models, primarily because of that disadvantage. It's just too expensive to serve, even if it is much lower latency.”
---
[06:13](https://www.youtube.com/watch?v=r305-aQTaU0&t=373s) Hardware Efficiency: Why Text Diffusion is Faster
Explaining why text diffusion achieves radically lower latency requires examining modern accelerator topology across GPUs and TPUs. Accelerators contain high-performance execution blocks like Tensor Cores designed for dense matrix multiplications, providing massive floating-point operations per second (FLOPS). However, model parameters, dynamic activations, and Key-Value (KV) caches reside inside High Bandwidth Memory (HBM). To execute a forward pass, weights and data must be streamed from HBM into the compute cores across a physical bandwidth bus. In modern hardware, adding raw FLOPS to silicon is relatively cheap, but scaling memory bus bandwidth is difficult and costly. As a result, accelerator architectures present an acute imbalance: abundant compute capacity constrained by a tight memory bandwidth channel.
```mermaid flowchart TD accTitle: Accelerator Memory Bandwidth Bottleneck accDescr: Flow diagram showing how weights and KV cache transfer from HBM over a constrained bus to tensor cores for computation.
subgraph Accelerator["GPU / TPU Architecture"] HBM["High Bandwidth Memory (HBM)<br>• Model Weights<br>• Dynamic Activations<br>• KV Cache"] Bus["Tight Memory Bus Channel<br>(Primary Bottleneck)"] Compute["Tensor Cores<br>(Abundant Compute / High FLOPS)"]
HBM -->|"Must stream data repeatedly"| Bus Bus -->|"Constrained transfer rate"| Compute end ```
When running single-stream (batch size one) autoregressive generation, serving is severely memory-bound. Generating each successive token demands streaming the entire model's weights and cumulative KV cache out of HBM into the cores. Generating 256 tokens autoregressively therefore requires 256 individual read cycles over the entire model parameter footprint. In contrast, a text diffusion model generating a 256-token canvas might only require 24 iterative denoising passes. While each forward pass streams the model weights, performing 24 passes rather than 256 yields roughly a tenfold reduction in overall memory transfers:
| Architectural Metric | Autoregressive Next-Token Model | Text Diffusion Model | |:--- |:--- |:--- | | **Tokens Generated** | 256 tokens | 256 tokens | | **Model Forward Passes** | 256 passes (1 token per pass) | 24 passes (canvas denoised in parallel) | | **Full Memory Weight Transfers** | 256 streaming transfers from HBM | 24 streaming transfers from HBM | | **Memory Bandwidth Efficiency** | Heavily memory-bandwidth bottlenecked | ~10x fewer data roundtrips over the memory bus | | **Observed Latency Profile** | High per-token latency at batch size 1 | Up to ~10x lower latency when memory bound |
Under genuine memory-bound operating regimes, this ~10x drop in bus traversals translates directly into an equivalent speedup in response latency. O'Donoghue notes that the Gemini Diffusion preview reliably hit approximately 2,000 tokens per second for real users. This 2,000 tokens/sec figure was not an idealized synthetic metric; it measured end-to-end delivery right into the client browser, fully accounting for the initial context prefill phase. While short single-token outputs remain dominated by prefill overhead, longer output sequences allow the system to fully realize this massive generational speed advantage.
---
[08:47](https://www.youtube.com/watch?v=r305-aQTaU0&t=527s) Bidirectional Reasoning and Self-Correction
Moving past raw latency metrics, O'Donoghue turns to the unique algorithmic properties of bidirectional attention within text diffusion, highlighting capabilities that linear next-token autoregressive models fundamentally cannot replicate. To demonstrate, he references a math problem showcased at Google I/O: evaluating an arithmetic expression with nested fractions and operations, specifically:
$$\sqrt{81} \times \left(\frac{2}{3}\right)^2 + \dots = 39$$
When standard autoregressive models tackle such prompts, they are forced to commit causally to every preceding token. If their early calculations drift off track, they become trapped by their own context history, frequently hallucinating or doubling down on erroneous reasoning steps to remain consistent with their previous tokens.
Gemini Diffusion behaves entirely differently due to its bidirectional canvas. After just a single initial forward pass over the noisy sequence, the model stakes out a rough overall structure: it writes `answer = 60`, which is incorrect, and immediately starts drafting out an explanatory derivation beneath it (`solution: calculate the square root of 81...`). By the second forward pass, as the intermediate calculations on the canvas begin to take shape, the model revises its top-level prediction from `60` down to `49`, updating its forward-looking draft on the fly as it works through the problem.
- In conventional causal generation, early mistakes propagate downstream because attention cannot reach ahead into future reasoning steps. - In diffusion generation, the entire response layout—both the final conclusion and the intermediate reasoning steps—co-evolves across multiple passes, allowing the network to continuously refine its answers as downstream details resolve.
[10:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=600s) Bidirectional Reasoning and Self-Correction
Brendan O’Donoghue walks through the mechanics of how the text diffusion model solves a multi-step arithmetic problem across forward passes. In an early forward pass, the model reaches five steps into the derivation ($2^2 = 4$), with several uncertain "blue" tokens still destined to flip. By the third pass, it has successfully worked through the full chain of logic to calculate that $36 + 3 = 39$. Because diffusion models enjoy bidirectional attention across the entire sequence rather than strictly predicting left-to-right, the model can observe its own downstream reasoning and go back to overwrite its earlier attempts. In this run, the initial response token had mistakenly predicted $60$, then corrected to $49$, before finally fixing the answer at the beginning of the text to $39$ once the full derivation resolved. A couple of subsequent forward passes then clean up remaining token artifacts.
```mermaid flowchart LR accTitle: Bidirectional Self-Correction in Diffusion Models accDescr: Diagram contrasting how diffusion corrects mistakes retroactively with autoregressive left-to-right compounding errors. subgraph DIFF["Diffusion Model (Bidirectional)"] direction TB D1["Pass 1: Guesses '60'"] --> D2["Pass 2: Adjusts to '49'"] D2 --> D3["Pass 3: Computes '36 + 3 = 39'"] D3 -->|"Retroactively corrects initial token"| D4["Final Output: '39'"] end subgraph AR["Autoregressive Model (Causal)"] direction TB A1["Outputs initial token: '40' or '42'"] --> A2["Generates reasoning left-to-right"] A2 --> A3["Forces error: '36 + 3 = 42'<br/>or issues awkward apology"] end ```
O’Donoghue emphasizes that this capacity for bidirectional reasoning and self-correction exposes a fundamental flaw in traditional autoregressive architectures. When fed the identical prompt, leading autoregressive models from that period—both GPT-4o and Gemini 2.5 Flash, which were vastly larger than the Gemini Diffusion research preview—stumbled: - GPT-4o guessed $40$ up front because that was its highest-probability single-token prediction without looking ahead; after generating the full chain of logic and finding $39$, it was forced to awkwardly apologize in-line ("Sorry, I made a mistake... It's 39, not 40"). - Gemini 2.5 Flash initially guessed $42$, refused to backtrack, and hallucinated its arithmetic downstream to preserve continuity, asserting that $36 + 3 = 42$. - Modern thinking models handle this via explicit scratchpads, but O'Donoghue observes: “And, you know, you can fix this with modern reasoning thinking models, but, you know, then you just kind of punt in the problem into something else.”
[12:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=720s) Dynamic and Adaptive Computation
Beyond bidirectional revision, text diffusion provides two distinct computational advantages: dynamic computation and adaptive computation. Dynamic computation allows the user to allocate an arbitrary compute budget at inference time. Supplying more denoising steps (additional forward passes) yields an almost monotonic improvement in performance across evaluation benchmarks. Even if an early draft is mostly complete, granting the model extra passes gives it the opportunity to spot edge-case errors and refine token assignments. Internal testing tracking six coding benchmarks revealed a consistent, smooth curve showing output quality climbing directly with the step count.
Adaptive computation operates in reverse: the model is trained to decide autonomously when an answer has converged and stop denoising, spending minimal compute on simple prompts and reserving deep computation for difficult ones. O'Donoghue illustrates this dynamic behavior with examples from the Gemini Diffusion model: - Memorized/simple tasks: Generating the first 100 digits of $\pi$ requires 100 tokens, yet the model completes it in only 4 denoising steps because the sequence is readily recalled from weights. An autoregressive model would have produced only 4 tokens in that same time budget. - Moderate tasks: Writing standard code like FizzBuzz required roughly 18 forward passes. - Complex tasks: Explaining quantum mechanics in a single concise paragraph required 31 denoising steps, as the model spent more iterations settling on phrasing and conceptual accuracy. - Benchmark scaling: Across evaluations, easy suites like MBPP (mostly basic Python programs) completed rapidly, whereas challenging benchmarks like GPQA Diamond required substantially longer compute budgets, fully determined by the model's self-assessed convergence.
[14:26](https://www.youtube.com/watch?v=r305-aQTaU0&t=866s) In-place Text Editing
O'Donoghue highlights fast in-place text editing as another capability native to diffusion models. In computer vision, diffusion models perform inpainting by observing the surrounding unmasked context to cleanly reconstruct cut-out sections—unlike autoregressive image generators that are forced to proceed in strict left-to-right, raster order. The identical mechanism applies directly to text diffusion. Because the network processes all sequence positions simultaneously, it can manipulate specific token spans without regenerating surrounding content from scratch.
In live demonstrations, O'Donoghue showcases several in-place workflows: - Targeted bug fixing: Presenting a snippet of code with the command "There's a bug in this code. Can you fix it?" prompts the model to denoise and rewrite strictly the buggy lines while leaving adjacent code untouched. - In-place documentation: Inserting docstrings and explanatory comments directly into existing function blocks without re-emitting the entire script. - Narrative insertion: Providing the first and third paragraphs of a story and asking the model to insert an intermediate paragraph; the model leverages full bidirectional visibility of both the preceding and following prose to craft a contextually seamless bridge.
[16:09](https://www.youtube.com/watch?v=r305-aQTaU0&t=969s) Low-Latency Applications and Demos
The most profound practical differentiator of text diffusion is ultra-low latency. Rather than simply serving familiar text interfaces slightly faster, radical speedups unlock entirely new software paradigms. O'Donoghue shares several internal prototypes and community experiments showcasing dynamic generation:
- Real-time generated Wikipedia: An interactive encyclopedia prototype where clicking any hyperlink generates the entire target article on the fly—including complete HTML structure, body copy, and styling—fast enough to feel like browsing live web servers. - Dynamic Reddit clone: A simulated forum generating synthetic community threads, bot discussions, and commentary on the fly for arbitrary user-invented topics (such as a subreddit dedicated to sharks). The interface combines text diffusion with Google's image model (Juno) to dynamically populate images alongside text, though the image model lags slightly behind the near-instant text generation. - The Generative Operating System: O'Donoghue's personal favorite demo, an entire desktop operating system where every click dynamically generates the next state: “So it looks like a real operating system, but it's all been generated by the model on the fly, responding to every click.” Clicking folders, opening a README file, or launching browser views renders the corresponding OS interface in real time. - Voice-driven "vibe coding": A community project by a developer on X (Twitter) using the Gemini Diffusion web interface. In a recorded clip, the developer speaks instructions aloud to construct a functioning task manager: > “Create a to-do app. Add 10 random to-dos. Allow to-dos to have a completed state. Mark four random to-dos as completed. Allow me to sort to-dos by name and by state... Please convert this to dark mode.” The model generated, updated, and re-skinned the interactive web application end-to-end in approximately 15 seconds.
O'Donoghue closes the presentation by reiterating to the audience of AI engineers that ultra-low latency will redefine user software experiences, inviting the developer community to explore these affordances as next-generation text diffusion models roll out from the team.
[20:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1200s) Q&A: Training Data, Distillation, and Step Scaling in Larger Models
Brendan O'Donoghue opens the floor to audience questions, first addressing how text diffusion models are trained relative to standard autoregressive architectures. He confirms that the data pipelines remain essentially identical, explaining that while the underlying algorithms and corruption objectives have to be adapted for diffusion, the team feeds the models the exact same training data. When an audience member asks about distillation techniques, O'Donoghue notes that text diffusion models can indeed be distilled, though he is not certain whether any formal papers on the topic have been published externally yet, hinting that the team plans to release work in this area soon.
The discussion shifts to how model scale affects diffusion efficiency. O'Donoghue points out an advantageous scaling property: as parameter counts grow, models actually need fewer denoising passes to reach the same final text output. Even though each individual forward pass becomes computationally heavier in terms of FLOPs, the total number of required steps drops significantly, creating what he describes as a diminishing marginal cost when serving larger models.
- Training uses standard text corpora; only the training algorithm and corruption schedule change. - Larger parameter models require fewer diffusion steps for equivalent output quality compared to smaller variants. - While per-step FLOPs rise with model size, the reduction in total forward passes offsets server evaluation costs.
[21:40](https://www.youtube.com/watch?v=r305-aQTaU0&t=1300s) Blockwise Autoregressive Generation and Context Windows
An audience member asks how text diffusion models handle arbitrary output lengths and whether they are locked to a rigid maximum token budget. O'Donoghue explains that generating text across long horizons typically relies on a hybrid setup known as blockwise autoregressive generation. Rather than trying to diffuse an infinite sequence all at once, the system fixes a specific block length—such as 32, 512, or 1,024 tokens—diffuses that block across its denoising steps, and then treats that generated block as standard context to predict the subsequent block.
When asked whether models must know the output length in advance, O'Donoghue clarifies that models can either use an auxiliary prediction head to guess the exact response length or operate within a fixed upper limit and terminate early. Addressing questions about whether newly generated windows can go back and modify previously generated blocks, he notes that while bidirectional cross-block editing is theoretically possible, in practice the generated context is typically locked in place once finalized, functioning similarly to standard autoregressive history.
```mermaid flowchart LR accTitle: Blockwise Autoregressive Generation accDescr: Text diffusion processes input context, diffuses fixed-size token blocks, and appends them to autoregressive history. subgraph Inputs ["Input Stage"] A["Context Prompt"] end subgraph DiffusionBlock ["Fixed-Window Denoising"] B["Noisy Mask Tokens"] -->|"24 Denoising Steps"| C["Clean Block (e.g. 512 tokens)"] end subgraph OutputHistory ["Autoregressive Chain"] D["Lock Output into History"] --> E["Next Block Denoising"] end A --> B C --> D ```
- Standard generation applies diffusion within fixed-size sequence windows (for instance, 32, 512, or 1,024 tokens). - The prefill phase functions identically to autoregressive models; the generation step then runs blockwise. - Models operate with a token budget limit but retain the ability to finish and stop before hitting that threshold.
[23:44](https://www.youtube.com/watch?v=r305-aQTaU0&t=1424s) Discrete Diffusion vs. Latent Space Representation
A questioner inquires whether the diffusion process takes place in a continuous latent space or directly over token vocabularies. O'Donoghue explains that their implementation relies on discrete diffusion. At every denoising step, discrete tokens are corrupted or masked, and the model directly outputs discrete vocabulary tokens via standard prediction logits at the top layer.
He acknowledges that continuous latent-space text diffusion exists in academic literature, but notes that discrete token-level diffusion remains the predominant approach for state-of-the-art text models. Because tokens remain discrete throughout the process, the architecture relies on standard classification heads rather than continuous autoencoders.
> “Well, so this is all like discrete diffusion, right? So it's always tokens in, tokens out.” — Brendan O’Donoghue
[25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Deployment Niches: Robotics, Edge Devices, and RL Reasoning
Looking ahead at the architectural battle between autoregressive and diffusion models, O'Donoghue argues that they serve entirely different deployment regimes rather than competing for the exact same workloads. The major challenge for text diffusion is serving throughput at high batch sizes, making it expensive for massive centralized server farms like those powering consumer web chatbots. However, for single-stream, on-device tasks where queries are not batched with thousands of other users, text diffusion's raw low latency gives it a massive advantage.
O'Donoghue reveals that Google is already running text diffusion models in production on edge devices within the Alphabet ecosystem, specifically highlighting robotics and phone applications where local latency is paramount and batch size is effectively one. When attendees ask whether diffusion models can match frontier reasoning benchmarks or handle reinforcement learning, O'Donoghue answers affirmatively, explaining that standard post-training techniques like RL transfer over once the optimization objective is adapted to diffusion trajectories.
Addressing an audience proposal to speed up diffusion by initializing generation with a smaller draft model rather than pure noise, O'Donoghue cautions against the added training complexity: “The problem with that is if you train it with noise, it expects noise. You'd have to train it with this smaller model as you like the output. It just adds complexity. So we don't usually do that.” After answering a final lighthearted question about generating graph structures, O'Donoghue thanks the crowd and wraps up the session.
> “Quality isn't the concern. It's the throughput for serving in the big batch setting.” — Brendan O’Donoghue
- Text diffusion is already deployed in real-world on-device applications inside Alphabet, including robotics and smartphones. - For single-batch inference, diffusion matches autoregressive quality while eliminating sequential latency overhead. - Reinforcement learning (RL) reasoning pipelines work on text diffusion architectures by adjusting the RL policy update algorithms to the multi-step denoising path.
## Notable quotes > [25:00](https://www.youtube.com/watch?v=r305-aQTaU0&t=1500s) Quality isn't the concern. It's the throughput for serving in the big batch setting. — Brendan O’Donoghue
## Mentioned Brendan O'Donoghue, Google DeepMind, Gemini Diffusion, Gemini 2.0 Flash, Gemini 2.5 Flash, GPT-4o, Claude, Gemma, Juno, Google, Alphabet, Reddit, Wikipedia, MBPP, GPQA Diamond
## Video description GPT-4o answered 40. Gemini 2.5 Flash answered 42 and stuck to it even after working through the reasoning incorrectly. The Gemini Diffusion model, considerably smaller than both, answered 60 on the first forward pass, then 49, then corrected itself to 39 once it finished reasoning. Bidirectional attention means it can see future tokens and go back to fix mistakes. Autoregressive models cannot do that.
Brendon O'Donoghue covers why text diffusion is fast (24 denoising steps to generate 256 tokens means roughly 10x fewer memory transfers than autoregressive generation), what the tradeoff is (lower throughput at large batch sizes makes it expensive to serve at scale today), and what gets unlocked when latency drops to 2,000 tokens per second. The demos include a fake Wikipedia generated on the fly, a Reddit clone with AI generated comments and images, an operating system where every click generates the next screen, and a todo app built in 15 seconds by voice.
Speaker info: - https://x.com/bodonoghue85 - https://bodono.github.io/ - https://www.linkedin.com/in/bodono/
Timestamps: 0:00 Introduction to Text Diffusion 1:02 How Text Diffusion Works (Training and Inference) 2:06 Gemini Diffusion Research Preview 3:04 Difference Between Autoregressive and Diffusion Models 4:02 Pros and Cons of Text Diffusion 6:13 Hardware Efficiency: Why Text Diffusion is Faster 8:47 Bidirectional Reasoning and Self-Correction 12:00 Dynamic and Adaptive Computation 14:26 In-place Text Editing 16:09 Low Latency Applications and Demos 20:05 Q&A Session
---
Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/r305-aQTaU0-text-diffusion-brendan-o-donoghue-google-deepmind
Join our upcoming AI Engineer conferences (see https://ai.engineer landing page):
- AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney
---
[Source](https://www.youtube.com/watch?v=r305-aQTaU0)