Panasonic Cuts Diffusion Model Latency 100x to Power Edge AI Agents

Shortcutting Time and Depth in Diffusion Models for Faster Agentic Applications with Panasonic — Ai4

Ai4

Pruning denoising iterations alongside transformer layer depth achieves a hundredfold latency reduction, allowing diffusion architectures to replace costly autoregressive models in real-time robotics and user-interface agents.

AI-assisted summary of Shortcutting Time and Depth in Diffusion Models for Faster Agentic Applications with Panasonic — original source. Check the original for full context.

Source published: 2026-08-21

Detailed notes

## Background Published on YouTube by Ai4 on August 21, 2026, this talk features Denis Gudovskiy, Distinguished AI Engineer at Panasonic North America. According to the description, the session explores few-step generation methods that lower the sampling complexity of diffusion models and details their architectural integration into next-generation AI agents across language, video, audio, robotics, and computer-use workflows.

## TL;DR Denis Gudovskiy explains why diffusion models provide superior parallel generation, error correction, and continuous signal processing compared to standard autoregressive LLMs. He highlights how diffusion-based action heads power Vision-Language-Action (VLA) models and Computer Use Agents (CUAs), as well as video-driven World Action Models (WAMs) like NVIDIA DreamZero for zero-shot physical planning. To solve the edge latency bottleneck of multi-step denoising integrals, Panasonic combines time-discretization consistency distillation with depth pruning across transformer layers, achieving up to a 100x reduction in latency and memory footprint with minimal quality loss. Finally, he outlines the strategic landscape between autoregressive LLMs, discrete diffusion LLMs, VLAs, and WAMs, advocating for lightweight, domain-optimized diffusion agents on edge hardware.

## Atmosphere & style The presentation has the crisp, authoritative tone of an advanced industrial AI research briefing, grounded in practical engineering rather than speculative hype. Gudovskiy speaks with understated, methodical confidence, balancing complex algorithmic concepts like differential equations and consistency distillation with tangible product constraints across cameras, laptops, and mobile devices. Light moments of dry humor—such as noting AI-generated cats across social media—punctuate an otherwise tightly structured, technically rigorous session focused on real-world compute efficiency.

## Key takeaways - [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) Diffusion models provide three key advantages over autoregressive LLMs: parallel token generation, inherent error correction through iterative denoising, and native support for continuous multimodal signals. - [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) Discrete diffusion language models (dLLMs) demonstrate that pure diffusion can effectively challenge transformer-based autoregressive models on discrete text generation tasks. - [03:46](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=226s) Vision-Language-Action (VLA) models combine multimodal LLM backbones with diffusion-based action experts to generate high-frequency continuous control trajectories for robotics. - [03:46](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=226s) Physical embodiment frameworks transfer directly to Computer Use Agents (CUAs) by swapping robotic action heads for UI coordinate heads without retraining the core language backbone. - [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) World Action Models (WAMs) replace LLM backbones with video diffusion engines to perform joint rollout, simultaneously forecasting continuous actions and future visual states. - [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) Architectures like NVIDIA DreamZero achieve zero-shot physical manipulation by operating in a predictive visual-action latent space that serves as a grounded world simulator. - [09:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=554s) Industrial and edge applications on domain-specific hardware require lightweight visual agents optimized for high execution reliability rather than massive, costly autoregressive web models. - [11:13](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=673s) The primary computational bottleneck in diffusion inference is not the neural network forward pass, but the iterative numerical integration across dozens or hundreds of denoising time steps. - [13:41](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=821s) Consistency distillation trains student models under self-consistency constraints to jump across sampling trajectories in just one to few steps using pre-trained teacher checkpoints. - [15:34](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=934s) Simultaneously pruning denoising iterations (time) and transformer block depth (architecture) achieves up to a 100x latency and memory reduction with negligible perceptual quality degradation. - [18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) The evolving agentic architecture landscape spans autoregressive LLMs, hybrid VLAs, discrete diffusion models, and WAMs, each presenting distinct trade-offs between reasoning capability, training cost, and continuous control efficiency.

## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=4s) Introduction and Panasonic AI Lab's Research Focus

Denis Gudovskiy introduces the talk, framing its core focus around accelerating agentic applications powered by diffusion models. Reflecting on his career, he notes that he joined the Panasonic AI Lab in Mountain View a decade ago, bringing over ten years of experience developing deep learning and machine learning systems. Within Panasonic, the lab's mandate balances internal artificial intelligence engineering efforts across multiple regional offices with external academic partnerships, actively collaborating with research teams at universities such as Arizona State University and UC Berkeley. These joint academic initiatives primarily focus on efficient AI, robust model architectures, and edge deployment.

- Denis Gudovskiy has worked at Panasonic AI Lab in Mountain View for over 10 years. - Research partnerships focus heavily on academic collaborations with UC Berkeley and Arizona institutions. - Core research pillars: computational efficiency, robustness, and deploying models across distributed edge hardware.

---

[01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) Why Diffusion Models Over Autoregressive LLMs

Addressing the foundational question of why the field is leaning into diffusion architectures, Gudovskiy explains that diffusion models can now augment or entirely replace conventional autoregressive large language models (such as ChatGPT, Claude, and similar systems) across numerous data modalities. This shift stems from three structural advantages: 1. **Parallel token generation**: Autoregressive models must decode text sequentially token by token, whereas diffusion models perform parallel token generation across an entire sequence. 2. **Inherent error correction**: The iterative denoising framework acts as built-in quality control, progressively refining intermediate representations rather than compounding sequential errors. 3. **Native multimodality**: Autoregressive models excel when dealing with discrete tokens like text, but struggle with continuous sensory signals such as time series, robot kinematics, images, and video feeds. Diffusion operates natively in both discrete and continuous state spaces.

The progression of diffusion models has moved rapidly from text-to-image generators like Stable Diffusion to large-scale video systems—acknowledging OpenAI's Sora alongside the broader ecosystem of Silicon Valley startups generating synthetic video (humorously noting how "the cats in the Facebook are now generated with the video generation models"). Beyond continuous media, the industry saw a major push toward discrete diffusion models (DLMs), spearheaded by researchers from Inception Labs and Stanford University, demonstrating that pure diffusion could challenge transformers even on discrete text generation tasks.

```mermaid flowchart TD accTitle: Comparison between Autoregressive LLMs and Diffusion Models subgraph AR["Autoregressive LLMs (e.g., GPT, Claude)"] A1["Discrete Text Tokens"] --> A2["Sequential Generation (Token-by-Token)"] A2 --> A3["Error Compounding / No Inherent Rollback"] end

subgraph Diff["Diffusion Models"] B1["Continuous & Discrete Signals (Sensors, Vision, Actions)"] --> B2["Parallel Generation across Latents"] B2 --> B3["Iterative Denoising (Built-in Quality Control & Correction)"] end ```

---

[03:46](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=226s) Vision-Language-Action Models (VLAs) and UI Agent Adaptation

While discrete diffusion language models represent an exciting theoretical branch, the most transformative advancements over the past six months have emerged in robotics through Vision-Language-Action Models (VLAs). A standard VLA pairs a multimodal vision-language model (VLM) backbone—containing vision and text encoders feeding into an LLM—with a dedicated, diffusion-based action expert. While the upstream VLM processes high-level camera frames, state vectors, and natural language prompts, the downstream diffusion expert generates high-frequency continuous action trajectories, enabling robotic arms to perform dexterous physical manipulations. Gudovskiy highlights the accessibility of these architectures, pointing to open-source foundations like Hugging Face's SmolVLA / LeRobot initiatives that release end-to-end training pipelines, datasets, and checkpoints.

The architectural synergy between robotics and computer use agents (CUAs) is immediately striking: both systems demand dexterous control in an environment devoid of clean APIs or command-line interfaces, relying entirely on visual perception and continuous coordinate control. A computer agent observes raw screen captures and must output precise mouse movements, drags, and clicks. Citing recent work such as ShowUI-π, Gudovskiy explains how researchers adapted small VLA pipelines by swapping the robotic arm action expert for an interface action expert. Crucially, this UI agent requires no fine-tuning or pre-training of the core language backbone—only the diffusion-based action head is adapted, underscoring how seamlessly physical embodiment transfers to graphical user interface manipulation.

```mermaid flowchart LR accTitle: VLA Architecture and Transfer to Computer Use Agents subgraph Perception["Multimodal Backbone (VLM)"] direction TB Inp["Visual Frame + User Command"] --> Enc["Vision & Text Encoders"] Enc --> LLM["Multimodal LLM Backbone"] end

subgraph ActionHeads["Diffusion-Based Action Experts"] direction TB LLM --> Latent["Latent Action Conditioning"] Latent --> RobHead["Robotics Head: Continuous Joint Actions"] Latent --> UIHead["UI Head: Screen Clicks & Cursor Trajectories"] end ```

---

[07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) World Action Models (WAMs) and NVIDIA DreamZero

Over the past few months, the robotics frontier has shifted beyond VLAs toward World Action Models (WAMs). Rather than routing vision through a standard autoregressive LLM to feed a discrete action head, a WAM entirely replaces the LLM backbone with a unified diffusion-based video generation engine. The core breakthrough of a WAM lies in joint rollout: the model simultaneously generates continuous actions and hallucinates the future visual frames that those actions will produce. Predicting multiple minutes into the future dramatically boosts environmental robustness and out-of-distribution generalization.

Gudovskiy highlights NVIDIA’s DreamZero framework as a landmark example of this paradigm. In DreamZero, the robot exhibits zero-shot generalization on completely unseen physical tasks—performing manipulation sequences it never encountered during training. Because the underlying image-to-video diffusion backbone plans entirely within a predictive visual-action latent space, it functions as a grounded simulator of the physical world, making WAMs "the next frontier model for robotics."

- World Action Models replace the conventional autoregressive LLM core with an image-to-video diffusion architecture. - Joint prediction: WAMs forecast both future environment frames and continuous control trajectories simultaneously. - Long-horizon capability: State-of-the-art models predict several minutes into the future to maintain planning stability. - Zero-shot task execution: Architectures like NVIDIA DreamZero enable robots to solve manipulation tasks without task-specific prior training data.

---

[09:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=554s) Deploying UI Agents to Domain-Specific Edge Hardware

Translating these embodied paradigms to Panasonic’s business needs brings immediate utility to graphical UI agents. Panasonic manufactures a vast spectrum of physical hardware—ranging from laptops and smart televisions to commercial security cameras—each running specialized, proprietary onboard interfaces. Unlike generic web-browsing agents that require broad internet knowledge, internal industrial applications demand 100% execution robustness across a strictly constrained set of domain-specific workflows, all while operating under stringent compute budgets on edge chips.

Gudovskiy demonstrates working prototypes of these lightweight UI agents in real-world scenarios: - **Mobile photo management**: A smartphone-based agent navigates a local photo gallery, filters and selects designated images based on natural language instructions, and composes an email attachment. - **Developer task automation**: A desktop UI agent opens GitHub, navigates into a specific code repository, generates a structured bug report, and submits an issue directly through the visual web interface without using external APIs.

By avoiding large-scale, high-parameter autoregressive models and utilizing visual diffusion foundations, these agents achieve reliable multi-step tool execution directly on client hardware.

---

[11:13](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=673s) The Computational Challenge of Denoising Integrals

Shifting to the technical mechanics of model acceleration, Gudovskiy breaks down how diffusion models function mathematically at inference time. Diffusion models bridge two probability distributions: a complex ground-truth data distribution (such as natural images of cats or robot control sequences) and an unstructured Gaussian noise distribution. Training teaches the model to approximate the trajectory between these distributions via physics-inspired differential equations.

At inference time, diffusion networks differ fundamentally from standard CNNs or LLMs because they are intrinsically time-dependent: - Conventional neural networks process inputs in a static forward pass without time encoding. - Diffusion backbones integrate explicit time-step embedding layers that track the current noise level $t$. - Data synthesis is formulated as a numerical integration problem across continuous time, starting from pure noise and stepping backward along the reverse denoising trajectory.

The underlying neural network architecture itself remains flexible—relying on convolutional networks, multi-layer perceptrons, vision transformers (DiTs), or hybrid CNN-transformer blocks. However, the bottleneck on edge devices is almost never the single forward pass of the backbone; rather, it is the iterative numerical integral itself. Performing dozens or hundreds of sequential neural network evaluations to denoise a single sample introduces unacceptable latency and compute costs, making trajectory shortcutting and step reduction the central engineering challenge for real-time agentic applications.

[13:41](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=821s) Time Discretization and Few-Step Consistency Distillation

Traditional continuous diffusion models rely on integrating continuous differential equations, which incurs massive computational overhead. To reduce this complexity, standard practice first discretizes the continuous trajectory into a fixed series of time steps—replacing continuous integration with a discrete summation. For example, breaking the trajectory into 100 steps and summing the outputs sequentially produces a sufficiently high-quality image. However, researchers have pushed to determine whether generation can be achieved with even fewer steps, culminating in the recent surge of few-step and single-step generation methods for images, video, and multimodal data.

```mermaid flowchart LR accTitle: Teacher-Student Consistency Distillation accDescription: Workflow illustrating how a teacher diffusion model transfers fast sampling capabilities to a student model via consistency constraints.

subgraph Teacher ["Teacher Model Pipeline"] A["Teacher Model (e.g., Stable Diffusion)"] --> B["Discretized Trajectory (100+ Steps)"] end

subgraph Distillation ["Distillation Objective"] B --> C["Self-Consistency Constraint"] C --> D["Student Model Fine-Tuning"] end

subgraph Inference ["Fast Inference"] D --> E["1-to-Few Step Generation"] end ```

The underlying mechanism relies on training a student model under self-consistency constraints anchored to a pre-trained teacher model, such as Stable Diffusion. By copying the teacher architecture and enforcing that points along the same sampling trajectory map to the same origin, the student learns to jump directly across the trajectory in just one or two steps. Fine-tuning a student in this consistency framework requires relatively few training steps given sufficient data, yielding dramatic reductions in generative latency.

- Time discretization replaces continuous differential equation integration with discrete summations over 100 or more steps. - Consistency models constrain intermediate trajectory points to map consistently to the final output, reducing sampling steps down to 1–2 iterations. - Distillation fine-tuning leverages existing pre-trained teacher checkpoints without requiring full architectural redesign from scratch.

---

[15:34](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=934s) Shortcutting Time and Depth for 100x Latency Reduction

While standard consistency models shortcut only the time domain, Panasonic's optimization framework introduces consistency constraints across both the temporal domain and the neural network's depth domain. In a standard diffusion transformer architecture with 24 blocks (such as in a Diffusion Transformer / DiT backbone), the network can be pruned dynamically down to 18, 15, or fewer layers. By training the network to remain consistent across reduced layer depths as well as reduced time iterations, practitioners obtain fine-grained control over the complexity-quality trade-off.

```mermaid flowchart TD accTitle: 2D Complexity Optimization: Time vs Depth accDescription: Diagram showing simultaneous pruning of denoising iterations and transformer layer depth.

subgraph DenoisingDim ["Temporal Dimension (Iterations)"] T1["Vanilla: 128 Steps"] --> T2["Intermediate: 4 Steps"] --> T3["Extreme: 1 Step"] end

subgraph DepthDim ["Depth Dimension (Layers)"] D1["Full: 24-12 Blocks"] --> D2["Pruned: 8 Blocks"] --> D3["Minimal: 4 Blocks"] end

subgraph Output ["Target Operational Point"] T2 & D2 --> O["Sweet Spot: ~100x Latency & Memory Reduction (Negligible Visual Degradation)"] end ```

Cutting both iterations (e.g., from 128 steps down to 4 or 1) and transformer blocks (e.g., from 12 down to 8 or 4) can yield up to a "100x latency reduction" alongside substantial memory savings. While aggressive pruning in both dimensions simultaneously can cause noticeable visual degradation, there is a wide operational sweet spot where the output is virtually indistinguishable to the human eye from an uncompressed baseline that requires 100 times more compute. Furthermore, because this joint time-depth optimization is solver-agnostic, it reliably cuts generation times from minutes or tens of seconds down to sub-second latencies suitable for real-time applications.

- **Depth pruning:** Dynamically skips or permanently prunes transformer blocks (e.g., from 24 down to 18, 15, 8, or 4 layers). - **Latency reduction:** Generates complete images in one second or less compared to vanilla multi-step pipelines taking multiple minutes. - **Memory footprint:** Shortcutting depth directly reduces the active parameter footprint and activation cache in memory.

---

[18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) The Next-Gen Agentic Architecture Landscape: LLMs, VLAs, dLLMs, and WAMs

The rapid evolution of agentic AI has sparked an active architectural debate on whether future systems should rely on standard autoregressive LLMs/VLMs, hybrid Vision-Language-Action (VLA) models, Discrete Diffusion LLMs (dLLMs), or World Action Models (WAMs). Conventional autoregressive LLMs continue to scale well for computer-use and general agentic tasks, but training them costs "hundreds of billions of dollars." Crucially, autoregressive models struggle with continuous sensorimotor trajectories and visual data, and they lack the fine-grained complexity-quality trade-offs inherent to iterative diffusion denoising.

Hybrid VLA frameworks extend the utility of autoregressive foundation models by pairing them with diffusion-based action experts, enabling effective control in robotics and Computer Use Agents (CUAs). In parallel, discrete diffusion LLMs (dLLMs) provide massive speedups via parallel token generation, though their high-level reasoning capabilities remain under active evaluation. World Action Models (WAMs) show exceptional promise for zero-shot environmental generalization, but their optimization remains nascent compared to established VLA setups. Ultimately, specialized, domain-tailored agents built on diffusion backbones can deliver robust real-world execution at a fraction of the inference and operational cost of monolithic frontier models.

## Notable quotes > [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) the cats in the Facebook are now generated with the video generation models — Denis Gudovskiy

> [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) the next frontier model for robotics — Denis Gudovskiy

> [15:34](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=934s) 100x latency reduction — Denis Gudovskiy

> [18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) hundreds of billions of dollars — Denis Gudovskiy

## Mentioned Denis Gudovskiy, Panasonic, Panasonic AI Lab, Ai4, ChatGPT, Claude, Stable Diffusion, Sora, OpenAI, Facebook, Inception Labs, Stanford University, Arizona State University, UC Berkeley, SmolVLA, LeRobot, Hugging Face, ShowUI-π, DreamZero, NVIDIA, GitHub

## Video description Denis Gudovskiy - Distinguished AI Engineer, Panasonic North America

Diffusion models have emerged as a powerful alternative to conventional autoregressive LLMs, offering fast parallel token generation, built-in quality control, and native multi-modality support. However, their iterative denoising process remains computationally expensive for cost-effective deployments. This talk will detail recent few-step generation methods that substantially reduce sampling complexity. I will then bridge these technical optimization strategies with the broader architectural shift of integrating diffusion models into next-generation AI agents across language, video, audio, and robotics. Attendees will walk away with an understanding of recent trends in computationally efficient AI architectures, preparing them to build and deploy the next wave of diffusion-based VLAs (Vision-Language-Action Models), WAMs (World Action Models), and Computer Use Agents (CUAs).

[Source](https://www.youtube.com/watch?v=W-6bXvmG0ao)

中文简报

松下将扩散模型延迟降低100倍，赋能边缘AI智能体通过在修剪去噪迭代次数的同时削减Transformer层深度，实现了百倍的延迟降低，使扩散架构能够在实时机器人和用户界面智能体中取代成本高昂的自回归模型。 Denis Gudovskiy 阐述了相比标准自回归 LLM，扩散模型为何在并行生成、错误纠正和连续信号处理方面更具优势。他强调了基于扩散的动作头如何赋能视觉-语言-动作（VLA）模型和计算机使用智能体（CUA），以及诸如 NVIDIA DreamZero 等用于零样本物理规划的视频驱动世界动作模型（WAM）。为解决多步去噪积分带来的边缘延迟瓶颈，松下将时间离散化一致性蒸馏与 Transformer 层的深度剪枝相结合，在质量损失极小的情况下将延迟和内存占用降低了高达 100 倍。最后，他梳理了自回归 LLM、离散扩散 LLM、VLA 与 WAM 之间的战略格局，倡导在边缘硬件上部署轻量级、针对特定领域优化的扩散智能体。相比自回归 LLM，扩散模型具备三大核心优势：并行 Token 生成、通过迭代去噪实现的固有错误纠正，以及对连续多模态信号的原生支持。 (1:20) 离散扩散语言模型（dLLM）表明，在离散文本生成任务中，纯扩散模型能够有效挑战基于 Transformer 的自回归模型。 (1:20) 视觉-语言-动作（VLA）模型将多模态 LLM 骨干网络与基于扩散的动作专家相结合，为机器人生成高频连续控制轨迹。 (3:46) 具身物理框架可直接迁移至计算机使用智能体（CUA），只需将机器人动作头替换为 UI 坐标头，无需重新训练核心语言骨干网络。 (3:46) 世界动作模型（WAM）使用视频扩散引擎取代 LLM 骨干网络来进行联合推演，能够同时预测连续动作和未来视觉状态。 (7:14)

Reading summary

Denis Gudovskiy of the Panasonic AI Lab argues that diffusion models offer critical advantages over autoregressive large language models for embodied and computer-use agents, and can achieve real-time edge deployment by jointly shortcutting denoising iterations and model depth. Structural Advantages Over Autoregressive Language Models (1:20) Gudovskiy identifies three primary technical reasons diffusion models can augment or replace standard autoregressive architectures in agentic pipelines: parallel token generation across complete sequences, inherent error correction through progressive denoising, and native handling of continuous data. While autoregressive transformers process discrete tokens sequentially and risk compounding errors, diffusion networks natively process both discrete text and continuous sensory signals, including video streams, time series, and kinematic trajectories. This continuous processing capacity enabled rapid transitions from text-to-image synthesis to video generation systems such as OpenAI's Sora. Concurrently, research from Stanford University and Inception Labs into discrete diffusion models (DLMs) demonstrates that non-autoregressive denoising can also challenge conventional transformers on discrete text tasks, avoiding the sequential decoding bottlenecks typical of models like ChatGPT and Claude. Unified Control for Robotics and Computer Interfaces (3:46) Recent breakthroughs in Vision-Language-Action (VLA) models combine multimodal vision-language backbones with diffusion-based action heads to generate high-frequency continuous control. Gudovskiy emphasizes that this paradigm transfers directly from physical robotics to Computer Use Agents (CUAs), as both environments lack uniform APIs and rely on continuous visual-spatial navigation. In systems like ShowUI-π, researchers swapped a robotic arm action head for a graphical interface head without retraining the underlying language backbone, demonstrating that physical manipulation and screen-based cursor interactions share identical latent control mechanics. At the robotics frontier, World Action Models (WAMs) like NVIDIA's DreamZero bypass the language backbone entirely in favor of image-to-video diffusion architectures. By jointly predicting continuous action trajectories and hallucinating future visual frames over multi-minute horizons, WAMs serve as grounded physical simulators capable of zero-shot task execution in unfamiliar environments. Accelerating Inference by Shortcutting Time and Depth (11:13) Deploying diffusion backbones on edge devices faces an inference bottleneck rooted in numerical differential equation integration rather than the parameter cost of a single forward pass. Standard continuous models require dozens to hundreds of iterative evaluations to synthesize samples from noise. While consistency distillation algorithms permit student models to compress trajectories into one or two sampling steps, Panasonic expands this framework across a second dimension: neural network depth. Panasonic’s joint optimization prunes both temporal iterations (reducing steps from 128 down to 4 or 1) and transformer blocks (reducing a 24-layer Diffusion Transformer down to 18, 15, 8, or 4 layers). Gudovskiy reports that simultaneous time-depth shortcutting delivers up to a 100x reduction in latency and memory footprint. This compression lowers sample generation times from minutes to under one second, reaching an operational sweet spot where output fidelity remains visually indistinguishable from uncompressed models. Edge Hardware Constraints and Architectural Trade-Offs (9:14) Panasonic’s commercial applications require running agents directly on constrained consumer and industrial hardware—including laptops, smart televisions, and security cameras—running proprietary graphical interfaces. Unlike broad web-browsing agents, internal industrial tasks require deterministic robustness across narrow workflows. Panasonic prototypes demonstrate this edge feasibility through mobile agents that organize gallery photos to send email attachments and desktop agents that visually navigate GitHub repositories to file structured bug reports without APIs. Evaluating the broader agentic landscape, Gudovskiy notes that while frontier autoregressive models incur astronomical training costs and struggle with continuous motor trajectories, diffusion alternatives present distinct trade-offs. Hybrid VLAs retain autoregressive latency bottlenecks in their language backbones, while standalone discrete diffusion LLMs and World Action Models remain in early developmental stages regarding complex reasoning benchmarks and mature distillation frameworks. While joint time-and-depth distillation enables real-time diffusion agents on edge hardware, the broader transition to World Action Models and discrete diffusion LLMs remains nascent, facing ongoing questions around reasoning depth, optimization maturity, and long-horizon stability.

精炼阅读摘要

松下人工智能实验室（Panasonic AI Lab）的Denis Gudovskiy指出，扩散模型凭借原生处理连续动作信号与内置纠错机制的优势，正在机器人与图形界面智能体中展现替代传统自回归模型的巨大潜力；通过在时间步数与网络深度两维度联合进行一致性蒸馏剪枝，可实现高达100倍的推理延迟削减，使复杂视觉控制模型能够在边缘硬件上实时运行。扩散模型对自回归架构的结构优势 (1:20) Gudovskiy阐述了扩散模型在多模态与智能体场景中挑战传统自回归大语言模型（如GPT、Claude等）的根本动因。传统自回归模型依赖逐个Token的串行解码，极易在长序列中累积不可逆的生成误差，且其设计本质上局限于离散文本，难以原生处理时间序列、传感器读数以及机器人关节角度等连续运动学信号。相比之下，扩散模型具备三大核心优势：全序列并行生成能力、通过逐步去噪实现的内建纠错与质量控制机制，以及跨离散与连续状态空间的原生多模态适应力。除了在文本生成图像（如Stable Diffusion）及视频合成领域的成熟应用外，来自Inception Labs和斯坦福大学等团队的研究还推动了离散扩散语言模型（dLLM）的发展，证明纯扩散范式同样可以在离散文本生成任务中与Transformer展开竞争。具身控制向端侧界面Agent的架构平移 (3:46) 在智能体应用落地层面，视觉-语言-动作模型（VLA）与计算机使用智能体（CUA）展现出强烈的技术协同性。标准的VLA系统将多模态视觉语言模型骨干与专门的扩散动作专家相挂载，前者解析高层指令与图像输入，后者输出高频连续轨迹以执行精准物理控制，如开源倡议SmolVLA和LeRobot所展示的流程。Gudovskiy指出，诸如ShowUI-π的研究表明，研发人员只需将机械臂动作专家替换为屏幕点击与光标移动专家，即可在无需重新微调底层语言模型的情况下，将物理控制能力无缝平移至图形界面操作中。针对松下自身的工业与消费电子需求（涵盖笔记本电脑、智能电视及商用安防摄像头等搭载专用板载界面的边缘设备），团队将此类轻量UI Agent部署于端侧。原型展示表明，智能体可在无外部API支持的受限环境下，高可靠地自主完成手机相册检索归档和GitHub提报缺陷等跨步骤任务，避免了依赖高成本庞大自回归模型带来的算力压力。世界动作模型与前瞻物理仿真 (7:14) 具身控制的前沿正在从混合VLA架构进一步延伸至世界动作模型（WAM）。WAM彻底舍弃了自回归语言模型内核，直接采用端到端的图生视频扩散架构。其核心突破在于动作与画面的联合展开（Joint Rollout）：模型在规划连续控制动作的同时，能够前瞻性地幻觉预测出执行该动作后数分钟内的未来环境视觉画面。 Gudovskiy以NVIDIA的DreamZero框架为例说明，由于底层的图生视频扩散骨干在预测性的视觉-动作潜空间内运行，模型实质上充当了具备物理接地的仿真器，从而赋予机器人无需针对特定任务训练即可执行全新操作序列的零样本（Zero-shot）泛化能力。演讲者因此将WAM视为机器人领域的下一个前沿模型。时间与深度双维剪枝实现百倍加速 (15:34) 尽管扩散模型优势显著，但其边缘部署面临迭代去噪积分带来的巨大计算瓶颈。与静态单次前向传播的神经网络不同，扩散网络在推理时必须沿时间步反向数值求解，评估几十甚至数百次神经网络才能生成单个样本。传统的一致性蒸馏（Consistency Distillation）通过约束采样轨迹映射至同一原点，仅在时间维度上将生成步数压缩至1到2步。松下提出的优化框架则进一步将一致性约束同时引入时间域与神经网络的深度域。在Diffusion Transformer（DiT）等架构中，除了将去噪迭代从128步压缩到4步或单步，还可将主干网络从24或12个模块动态或永久剪枝至8或4个模块。Gudovskiy指出，该联合优化方案能实现最高达100倍的延迟与显存削减，将原先长达数分钟的生成压缩至亚秒级，且在宽泛的操作平衡区间内，输出质量在人眼辨别下几乎无明显退化。在新一代智能体架构格局中，自回归模型面临训练成本高昂及难以原生拟合连续传感器信号的局限，而世界动作模型与离散扩散模型的优化蒸馏技术尚处起步阶段。专有端侧扩散模型证明了高效率、高鲁棒性本地执行的可行性，但如何在极端剪枝与超低步数下兼顾深度多步逻辑推理的严谨性，依然是该路线进一步扩展需直面的核心问题。

Detailed notes — zh

## Background 本内容由 Ai4 频道于 2026 年 8 月 21 日发布在 YouTube 平台。根据页面简介，松下北美杰出 AI 工程师 Denis Gudovskiy 围绕少步生成与深度剪枝等扩散模型前沿优化方法展开分享，探讨如何大幅降低采样复杂度，并将高效扩散架构融入下一代语言、视频、机器人视觉-语言-动作模型（VLA）、世界动作模型（WAM）及计算机操作智能体（CUA）中。 ## TL;DR 松下北美 AI 实验室的 Denis Gudovskiy 深入对比了扩散模型与传统自回归大语言模型，阐明扩散模型在并行 Token 生成、内置纠错以及原生连续多模态支持上的显著优势。演讲重点剖析了如何将具身智能中的 VLA 架构迁移至计算机操作智能体（CUA），以及去 LLM 化的世界动作模型（WAM）的前沿探索。针对扩散模型多步积分带来的延迟瓶颈，Denis 提出了融合一致性蒸馏的时间步压缩与网络深度联合裁剪技术，在保持生成质量的同时实现端到端推理延迟降低高达 100 倍。最后，他系统复盘了自回归 LLM、混合 VLA、离散扩散模型及 WAM 的架构权衡，为垂直领域低成本部署端侧智能体指明了技术路径。 ## Atmosphere & style 整个分享呈现出硬核务实、逻辑严密的工业界前沿技术讲座氛围。Denis 的讲解沉着从容且条理清晰，既有对微分方程与一致性蒸馏底层原理的严谨数学剖析，又穿插着“Facebook 猫咪视频”等生动风趣的行业洞察。现场节奏紧凑明快，从算法层面的联合剪枝平滑过渡到端侧硬件落地的工程痛点，带给观众一种兼具深厚学术功底与扎实工程落地视角的沉浸式听讲体验。 ## Key takeaways - [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) 扩散模型相较于自回归 LLM 具备三大核心优势：并行 Token 全局生成、基于迭代去噪的内置纠错与质量控制，以及对连续多模态信号（如传感器、视频、时间序列）的原生处理能力。 - [04:02](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=242s) 机器人具身智能广泛采用 VLA 架构，通过在视觉语言模型（VLM）主干上挂载扩散动作专家（Diffusion-based Action Expert），实现高精度的物理世界连续动作控制。 - [04:02](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=242s) VLA 架构可直接平移至计算机操作智能体（CUA），无需重训底层多模态大模型，仅需微调扩散动作专家即可实现对 UI 界面连续像素坐标的精准点击与拖拽交互。 - [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) 世界动作模型（WAM）彻底去除了中间自回归 LLM，直接利用扩散视频生成模型联合预测未来视频帧与动作轨迹，展现出优异的零样本物理任务泛化能力。 - [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) 松下积极推动专用领域端侧 UI 智能体落地，在低算力与极高鲁棒性约束下实现了手机相册智能管理与 GitHub 自动化提交等复杂工作流。 - [11:13](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=673s) 扩散模型推理本质上是在时间域上求解微分方程的反向去噪积分，其显式依赖时间步编码的多步数值迭代构成了端侧部署的核心计算延迟瓶颈。 - [13:41](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=821s) 基于一致性模型（Consistency Models）的自一致性约束蒸馏技术，成功将传统 100-200 步的迭代采样压缩至仅需 1 到 4 步即可生成高质量数据。 - [15:28](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=928s) 时间步数与网络深度的联合裁剪（Joint Shortcutting）方案在质量几乎无损的前提下，将端到端延迟降低高达 100 倍（达 1 秒以内），并显著缩减显存占用。 - [18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) 自回归 LLM 算力成本高昂且难以原生处理连续信号，而混合型 VLA 架构通过结合自回归理解与扩散动作生成，成为当前智能体落地的关键桥梁。 - [18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) 专用领域智能体（Domain-specific Agents）无需依赖庞大昂贵的通用大模型，借助轻量高效的扩散架构即可实现低成本、高可靠的端侧落地部署。 ## Detailed walkthrough ### [00:04](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=4s) 讲者背景与松下北美 AI 实验室的研究使命 Denis Gudovskiy 开门见山地指出，本次演讲的核心主题是加速依赖扩散模型（Diffusion Models）的智能体应用（Agentic Applications）。他回顾了自己的从业经历：早在十年前他就加入了位于加州山景城（Mountain View）的松下北美 AI 实验室（Panasonic AI Lab），在人工智能以及早期所谓的机器学习（ML）领域深耕已超过十年。松下北美 AI 实验室的核心使命一方面是与顶尖研究型大学开展深度学术合作，另一方面则是推进松下内部的自主 AI 技术研发。目前，Denis 负责主导并咨询松下全球多个分支办公室的跨团队工程项目。在学术合作方面，他目前主要与加州大学伯克利分校（UC Berkeley）以及亚利桑那州立大学（Arizona State University）的教授团队紧密合作，研究重点聚焦于高效 AI（Efficient AI）算法、高鲁棒性模型架构等前沿方向。 - **核心研究方向**：高效能计算（Efficient AI）、高鲁棒性模型（Robust Models）以及智能体落地部署。 - **合作高校**：加州大学伯克利分校（UC Berkeley）、亚利桑那州立大学（Arizona State University）。 - **产业定位**：结合学术界前沿突破与松下内部跨机构硬件及 AI 工程研发。 --- ### [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) 为什么选择扩散模型？对比自回归 LLM 的三大核心优势 Denis 提出了一个核心问题：为什么在当下我们需要高度重视扩散模型？他指出，在许多数据领域，扩散模型正在逐步增强甚至完全替代传统的自回归大语言模型（如 ChatGPT、Claude 等 Autoregressive LLMs）。这种范式转变主要归功于扩散模型具备的三大关键特性：第一是**快速的并行 Token 生成机制**（Parallel Token Generation）。传统自回归模型只能逐个生成 Token（Sequential Generation），而扩散模型能够在时间步内并行生成全部 Token。第二是**内置的纠错与质量控制能力**（Built-in Error Correction and Quality Control），这一特性直接源自其多步迭代去噪（Iterative Denoising）的数学机制。第三也是最关键的一点，在于其**对多模态的原生支持**（Native Multimodality Support）。自回归 LLM 依赖离散 Token（如文本切片），而现实世界中来自传感器的数据、时间序列、图像与视频本质上都是连续信号（Continuous Data），扩散模型能够同时原生处理连续与离散两种形态的数据，无需繁琐且失真的离散化过程。 ```mermaid flowchart TD accTitle: Autoregressive LLMs versus Diffusion Models Comparison subgraph AR["传统自回归模型 (如 ChatGPT / Claude)"] A1["离散 Token 输入"] --> A2["串行逐字解码 (Sequential)"] A2 --> A3["单向依赖 / 误差逐级累积"] end subgraph DM["扩散模型 (Diffusion Models)"] B1["离散与连续多模态输入"] --> B2["并行全局生成 (Parallel)"] B2 --> B3["多步迭代去噪 / 内置纠错控制"] end ``` 在产业演进上，扩散模型早已超越了早期的 Stable Diffusion 图像生成范式，进入了视频与离散文本生成领域。尽管 OpenAI 的 Sora 遗憾关闭，但硅谷涌现出大量基于扩散模型的视频生成初创公司——Denis 幽默地打趣道："现在 Facebook 上大部分猫咪视频其实都是用扩散视频模型生成的。" 此外，过去一年中由斯坦福大学背景的 Inception Labs 领衔推动的离散扩散模型（Discrete Diffusion Models, DLMs），也开始直接在离散文本生成任务上挑战自回归架构。 --- ### [04:02](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=242s) 机器人领域的 VLA 架构及其向计算机操作智能体（CUA）的迁移尽管离散扩散文本生成（DLMs）备受瞩目，但 Denis 强调过去半年中机器人（Robotics）领域的发展尤为迅猛。在具身智能中，典型的代表是视觉-语言-动作模型（Vision-Language-Action Models, VLAs）。VLA 的标准架构是在传统多模态大模型（VLM）的基础上，接入一个经过专门训练的、基于扩散模型的动作专家（Diffusion-based Action Expert）。机器人需要在物理世界中执行复杂的灵巧操作（Dexterous Manipulations），输出的必须是高精度的连续动作（Continuous Actions），而非离散文本词。VLA 将摄像头图像、操作指令和机器人当前状态输入 VLM 生成潜在表征，再由扩散动作专家解码为连续控制量。开源社区如 Hugging Face 推出的 SmallVLA 项目，已经完整开源了从数据集到模型架构的全套方案。 ```mermaid flowchart LR accTitle: VLA to CUA Model Architecture Transfer subgraph Inputs["输入感知"] I1["界面截图 / 摄像头图像"] I2["自然语言任务指令"] I3["当前状态 State"] end subgraph Backbone["多模态主干"] VLM["VLM 表征编码器"] end subgraph Expert["扩散动作专家"] DAE["扩散动作专家 (Diffusion Action Expert)"] end subgraph Outputs["执行端 Embodiment"] O1["物理机械臂连续轨迹 (Robotics)"] O2["UI 屏幕点击/滑动坐标 (CUA)"] end Inputs --> VLM VLM --> DAE DAE -.-> O1 DAE -.-> O2 ``` 更令人兴奋的是将 VLA 架构迁移至计算机操作智能体（Computer Use Agents, CUAs）。这类智能体不调用 CLI 命令行或后端 API，完全依靠屏幕截图和鼠标键盘操作与系统交互。正如近期开源的 ShowUI-Pi 所展示的那样，研究人员直接沿用 SmallVLA 架构，甚至完全不需要重新预训练或微调底层庞大的 LLM/VLM，仅需微调扩散动作专家，就能将机器人在物理世界的操作能力无缝迁移为在图形用户界面（UI）上的精确点击与拖拽。 - **VLA 构成**：多模态视觉语言模型（VLM Backbone）+ 扩散动作专家（Diffusion-based Action Expert）。 - **开源参考**：Hugging Face SmallVLA 开源生态。 - **跨模态迁移逻辑**：物理世界的连续机械臂操作与 UI 屏幕上的连续像素坐标交互具有高度的数学与行为同构性。 --- ### [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) 世界动作模型（WAMs）的兴起与松下端侧智能体实践在过去几个月中，机器人领域出现了一种比 VLA 更具颠覆性的新型范式——世界动作模型（World Action Models, WAMs）。WAM 彻底去除了中间层的自回归 LLM，直接采用基于扩散架构的图像转视频（Image-to-Video）生成模型作为核心骨干。与 VLA 相比，WAM 的核心优势在于它能够生成"未来的视频帧"（Future Frames）以及配套的动作轨迹。以 NVIDIA 的 DreamZero 架构为例，系统可以在潜在空间中预测未来数分钟的物理世界动态。这种对未来时空的推演能力赋予了模型强大的零样本泛化能力（Zero-shot Generalization），使其能够直接执行从未在现实训练中见过的全新物理任务。 ```mermaid flowchart TD accTitle: WAM Architecture and Panasonic Edge UI Agent Applications subgraph WAM["世界动作模型架构 (World Action Model)"] W1["当前帧与任务指令"] --> W2["扩散图像/视频生成骨干 (I2V Backbone)"] W2 --> W3["潜在空间联合生成"] W3 --> W4["未来视觉帧预测 (Future Frames)"] W3 --> W5["连续控制动作预测 (Actions)"] end subgraph EdgeApps["松下端侧专用智能体落地 (Edge Devices)"] E1["手机端相册管家 (智能筛选指定照片并邮件发送)"] E2["开发协同自动化 (自动定位 GitHub 仓库并提交 Issue)"] end WAM -.-> EdgeApps ``` 在松下内部，这项技术被高度关注并应用于工业与消费级硬件。松下生产制造海量硬件设备（涵盖笔记本电脑、电视机、安防监控摄像头等），这些设备运行着高度专用的领域内部界面（Domain-specific Interfaces）。在这些垂直场景下，智能体必须在极低计算复杂度（Low Complexity）的端侧硬件上运行，同时保持 100% 的绝对可靠性与鲁棒性。Denis 展示了松下目前开发的端侧专用 UI 智能体示例： 1. **手机相册管理任务**：智能体自主识别指令、进入本地图库筛选特定图像，并唤起邮件应用完成发送； 2. **开发流程自动化**：在 GitHub 界面中自主导航至指定代码仓库，自动填写并提交 Issue 工单。 --- ### [11:13](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=673s) 技术原理拆解：扩散模型推理机制与时间积分瓶颈 Denis 随后进入演讲的技术核心部分：如何从算法层面加速扩散模型，使其能够在算力受限的端侧设备上实现毫秒级快速响应。从数学本质来看，扩散模型是通过受物理学启发的微分方程（Differential Equations），学习从纯高斯噪声（Noise）或某一先验分布逐步还原生成目标真实数据分布（如高清猫咪图像）的反向去噪过程（Reverse Denoising Process）。在推理阶段，扩散模型与传统时间无关（Time-Agnostic）的 CNN 或 Transformer 有着根本不同：它必须包含专门的时间步编码层（Time-dependent Layers）来感知当前的去噪阶段 $t$。扩散模型的推理实质上是在时间域上的连续积分求解过程（Integration in Time Domain）。模型从纯噪声出发，每经过一个时间步长就剔除一部分噪声、恢复一部分语义特征，直至最终收敛为清晰图像。虽然其网络骨干可由 CNN、MLP、Transformer 或目前主流的 CNN-Transformer 混合架构搭建，但**多步迭代数值积分**是造成其推理延迟高、计算开销大的核心性能瓶颈所在。 - **数学本质**：基于随机微分方程/常微分方程（SDE/ODE）的数据分布反向去噪映射。 - **架构特性**：显式依赖时间步编码（Time-Dependent Architecture），骨干网络多采用 CNN 与 Transformer 的混合设计。 - **加速核心痛点**：时间域数值积分的高计算复杂度，迫切需要通过少步生成（Few-step Generation）等优化算法压缩采样步数。 ### [13:41](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=821s) 扩散模型的步数压缩：从连续积分到一致性蒸馏在扩散模型的采样过程中，标准的数学求解通常依赖于求解微分方程的连续积分，这带来了巨大的计算复杂度。最初的工程优化方式是进行时间维度的离散化处理（Time Discretization），将连续积分替换为累加求和：例如将轨迹切分成 100 到 200 个离散时间步，通过连续执行 100 次迭代输出相加，即可获得质量尚可的生成图像。然而，百步级别的迭代依然无法满足低延迟交互与高效部署的要求。近一年来备受行业瞩目的突破方向，是将原本数百步的去噪过程极限压缩至仅需一步或少数几步（One or Few Steps）即可完成图像、视频等多模态数据的生成，从而使计算复杂度呈现断崖式下降。这一类技术的核心机制通常基于一致性模型（Consistency Models）：首先选定一个成熟的教师模型（Teacher Model，如常规的 Stable Diffusion），复用其网络架构构建学生模型，并在离散的时间步上施加自一致性（Self-consistency）约束进行微调。微调过程并不需要海量的训练步数，只要在训练阶段引入这种简单的约束，就能训练出一个能在极短时间内完成高质量图像生成的轻量高效学生模型。 ```mermaid flowchart LR accTitle: Consistency Distillation Workflow subgraph S1["传统教师模型"] T["预训练扩散模型 (100-200 步去噪)"] end subgraph S2["微调约束"] C["时间离散化采样 + 自一致性损失 (Self-Consistency)"] end subgraph S3["极速学生模型"] S["少步/单步生成器 (1-4 步去噪)"] end T --> C C --> S ``` - **从积分到求和**：时间离散化将连续轨迹切片为约 100-200 步求和，提供基础图像质量。 - **少步生成突破**：一致性模型通过自一致性约束进行知识蒸馏，使采样复杂度下降数个数量级，仅需 1 到 2 步即可出图。 --- ### [15:28](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=928s) 时间与深度联合裁剪：实现百倍级推理加速与显存节省为了进一步拓展效率边界，团队提出了不仅在时间维度（Time Steps）、同时在神经网络深度维度（Neural Network Depth）施加一致性约束的全新联合压缩方法。传统架构可能在主干网络中堆叠了 24 个 Transformer 模块（Transformer Blocks），而在新方案中，可以灵活地根据需求剪裁并仅保留 18 或 15 个模块。这种在时间步数与网络层数上的双重自由度，赋予了工程师精细调节计算复杂度与生成质量平衡（Complexity-Quality Trade-off）的能力。在实际收益上，这种联合裁剪方法可以在生成质量几乎无损的前提下，将端到端实际延迟降低高达 100 倍（100x Latency Reduction），同时深度剪枝显著减小了模型的显存占用（Memory Size）。实验展示了从 128 步迭代压缩至 4 步甚至 1 步，以及将网络深度从 12 层压缩至 8 层或 4 层的消融效果：虽然极端压缩时间步和模块数会导致质量严重下滑，但只要在设计空间内选取合理的折中点，生成结果与耗费百倍计算量的基线模型在肉眼上几乎无法分辨差异。在实际延迟曲线上，原始扩散模型出图可能耗费数分钟，而新方案可以在 1 秒甚至更短的时间内完成高质量渲染，且对底层微分方程求解器（Solver）具有极高的通用兼容性。 | 优化维度 | 传统基准 (Vanilla Diffusion) | 时间与深度联合优化 (Shortcutting) | |:--- |:--- |:--- | | **采样步数 (Time Steps)** | 100 ~ 128 步迭代 | 1 ~ 4 步极速去噪 | | **网络深度 (Transformer Blocks)** | 完整层数 (如 24 / 12 层) | 灵活剪裁 (如 18 / 8 / 4 层) | | **端到端延迟 (Latency)** | 较高，可能耗时数分钟 | 降低达 **100x**，可达 ≤1 秒 | | **显存占用 (Memory Footprint)** | 庞大，受限于多层网络激活值 | 随层数削减显著下降 | --- ### [18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) 下一代智能体架构之辨：自回归 LLM、VLA、dLLM 与世界动作模型在迈向下一代智能体（Agentic Models）的技术路线上，学术界与工业界正在经历一场深刻的架构之争：究竟应该继续沿用纯自回归大语言模型（Autoregressive LLMs）、视觉语言模型（VLMs），还是转向全新的架构形态？传统的自回归模型在通用智能体交互（如电脑端操作 Computer Use Agents）中依旧展现出强大的 Scaling 规律，但每一次能力的提升都伴随着数千亿美元级别的算力狂飙；更致命的是，自回归模型在处理图像、传感器读数等连续信号（Continuous Data）时表现挣扎，且缺乏扩散模型所具备的迭代去噪控制能力，在复杂度与质量的动态调节上极为僵硬。为了弥合这一差距，当前工业界广泛采用视觉-语言-动作模型（VLA）这一混合架构，即在现有自回归 LLM 基础上挂载一个基于扩散模型的动作专家（Diffusion-based Action Expert），这种折中方案不仅延续了 LLM 既有推理生态的生命力，还在机器人控制和电脑操作智能体上展现了出色的动作规划能力。与此同时，更为激进的原生新架构也在快速演进：离散扩散大模型（Discrete Diffusion LLMs, dLLM）在并行生成速度上展现出显著优势，但其复杂逻辑推理能力是否能比肩成熟 LLM 仍有待验证；而世界动作模型（World Action Models, WAM）虽在零样本泛化（Zero-shot Generalization）上极具潜力，但目前仍处于早期雏形阶段，工程优化成熟度尚无法与 VLA 媲美。演讲在最后指出，专用领域智能体（Domain-specific Agents）完全可以借助此类高效扩散架构，以通用大模型极小一部分的成本实现稳健的落地部署。 - **自回归 LLM 的困境**：训练成本高昂，难以原生处理连续多模态信号，缺乏渐进式去噪的质量调节灵活性。 - **混合型 VLA 模型**：通过外挂扩散动作专家，为机器人与 CUA 场景下的传统自回归模型续命。 - **新兴架构探索**：离散扩散模型（dLLM）速度极快但推理能力存疑；世界动作模型（WAM）零样本泛化突出但优化技术尚处初级阶段。 ## Notable quotes > [00:04](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=4s) 松下北美 AI 实验室的核心使命一方面是与顶尖研究型大学开展深度学术合作，另一方面则是推进松下内部的自主 AI 技术研发。 — Denis Gudovskiy > [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) 在许多数据领域，扩散模型正在逐步增强甚至完全替代传统的自回归大语言模型。 — Denis Gudovskiy > [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) 扩散模型能够同时原生处理连续与离散两种形态的数据，无需繁琐且失真的离散化过程。 — Denis Gudovskiy > [01:20](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=80s) 现在 Facebook 上大部分猫咪视频其实都是用扩散视频模型生成的。 — Denis Gudovskiy > [04:02](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=242s) 机器人需要在物理世界中执行复杂的灵巧操作，输出的必须是高精度的连续动作，而非离散文本词。 — Denis Gudovskiy > [04:02](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=242s) 仅需微调扩散动作专家，就能将机器人在物理世界的操作能力无缝迁移为在图形用户界面（UI）上的精确点击与拖拽。 — Denis Gudovskiy > [07:14](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=434s) 这种对未来时空的推演能力赋予了模型强大的零样本泛化能力，使其能够直接执行从未在现实训练中见过的全新物理任务。 — Denis Gudovskiy > [11:13](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=673s) 扩散模型的推理实质上是在时间域上的连续积分求解过程。 — Denis Gudovskiy > [13:41](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=821s) 只要在训练阶段引入这种简单的约束，就能训练出一个能在极短时间内完成高质量图像生成的轻量高效学生模型。 — Denis Gudovskiy > [15:28](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=928s) 这种联合裁剪方法可以在生成质量几乎无损的前提下，将端到端实际延迟降低高达 100 倍，同时深度剪枝显著减小了模型的显存占用。 — Denis Gudovskiy > [18:09](https://www.youtube.com/watch?v=W-6bXvmG0ao&t=1089s) 专用领域智能体完全可以借助此类高效扩散架构，以通用大模型极小一部分的成本实现稳健的落地部署。 — Denis Gudovskiy ## Mentioned Denis Gudovskiy, 松下 (Panasonic), 松下北美 AI 实验室 (Panasonic AI Lab), 加州大学伯克利分校 (UC Berkeley), 亚利桑那州立大学 (Arizona State University), ChatGPT, Claude, OpenAI, Sora, Facebook, Inception Labs, Hugging Face, SmallVLA, ShowUI-Pi, NVIDIA, DreamZero, Stable Diffusion, GitHub ## Video description Denis Gudovskiy - Distinguished AI Engineer, Panasonic North America Diffusion models have emerged as a powerful alternative to conventional autoregressive LLMs, offering fast parallel token generation, built-in quality control, and native multi-modality support. However, their iterative denoising process remains computationally expensive for cost-effective deployments. This talk will detail recent few-step generation methods that substantially reduce sampling complexity. I will then bridge these technical optimization strategies with the broader architectural shift of integrating diffusion models into next-generation AI agents across language, video, audio, and robotics. Attendees will walk away with an understanding of recent trends in computationally efficient AI architectures, preparing them to build and deploy the next wave of diffusion-based VLAs (Vision-Language-Action Models), WAMs (World Action Models), and Computer Use Agents (CUAs). [Source](https://www.youtube.com/watch?v=W-6bXvmG0ao)

Source quotes

[0:04] Denis Gudovskiy: 松下北美 AI 实验室的核心使命一方面是与顶尖研究型大学开展深度学术合作，另一方面则是推进松下内部的自主 AI 技术研发。 [1:20] Denis Gudovskiy: 在许多数据领域，扩散模型正在逐步增强甚至完全替代传统的自回归大语言模型。 [1:20] Denis Gudovskiy: 扩散模型能够同时原生处理连续与离散两种形态的数据，无需繁琐且失真的离散化过程。 [1:20] Denis Gudovskiy: 现在 Facebook 上大部分猫咪视频其实都是用扩散视频模型生成的。 [4:02] Denis Gudovskiy: 机器人需要在物理世界中执行复杂的灵巧操作，输出的必须是高精度的连续动作，而非离散文本词。 [4:02] Denis Gudovskiy: 仅需微调扩散动作专家，就能将机器人在物理世界的操作能力无缝迁移为在图形用户界面（UI）上的精确点击与拖拽。 [7:14] Denis Gudovskiy: 这种对未来时空的推演能力赋予了模型强大的零样本泛化能力，使其能够直接执行从未在现实训练中见过的全新物理任务。 [11:13] Denis Gudovskiy: 扩散模型的推理实质上是在时间域上的连续积分求解过程。 [13:41] Denis Gudovskiy: 只要在训练阶段引入这种简单的约束，就能训练出一个能在极短时间内完成高质量图像生成的轻量高效学生模型。 [15:28] Denis Gudovskiy: 这种联合裁剪方法可以在生成质量几乎无损的前提下，将端到端实际延迟降低高达 100 倍，同时深度剪枝显著减小了模型的显存占用。 [18:09] Denis Gudovskiy: 专用领域智能体完全可以借助此类高效扩散架构，以通用大模型极小一部分的成本实现稳健的落地部署。

Source quotes

[1:20] Denis Gudovskiy: the cats in the Facebook are now generated with the video generation models [7:14] Denis Gudovskiy: the next frontier model for robotics [15:34] Denis Gudovskiy: 100x latency reduction [18:09] Denis Gudovskiy: hundreds of billions of dollars

Original transcript

[0:04] This presentation uh is about accelerating uh agentic applications that rely on diffusion [0:11] models. So I'm going to talk mostly about the diffusion models, not like the the other stuff. [0:18] And uh so yeah, let me uh yeah, just a brief introduction. Uh I joined Panasonic AI lab in [0:26] Mountain View and u 10 years ago. So I have uh more than 10 years of experience uh developing [0:33] uh AI and what we call earlier ML models. So and [clears throat] uh our mission is to collaborate [0:39] with top uh research universities uh as well as we have some internal uh AI development. So [0:46] uh I kind of consult or lead several uh uh engineering efforts uh across multiple office [0:53] uh in Panasonic. Uh could you go to the next slide please? Yes. Oops. Yeah, [1:00] I think you missed one. Yeah. So, currently I'm mostly working with the Arizona and Berkeley [1:07] universities with the these professors and our kind of topics are mostly related to efficient AI, [1:12] robust models and so on. And uh yeah, could you go to the next slide? [snorts] Uh so yeah, [1:20] like I said, why we are why we care about diffusion models these days? [1:24] uh and uh the the answer is that currently we can augment or completely replace uh auto reggressive [1:32] uh LLMs in many data domains uh due to several key u features. So first diffusion models have fast [1:41] parallel token generation versus sequential in auto reggressive models and uh by auto reggressive [1:48] models here I mean the conventional LLMs like chat GPT or clo or like some others. Uh also they have [1:56] a built built in error correction and quality control uh due to iterative den noising. Uh and [2:03] lastly and most importantly they natively support uh multi multimodality because they can work both [2:09] with the continuous and discrete tokens. Uh and here uh what I mean by the continuous and discrete [2:15] tokens is that discrete tokens are usually uh like a tokenized text into some discrete values while [2:22] uh like uh data that comes from the sensors like time series or images or videos are usually more [2:30] like continuous. So we cannot really tokenize them uh for conventional autogressive LLMs. Next slide [2:37] please. And uh yeah probably you're familiar with the stable diffusion that is uh basically [2:43] one of the examples of diffusion models and also uh we got uh Sora that was unfortunately recently [2:51] closed but there are many other uh startups in the uh in the valley that do video generation. [2:58] uh for example like uh most of the like you know the cats in the Facebook are now generated with [3:04] the uh video generation models uh that are relying on diffusion models also last year uh we've got a [3:12] big push uh to replace even text generation the discrete text generation using diffusion [3:18] models so there are so-called discrete diffusion models or DLMs uh that are pioneered by inception [3:24] labs from Stanford actually the one of the early the developers of the diffusion models. Uh and [3:31] initially I was thinking about talking about DLMs. Uh but I think in the past six months we've seen [3:37] a lot of uh additional uh interesting developments uh in the robotics. So and uh so this presentation [3:46] will be uh split between like uh trying to explain why we care about diffusion in the context of [3:52] uh robotics and agents and second the the acceleration of them. So uh yeah so for robotics [4:02] uh maybe you're familiar with the VAS. VLAS are vision language action models uh which basically [4:09] a conventional uh multimodel LLM which is called VLM that contains vision encoder text encoder [4:15] and all these tokens are fed into LLM uh and on top of that they have a trained diffusion based [4:23] uh action expert that generates continuous actions uh for robots. So here you can see the [4:30] uh the typical robot that makes some kind of dictious manipulations with the objects. So it [4:37] has to operate in this uh physical world like uh which means this kind of continuous actions [4:43] unlike the the the tokens from the LLM. So how we can uh build this uh and what is the VLA doing? [4:51] So it's very simple. So, V like I said just has a VLM that is um uh basically feeding this uh [4:59] uh takes as the input the images the the command and the state uh and then generate some kind of [5:06] latent representation for the VA which uh for VA's diffusion based action expert. So uh yeah [5:14] that's the very simple uh example how we can beat VA and actually you can take the hugenface small [5:20] VA project they open source everything starting from the data sets to the model itself and [5:28] uh and then basically we can ask how we can use this similar architecture to for example build the [5:34] computer use agents uh uh because the VA is kind of support multimodality from the from uh from the [5:43] beginning. So this computer agents that do not use CLI or API basically there is no like uh textual [5:51] input except the command are uh uh kind of very good for uh for like being based on VAS. So yeah [6:01] so essentially the question is can we replace the robotic emod robotic embodiment of the VA [6:07] with the UI actions for computer use actions. uh essentially we can take for example the screenshot [6:14] from the interface and generate this uh clicks or actions for uh this computer use actions and uh [6:20] surprisingly uh there's a recent paper called show show ui pi uh basically adopts this small VA and [6:27] what they do they basically replace this diffusion based action expert uh with the expert that can do [6:33] this clicks over the user interfaces and this is uh surprisingly well because uh even though they [6:40] don't really touch or pre-train or fine-tune the LLM. They only fine-tune the action expert and [6:48] uh uh yeah, so actually there is a good analogy uh between the robotics and the uh the UI agent [6:55] because eventually they do dexterious kind of manipulations either in the interface or the uh uh [7:00] for the robots. So yeah, it's a very good example how you can transfer one model like one embodiment [7:07] to another embodiment because this V approach is kind of uh quite universal. But now uh in the past [7:14] few months even more u uh interesting uh type of models appeared for robotics called world action [7:20] models. So what is the world action model? uh world action model actually completely replaces [7:26] the LLM with the uh diffusion based uh backbone that generates images from videos and the key [7:33] difference compared to VS it can generate future frames uh as well as the actions. So this kind of [7:40] uh approach allows you not only to increase like general generalization also the robustness because [7:47] you're kind of predicting to the future. So you kind of uh uh for example the recent Nvidia like [7:53] uh world action model can predict up to several minutes into the future which is uh really [8:00] important to uh for practical uh applications of the in the robotics. U next slide please. [8:12] Yeah. So here is the example from the uh uh dream zero architecture of Nvidia where the robot can [8:18] do unseen tasks. Uh so this kind of robot never uh experience like doing this uh tasks in real [8:26] life. So it means this uh generalization zero show generalization ability of world action models is [8:32] actually uh kind of the key differentiation from the VAS and that's why I think this approach will [8:39] be quite u uh impactful in the recent uh uh like uh maybe year. So and what is the architecture? [8:48] The architecture is very simple. So we have a kind of image to video generation backbone uh that uh [8:57] replaces the LLM and it allows you to generate actions and frames in this uh latent space. Uh [9:04] and uh yeah I think this is u uh the next frontier model for robotics. Uh could you go to the next [9:14] slide? But can we use this approach for uh UI agents like in like in the case of VAS? So here [9:22] uh so the the answer is yes. Uh and why we really need to care about this. Uh so here [9:30] in Panasonic we uh develop a lot of hardware uh and that hardware has a lot of uh interfaces like [9:38] internal internal interfaces. Uh usually they are very domain specific. So for example for certain [9:44] uh like uh applications like you need to support only few uh applications. So the agent should be [9:52] running over those uh applications with the 100% robustness but at the same time with the [9:58] low complexity uh because usually those interfaces are running on the edge devices like laptops or [10:04] like TVs or like a security cameras and so on. uh and uh yeah and uh here I show the example [10:12] uh how we develop this uh kind of uh UI agent uh on the for the phone application where this [10:20] uh in this phone our kind of domain specific uh agent can perform uh actions in the uh in the uh [10:30] gallery. So we can take select certain pictures from the gallery and send to the certain email or [10:37] we have a application where we do some uh actions in the GitHub space. So for example we want to [10:44] uh open the u uh issue and then uh kind of uh write down uh this uh issue on the GitHub and [10:51] the specific uh repo. So yeah, so here I kind of try to motivate motivate you to go beyond [10:59] the conventional lms and look at the VLAS and the world action models that are specifically will be [11:05] like uh useful for robotics, UI agents and other and other multimodel applications and uh okay so [11:13] now I go into the second technical part of this uh talk uh how to accelerate uh diffusion models u in [11:21] practice uh because like I said we want to run them on the edge devices. Uh ideally uh uh just [11:28] a simple background. So these diffusion models uh are trained to generate data from noise or another [11:36] data using some kind of differential equation. So I'm not going to go into the details technical [11:42] details of this differential equations uh because they're kind of physics inspired. They're quite [11:47] complicated. So what we need to know here is that we have a two u data distributions. one is uh [11:54] our actual data for example this uh cat or on the other side we have a noise. So we want to train a [12:01] model that will uh using the uh that will generate image from the noise using this uh re uh reverse [12:08] uh den noising process and uh yeah that's what we uh are looking for uh and uh for the I'm not going [12:17] to talk about the training but for the inference part essentially what we have in diffusion models [12:23] is the uh like architecture that is uh time dependent. So unlike the conventional LLMs or [12:30] like CNN's uh that are time agnostic here we have additional uh layers that uh encode the time and [12:38] why we do this we want to generate data using [clears throat] the integration. So integration [12:43] basically means that we uh take the as initial step noise and then we uh integrate in time domain [12:50] uh to generate data. So here is the we take some kind of uh noise and maybe at each step we add uh [12:57] we generate a little bit more of a cat and then eventually we completely den noiseise this uh [13:04] uh cat and then we have a perfect image. But the architecture itself is quite um uh conventional. [13:11] So it could be CNN, it could be uh MLPS, it could be transformers. uh and uh currently most of the [13:18] people using kind of hybrid approaches where you kind of combine CNN and the transformer at the [13:24] same time uh so from the uh acceleration point of view this um integral is the most critical [13:32] one because it's a computationally challenging so how we can generate uh an image from the noise uh [13:41] uh using the integration but with less complexity. So first uh what people do they just do uh time [13:46] discretization. So instead of doing this continuous integration you can replace it [13:50] with the summation. So for example we discretize time and then we uh chunk this uh trajectory for [13:57] example 200 steps and then we just sum the outputs 100 times and uh we got a good enough image. But [14:05] can we do better than that? So there are a number of approaches that try to minimize number of time [14:10] steps and so eventually we get only one or few steps to generate the image or video or some [14:16] other multimodel data and uh so I think this uh became very popular last from last year because [14:24] we found that instead of doing hundreds of steps we can do only one or two steps. So the complexity [14:31] decreases dramatically and if you're interested in this you can look at the reference below. [14:37] Uh okay. So how does it work? Uh basically we take the conventional kind of uh uh teacher model. [14:45] U for example we can take stable diffusion or something similar and then we uh basically uh copy [14:52] the same architecture and uh train the student by imposing self-consistency at this uh discretized [14:59] time steps. So it means that basically we can fine-tune the model uh and finetuning doesn't take [15:05] a lot of uh uh steps uh of course you need data uh but uh just using this kind of simple additional [15:12] uh uh constraint during training you can kind of uh uh basically learn a student that can do [15:19] uh uh image generation uh much faster. [clears throat] So yeah uh I think this kind of [15:28] consistency type of models became quite popular these days. Uh and also in addition to that [15:34] uh in our approach we uh uh impose consistency not only in the time domain but also in the depth of [15:41] the neural network domain. So for example we can have like u 24 transformer blocks in the itl but [15:50] instead we can for example cut it and I use maybe 18 or 15 or so. So uh in this approach we're kind [15:57] of flexible in selecting not uh only how many time steps we can do but also how many uh transformer [16:05] blocks or other type of neural network layers. Uh and uh with this uh uh you can find that this you [16:12] can kind of flexibly control the complexity quality trade-off. uh so we can reduce the [16:18] uh latency actual latency by uh like 100x with a very minor performance degradation and also if you [16:27] uh cut the depth in the neural network you can also reduce the uh memory size. So and this is the [16:34] uh the actual example. So for example you have this uh uh same image generated by uh 128 steps [16:45] like or iterations you can you can say or four or one at the same time you can also reduce number [16:51] of dats from let's say 12 to 8 or four and you can see there is like a maybe large degradation [16:58] if you come like reduce number of time steps and blocks like significantly but you can kind [17:05] of select this kind of u good trade-off that is where you cannot even see much difference between [17:12] generated image using uh let's say uh 100 times more complexity and uh quantitatively we also can [17:22] plot this kind of scaling how the performance uh uh depends on the uh actual latency and [17:29] uh yeah I can show yeah I can I cannot show you here how the like vanilla diffusion model [17:35] uh uh how many how much latency it has because it's just uh too too too large. Like for example, [17:42] it's going to be like 100 times latency. It could be several minutes to generate one image. But with [17:47] this approaches, we can generate images with like uh one one or even less seconds with reasonable [17:53] uh complexity. uh and uh also in addition to that because uh in our method we uh are uh agnostic [18:02] to the solver we use we can also uh have a quite good uh uh uh performance metrics with reasonable [18:09] latency u okay so the conclusions so for next generation agentic models there is a debate should [18:19] it be like ultra aggressive fellow lamps or vms or some something else and what we see is that the [18:26] u the conventional type of LLMs they scale still scale uh for CL CLA type of aentic applications. [18:37] Of course every step uh uh incurs even more compute like uh like now companies spend like [18:44] hundreds of billions of dollars for this uh for training for such models. But at the same time [18:50] this kind of existing models they still struggle struggle with the continuous data like images or [18:56] sensory data and also they are less flexible in uh controlling the uh complexity quality tradeoff. So [19:05] uh usually uh yeah they are just quite limited they don't have the option of this [19:10] uh iterative den noising like in diffusion models. uh at the same time the hybrid models like VAS uh [19:17] that basically attach additional kind of diffusion based action expert uh can extend the kind of the [19:24] life of existing uh autogressive LLMs and they can be applicable to robotics and computer use [19:30] agents uh but in parallel uh uh people start to build like completely different architectures [19:37] like uh I mentioned discrete diffusion LLMs DLM and the world action models So [19:43] the DLMs are significantly faster. You can look at the uh independent um independent [19:50] u measurements of that. Uh but yeah, the only question is that we are not sure if they are [19:57] uh still capable to do the same type of reasoning as the LLMs. Uh and also the world action models [20:05] uh excel in this uh zero short generalization. Uh but they are like very uh like in infancy. [20:11] So we are not sure if we can optimize them as uh as good as VAS. So but for [20:18] uh also for multimodels probably want to use some kind of uh uh the best possible approaches but [20:24] this domain specific agents could be uh uh robust at the fraction of cost of the generalist models. [20:32] So yeah I'm just concluding the my current yeah I think I'll probably miss uh skip this one. So, [20:38] uh, yeah, that's actually all the time we have. Uh, thank you. Huh? That's all the [20:44] time we have. Yeah. Yeah. 11:50. Yeah. So, if you're interested in this type of models, yeah, [20:49] I'll probably post my uh slides on LinkedIn and you can take a look if you're interested. figure.