Beyond VLAs: NVIDIA Introduces World Action Models and Cosmos 3 for Robot Manipulation

Beyond VLAs: NVIDIA Introduces World Action Models and Cosmos 3 for Robot Manipulation NVIDIA is exploring a shift in robot policy design from vision-language-action (VLA) models to world action models (WAMs) powered by video world models. By incorporating physics priors and world dynamics, WAMs aim to overcome the generalization limits of VLAs, utilizing the open-source Cosmos 3 omni-model foundation. Traditional VLAs rely on vision-language models that lack physical dynamics modeling, limiting their ability to anticipate scene changes and generalize physically. WAMs pretrain on world dynamics and jointly predict video and action, enabling learning from varied interaction data, better open-world generalization, and rapid embodiment adaptation. NVIDIA Cosmos 3 is an omni-model foundation using a Mixture-of-Transformers architecture with an autoregressive transformer for discrete reasoning and a diffusion transformer for continuous modalities. The 16B Cosmos3-Nano-Policy-DROID and 4B Cosmos3-Edge-Policy-DROID models can jointly output action trajectories and predicted outcome videos, boosting RoboLab task success from 28.1% to 36.8% over a base checkpoint. Cosmos 3 supports flexible deployment tiers, including workstation streaming with the 16B Nano model and 15 Hz on-device real-time control with the 4B Edge model on NVIDIA Jetson Thor. Limitations of the VLA Paradigm Standard language-conditioned robot policies augment pretrained vision-language models with an action module to create vision-language-action models. While effective for semantic understanding, VLMs are trained to describe visual scenes rather than predict how physical scenes evolve over time. Consequently, VLAs often fail to model what happens when grippers close, towels fold, or objects drop, leading to poor physical generalization in unseen environments and a reliance on near-identical demonstration trajectories. Advantages of World Action Models World action models replace the language backbone with a video world model, instilling a physics prior before robot specialization. Research detailed in 'World Action Models are Zero-shot Policies' shows that jointly predicting video and action allows policies to learn from diverse interaction data—including pushes, grasps, and drops—rather than requiring identical task demonstrations. Because fundamental dynamics apply across objects, WAMs demonstrate superior open-world generalization and require far fewer task-specific demonstrations when transferring to new robotic embodiments. The Cosmos 3 Omni-Model Architecture Cosmos 3 is built on a Mixture-of-Transformers architecture that processes multimodal input through an autoregressive transformer for discrete tokens like text, which then guides a diffusion transformer that generates continuous modalities such as video, audio, and actions via iterative denoising. Pretrained on approximately 767 million images, 348 million real-world dynamics videos, and 8 million action samples spanning robotics, driving, and egocentric motion, Cosmos 3 is released in 4B Edge, 16B Nano, and 64B Super sizes. Cosmos 3 Policy DROID and Empirical Gains Cosmos 3 can be specialized via post-training into policies such as the 16B Cosmos3-Nano-Policy-DROID and the 4B Cosmos3-Edge-Policy-DROID for Franka Panda arms with Robotiq grippers. These policies maintain the full omni-model architecture and can simultaneously generate robot actions alongside predicted camera observation videos. In technical evaluations, starting from an omni checkpoint pretrained on multi-domain action data increased RoboLab task success from 28.1% to 36.8% compared to training from a base checkpoint with identical recipes and compute. Deployment Options and Open Release Cosmos 3 accommodates varied hardware setups. The 16B Nano policy can be hosted beside the robot on a workstation with a single NVIDIA RTX PRO 6000, streaming observations and actions over the network. Meanwhile, the 4B Edge model runs directly on embedded edge hardware like NVIDIA Jetson Thor, generating 32 actions per inference at 640×360 resolution to achieve 15 Hz real-time control. NVIDIA has released the base model, dataset tools, weights, and recipes under licenses permitting commercial use.

超越VLA：世界动作模型重塑机器人操作 传统视觉-语言-动作（VLA）模型由于缺乏物理动态建模能力，在面对新场景时物理泛化表现欠佳。NVIDIA提出采用视频世界模型作为骨干的世界动作模型（WAM），并基于开源全模态模型Cosmos 3构建了高效的机器人策略模型。 VLA侧重语义理解而缺乏对世界演化规律的建模，WAM则通过视频世界模型融入物理先验，具备更强的数据利用率、开放世界泛化能力及跨构型适应性。 Cosmos 3采用混合Transformer（MoT）架构，包含自回归推理与扩散去噪生成，预训练数据覆盖约7.67亿张图像、3.48亿段视频及800万条动作样本。 在DROID平台测试中，基于全模态检查点后训练的策略使RoboLab成功率从28.1%提升至36.8%，且支持在输出动作的同时生成未来视觉演化视频。 模型推出4B Edge、16B Nano和64B Super三种规格，4B版本可在NVIDIA Jetson Thor上实现640×360分辨率下15 Hz的实时闭环控制。 从VLA向WAM的范式转变 常规VLA通过预训练VLM与动作模块结合来实现语义指令控制，但其骨干网络主要优化文本输出，难以预测抓取、折叠或释放物体时的物理动态变化。WAM将骨干替换为视频世界模型，使联合预测视频与动作成为可能。具备物理先验的WAM能够从更多样的交互数据中提取训练信号，在非训练分布场景下展现更优的物理泛化能力，并大幅降低新机械臂或夹爪所需的示范数据量。 Cosmos 3基础模型与DROID策略实现 Cosmos 3全模态模型结合了自回归Transformer（用于离散文本推理）和扩散Transformer（用于去噪合成图像、视频、音频及动作）。基于该底座开发的Cosmos3-Nano-Policy-DROID（16B）和Cosmos3-Edge-Policy-DROID（4B）保留了完整的全模态能力，在执行动作的同时可同步生成预测的多视角视觉结果。对比实验显示，在同等数据和算力下，由跨领域动作数据预训练的全模态检查点后训练后，RoboLab成功率相较基模型从28.1%显著提升至36.8%。 端侧与工作站部署架构 针对WAM体积较大的特点，Cosmos 3提供了分层部署路径。16B的Nano版本采用工作站外置部署模式，例如在单张NVIDIA RTX PRO 6000上运行，通过网络流式传输观测数据并返回动作分块；4B的Edge版本则专为嵌入式硬件设计，支持在NVIDIA Jetson Thor等边缘计算平台以15 Hz的频率实时运行，单次推理输出32个动作指令，适配Franka、双臂、UR等多种机器人形态。

A central challenge in robotics is building policies that generalize beyond the demonstrations they’re trained on. A policy that succeeds in a training scene often fails when object shapes, positions, or lighting change. Generalizing to these new conditions requires the policy to understand the tasks underlying physics, not just mimic the demonstrations. This ability comes from the backbone it’s built on. The standard way to build a language-conditioned robot policy is to add an action module to a pretrained [vision-language model (VLM)](https://www.nvidia.com/en-us/glossary/vision-language-models/), producing a vision-language-action (VLA) model. This approach has carried generalist manipulation a long way. But a VLM backbone learns to *describe* the world, not *predict how it evolves*. That missing dynamics model is exactly what a robot needs when a task depends on anticipating how a scene will change. A growing line of research replaces the language backbone with a video [world model](https://www.nvidia.com/en-us/glossary/world-models/), producing [world action model (WAM)](https://www.nvidia.com/en-us/glossary/world-action-model/). NVIDIA researcher Jim Fan explored this shift in his *Robotics’ End Game* talk—an idea later summarized as [“VLAs are dead, long live World Action Models.”](https://www.eventual.ai/blog/vlas-are-dead-long-live-world-action-models) This post explores how post-training can turn WAMs into specialized robot policy, how WAMs compare to VLAs, and why the open [NVIDIA Cosmos 3](https://www.nvidia.com/en-us/ai/cosmos/) world model provides a strong foundation for building WAMs. ## How post-trained policies are built today In the VLA paradigm, a pretrained VLM provides semantic understanding of scenes and language instructions, while post-training learns to map that understanding to robot actions. Modern generalist robot policies increasingly build on this approach. However, a VLM is optimized to produce text about images, not to model how a scene will evolve. It does not learn what happens to a mug when the gripper closes, how a towel folds, where an object lands when released. VLAs generalize well semantically but are less effective at physical generalization to unseen behaviors and environments, as their backbones model language and perception rather than world dynamics. ## What a WAM changes A WAM overcomes that dynamics-modeling limit by building the policy on a video world model. Because the backbone models how the world evolves, post-training does not have to teach dynamics from scratch, it specializes a model that already has a physics prior. The NVIDIA research paper [*World Action Models are Zero-shot Policies*](https://arxiv.org/abs/2602.15922) shows that jointly predicting video and action gives a policy properties a VLA cannot easily acquire: - **It learns from diverse data.** A VLA learns to map instructions to trajectories, and often requires near-identical demonstrations of the same task. A WAM learns physics: how objects move when pushed, grasped, or dropped. Any interaction data teaches it something. A varied dataset that would be wasted on a VLA becomes a training signal, and data collection gets cheaper. - **It generalizes in the open world.** Physics is more general than semantics. The way an object falls or slides doesn’t change with the object, so a model that has learned dynamics carries that knowledge into scenes and motions it was never trained on. - **It adapts to new robots with few demonstrations.** A model that already understands physical interaction needs far less task-specific data to specialize to a new arm or gripper. For a team building a policy, these are the practical wins: less data to reach a given capability, better behavior outside the training distribution, and a shorter path to a new embodiment. They are properties of the *pretraining backbone*, so they show up in every policy post-trained from it. ## Cosmos 3 is a strong WAM foundation [Cosmos 3](https://huggingface.co/collections/nvidia/cosmos3) is an [omni-model](https://www.nvidia.com/en-us/glossary/omni-model/) world foundation model built on a [Mixture-of-Transformers (MoT)](https://www.nvidia.com/en-us/glossary/mixture-of-transformers/) architecture. Multimodal input flows through an autoregressive transformer for reasoning producing discrete tokens such as text. This guides a diffusion transformer for continuous modalities, including image, video, audio, and action. Text is generated by next-token decoding; everything else, including actions, is synthesized through iterative denoising. A single model spans these modalities while keeping the generation mechanism best suited to each. Cosmos 3 comes in three sizes: 4B NVIDIA Cosmos Edge, 16B NVIDIA Cosmos Nano, and 64B NVIDIA Cosmos 3 Super. What makes Cosmos 3 a strong foundation for post-training is the breadth of its physical-world data. The dataset includes roughly 767M images,348M videos of real-world dynamics, 8M action samples spanning robot manipulation, autonomous driving, camera motion, and egocentric motion. ## From world model to robot policy: Cosmos 3 Policy DROID models Cosmos 3 is the starting point for specialization. [Cosmos3-Nano-Policy-DROID](https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID) is a 16B-parameter policy post-trained from Cosmos 3 Nano for the [DROID](https://droid-dataset.github.io/) platform, which is a Franka Panda arm with a Robotiq gripper.A 4B version, [Cosmos3-Edge-Policy-DROID](https://huggingface.co/nvidia/Cosmos3-Edge-Policy-DROID), is post-trained the same way, which can be used for on-device deployment. Given a language instruction and multi-camera observations, it generates robot action trajectories. Three properties follow from its omni foundation: - **It imagines while it acts.** When the model outputs actions, it can also output a video: what the robot’s cameras will see if those actions are executed. The action and the predicted outcome come from the same model, at the same time. - **It keeps the full omni architecture.** Post-training removes nothing. The policy checkpoint can still reason and generate video, not just output joint positions. - **The prior is measurable.** The [Cosmos 3 technical report](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) compares two DROID policies trained with the same recipe,  data, and compute. One started from the base checkpoint, while the other started from an omni checkpoint trained on multi-domain action data. The omni checkpoint raised RoboLab success from 28.1% to 36.8%. This is clear evidence that the improvement comes from the architecture, not just from scale. ## Deployment considerations A WAM carries the full generative world model, not just an action head. It’s larger than compact VLAs, but Cosmos 3 map to different deployment tiers rather than forcing one trade-off: - **Workstation serving (Nano, 16B ).** Cosmos3-Nano-Policy runs beside the robot rather than on board. Real-world DROID deployment serves it on a single NVIDIA RTX PRO 6000, with the robot streaming observations over the network and receiving action chunks back. - **On-device (Edge, 4B).** Cosmos 3 Edge runs the same policy workload directly on embedded hardware. It operates at robot-control resolution (640×360 observations) and generates 32 actions per inference on NVIDIA Jetson Thor while achieving real-time control at 15 Hz. It’s supported across NVIDIA edge computers including RTX PRO GPUs, DGX, GeForce RTX GPUs, and Jetson, including the new Jetson T2000 and T3000 modules. ## Why build robot policies with Cosmos 3? WAMs represent a shift from learning to act to learning how the world evolves. Cosmos 3 makes this approach practical: 1. **Open foundation, SOTA starting point.** The base model, datasets, post-training recipe, trained weights, evaluation tools, and serving stack are all released under a license that permits commercial use. 2. **Faster adaptation.** Strong physical priors cut the task-specific data a new policy needs. Convert your data to the LeRobotDataset format the robot-learning ecosystem already records into, run the published recipe. 3. **One foundation, many robots.** Each new embodiment, such as Franka, dual-arm setups, UR, WidowX, still requires its own post-training run. But all of them start from the same pretrained foundation rather than pretraining a world model from scratch, reducing the demonstrations needed for each. ## Get started The fastest way to evaluate whether a WAM beats your current VLA is to post-train one from Cosmos 3 on your own data and compare. - Explore the open [Cosmos 3 model collection](https://huggingface.co/collections/nvidia/cosmos3) and datasets on [Hugging Face](https://huggingface.co/nvidia) and [GitHub](https://github.com/nvidia-cosmos). - Read the [Cosmos 3 technical report](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) for full architecture details and evaluations. - Tune in to the [Cosmos Labs livestream](https://www.addevent.com/calendar/ss55fmjpm04t) on August 13. - Become part of the community and join the [Cosmos Discord channel](https://discord.gg/u23rXTHSC9).