NVIDIA Releases Cosmos 3 Edge Open World Model for Robotics and Edge AI

NVIDIA Releases Cosmos 3 Edge Open World Model for Robotics and Edge AI NVIDIA has launched Cosmos 3 Edge, an open 4-billion-parameter world model designed to enable real-time visual reasoning and action generation on edge hardware. Alongside the edge model, NVIDIA published policy checkpoints and distilled variants for accelerated generation. Cosmos 3 Edge is a 4B-parameter model providing real-time reasoning and robot control at 15 Hz on NVIDIA Jetson Thor. The architecture couples an autoregressive transformer tower for language and vision with a diffusion tower for vision, audio, and action generation. Different robotic embodiments are mapped into a unified action representation of geometric vectors covering translation, rotation, and manipulation states. NVIDIA also launched step-distilled 4-step Cosmos3-Super models that speed up inference by up to 25x without classifier-free guidance. Edge Performance and Two-Tower Architecture Cosmos 3 Edge is built to run on memory-constrained systems including NVIDIA RTX PRO, DGX, GeForce RTX, and Jetson platforms, including the Jetson T2000 and T3000 modules. Operating as a post-trained world action model at 640×360 resolution, it generates 32 actions per inference at 15 Hz on Jetson Thor. It combines an autoregressive transformer tower for vision and text tokens with a diffusion tower for vision, audio, and action tokens, linking them through shared multimodal attention layers. Action Representation and Policy Modes The model translates distinct robotic embodiments into geometric vectors defining translation, rotation, and manipulation states. Operating in policy mode, it simultaneously generates actions and forecasts their visual consequences, enabling bidirectional inference between causes and effects. Alongside the core model, NVIDIA released Cosmos 3 Edge Policy (DROID), a manipulation checkpoint post-trained on the DROID dataset for pick-and-place tasks. Distilled Variants and Benchmarks Among models of comparable 4B size, Cosmos 3 Edge ranks first on VANTAGE-Bench for vision analytics. NVIDIA also released 4-step distilled models, Cosmos3-Super-Image2Video-4Step and Cosmos3-Super-Text2Image-4Step, which cut denoising steps from 50 to 4 and bypass classifier-free guidance to deliver up to 25x faster inference. Training scripts, distillation recipes, and checkpoints have been published via Hugging Face and GitHub.

英伟达发布开源物理AI世界模型 Cosmos 3 Edge 英伟达在Hugging Face上发布了40亿参数的开源世界模型Cosmos 3 Edge，旨在为内存受限的边缘设备提供实时推理、环境理解与机器人动作生成能力，同时还推出了策略模型及4步蒸馏加速版本。 Cosmos 3 Edge拥有40亿参数，在同尺寸模型中位列VANTAGE-Bench视觉分析第一，在NVIDIA Jetson Thor上以640×360分辨率实现15 Hz实时控制（单次推理生成32个动作）。 架构结合了用于理解与推理的自回归塔和用于预测与生成的扩散塔，两塔共享多模态注意力层。 采用紧凑几何向量将平移、旋转与操作状态映射为通用动作表征，支持预测动作后果或反推执行动作。 同步发布基于DROID数据集微调的机械臂操作策略模型，以及仅需4步去噪、推理速度提升达25倍的Cosmos3-Super蒸馏版本。 边缘部署与双塔架构设计 Cosmos 3 Edge适配RTX PRO、DGX、GeForce RTX以及Jetson T2000/T3000等边缘计算平台。模型融合了自回归塔与扩散塔，前者负责视觉与文本标记的推理，后者处理视觉、音频及动作标记的神经模拟生成。两塔拥有独立的归一化层和多层感知机，但共享多模态注意力层，使得模型在生成输出前能全面对齐语言、音视频和动作信息。 通用动作表征与策略模式 该模型通过几何向量整合不同硬件实体的平移、旋转和抓夹状态，将物理运动与像素变化建立关联。在策略模式下，模型输入当前状态即可同时输出动作及其预期的视觉后果，具备双向推理能力。此外，英伟达发布了适用于拾取和放置任务的Cosmos 3 Edge Policy (DROID)模型与微调脚本，支持开发者在H100或DGX Station上进行高效后训练。 步数蒸馏版本与后续规划 英伟达推出了4步去噪的Cosmos3-Super图像转视频和文本转图像蒸馏版本，去除了无分类器引导，在保持质量的同时实现了最高25倍的推理速度提升。未来英伟达计划继续推进交互式世界生成、自动驾驶仿真及机器人策略，并借助vLLM等开源框架进一步优化推理效率。

The real world is vast and to operate in it physical AI systems need to understand how a scene is changing, anticipate what could happen next, and determine how an action will affect the world. Machines operate at the edge in factories, warehouses, hospitals and beyond. To understand and act in these environments, they need models that deliver data center–level performance on memory-constrained systems. Today we are releasing NVIDIA Cosmos 3 Edge on Hugging Face’s Cosmos 3 repository. It’s a 4-billion-parameter open world model that helps robots and vision AI agents understand their surroundings, reason in real time and generate robot actions on edge devices. Download the model: https://huggingface.co/nvidia/Cosmos3-Edge The model delivers memory-efficient, high-throughput inference across NVIDIA edge computers including NVIDIA RTX PRO GPUs, NVIDIA DGX, NVIDIA GeForce RTX™ GPUs, NVIDIA Jetson including the newly announced Jetson T2000 and T3000 modules. Designed as a compact open model that can serve as a small vision language model (VLM) with best-in-class throughput and accuracy with real-time inference. As a post-trained world action model (WAM), Cosmos 3 Edge operates at robot-control resolution (640×360 observations), delivering real-time reasoning and generating 32 actions per inference on NVIDIA Jetson Thor - while achieving real-time control at 15 Hz. Among similar size (4B parameters) models, Cosmos 3 Edge ranks #1 on VANTAGE-Bench for vision analytics and state-of-the-art for robot policy learning, setting the state of the art for smart infrastructure and robotics. What Is World Modeling? A world model learns how an environment changes over time. It represents objects, motion, spatial relationships, and the effects of actions. Consider a robot reaching for an object. Recognizing the object is only the first step. The robot must also understand where the object is, how its gripper is moving, what may happen when contact occurs, and which action is most likely to complete the task successfully. A world model helps the robot reason about these relationships. It can predict the visual result of an action, infer the action that caused a change, or generate an action to produce a desired outcome. Cosmos 3 Edge brings these capabilities together in one model, on-device. Its shared representation enables physical AI systems to understand the current world state, simulate possible futures, and connect those futures to actions. Two Transformer Towers, One Shared Representation Cosmos 3 combines two transformer towers: Autoregressive tower: Processes vision and text tokens for understanding and reasoning. Diffusion tower: Processes vision, audio, and action tokens for prediction, generation, and neural simulation. The two towers maintain separate normalization layers and multilayer perceptrons. They share multimodal attention layers, which align information across language, video, audio, and action. This design allows the model to reason about a scene before generating an output. Depending on the task, Cosmos 3 can produce reasoning tokens from the autoregressive tower or denoised video and action tokens from the diffusion tower. The attention pattern is also adapted to each type of information. Language uses causal attention, where each token attends to the tokens that came before it. Diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation. Together, these components give the model a common representation of what is happening, what may happen next, and how an action could influence the outcome. A Common Representation for Action Physical systems describe actions differently. A vehicle may represent an action through ego pose and movement. A camera uses camera motion. A robot arm uses the pose of its end effector, while a hand or gripper also needs to represent grasp state. Cosmos 3 maps these different embodiments into a common action representation. Actions are encoded as compact geometric vectors that capture: Translation Rotation Manipulation state This creates a direct connection between control and the visual structure of the world. The model can associate changes in pixels with physical motion, spatial relationships, and control inputs. As a result, generated video becomes more than a visual prediction. It can represent how the world is expected to change in response to an action. This gives developers training data grounded in motion, control, and cause and effect. Policy Mode As a policy, Cosmos 3 predicts an action together with its expected visual consequence. The model can generate what the system should do and simulate what is likely to happen next. This connects world modeling directly to robot policy training and evaluation. Current state in, action and visual consequence out. These modes enable one model to learn cause, effect, and policy. Action can flow through the model in both directions, allowing Cosmos 3 Edge to predict the effects of an action or infer an action from its effects. Prompt: Pick up the banana and put it in the plate. We are also releasing Cosmos 3 Edge Policy (DROID): A robot manipulation policy post-trained on the DROID dataset for pick-and-place tasks, with accompanying post-training scripts. Developers can use a small cluster of H100 or NVIDIA DGX Station to efficiently fine-tune Cosmos 3 Edge for their target workloads before deploying to NVIDIA edge and accelerated computing platforms. Post-trained samples for improved performance Alongside the base models, we're releasing reference post-trained checkpoints and training recipes to help developers adapt and optimize Cosmos 3 for their own applications. Cosmos 3 is an open world foundation model platform. As models are post-trained with high-quality, domain-specific data, their accuracy continue to improve for specialized applications. Cosmos serves as the starting point for building domain-adapted world models using open frameworks. Post-training enables developers to improve model performance while preserving output quality. Step-Distilled Cosmos3-Super: 25x Faster, Same Quality We're releasing step-distilled versions of Cosmos3-Super: Cosmos3-Super-Image2Video-4Step and Cosmos3-Super-Text2Image-4Step . They cut sampling from 50 denoising steps to just 4, and remove the need for classifier-free guidance — which otherwise doubles the work per step — for up to 25× faster inference than the base model, with little to no loss in generation quality. On the Artificial Analysis Image-to-Video leaderboard (as of July 23, 2026), the distilled I2V model ranks #1 among open-weight models, ahead of the full Cosmos3 Super teacher it was distilled from; the distilled Text-to-Image model ranks #2 — meaning you get top-tier open-weight video quality at a fraction of the compute. We're also open-sourcing the full distillation recipe and training scripts at the Cosmos github repository , so developers can distill their own Cosmos3 checkpoints for faster, application-specific image and video generation. Try New Cosmos 3 Models Download the Edge models: Cosmos3-Edge | Cosmos3-Edge-Policy-DROID Download the distilled models: Cosmos3-Super-Image2Video-4Step | Cosmos3-Super-Text2Image-4Step Checkout Cosmos 3: https://huggingface.co/collections/nvidia/cosmos3 What's ahead We continue to advance Cosmos 3 for Physical AI, with upcoming improvements in interactive world generation, driving scenario simulation, and robotics policies. We are also investing in faster inference and more efficient post-training across a broader range of hardware, reducing the time and compute required to build downstream models. This includes optimizing Cosmos 3 checkpoints with open inference and optimization frameworks such as vLLM, with more optimizations and developer tooling coming soon.