NVIDIA Introduces Cosmos-H-Dreams for Real-Time Generative Simulation in Surgical Robotics

NVIDIA Introduces Cosmos-H-Dreams for Real-Time Generative Simulation in Surgical Robotics NVIDIA has unveiled Cosmos-H-Dreams, an action-conditioned, real-time generative simulator that distills surgical world foundation models into an interactive environment running on a single GPU. Designed to address the difficulties of modeling complex surgical dynamics and costly hardware evaluation, the system supports closed-loop interaction for human operators and autonomous policies. Distills the Cosmos-H-Surgical-Simulator world model into a causal student that runs at approximately 160 frames per second on a single NVIDIA RTX PRO 6000 GPU. Uses a multi-stage training pipeline including teacher fine-tuning with failures, causal warmup, and self-forcing distillation to avoid error accumulation. Serves autoregressive video generation via FlashDreams using streaming KV caches, CUDA Graph capture, and model compilation. Supports multiple control interfaces including WebRTC browser controls, Meta Quest WebXR tracking, and CMR Surgical's Versius surgeon controller. Interactive Generative Simulation Evaluating and training vision-language-action policies in surgery is constrained by expensive hardware, slow physical iteration, and the challenge of simulating deformable tissue, smoke, sutures, and occlusions. While NVIDIA's earlier Cosmos-H-Surgical-Simulator operated offline to predict future visual rollouts from action trajectories, Cosmos-H-Dreams transitions the technology into real time. Fine-tuned for da Vinci Research Kit tabletop suturing and also integrated with CMR Surgical's Versius controller, the model processes an initial frame and a continuous stream of kinematics to autoregressively synthesize sequential visual frames in an interactive closed loop. Teacher-to-Student Distillation Pipeline To achieve low-latency generation without sacrificing physical plausibility, Cosmos-H-Dreams employs a multi-step distillation workflow. The bidirectional teacher model, built with a 44-dimensional action space, is fine-tuned on tabletop datasets that intentionally incorporate failures and out-of-distribution events like dropped needles and missed throws across horizons gradually lengthened up to 72 frames. A causal student model is then warmed up on precomputed denoising paths with a streaming key/value cache, followed by self-forcing distillation where the student generates rollouts conditioned on its own prior outputs under frozen teacher distribution-matching guidance. This permits few-step diffusion using as few as two denoising steps per latent frame. Inference Acceleration and Human-Machine Interfaces The model is deployed via FlashDreams, an accelerated streaming runtime that incorporates CUDA Graph capturing, compilation, and streaming key/value caching. These optimizations elevate rollout generation from standard ten-frame-per-second rates to roughly 160 frames per second on an NVIDIA RTX PRO 6000 GPU. The system features multiple interactive frontends, allowing operators to send commands via browser-based WebRTC or Meta Quest WebXR headsets, or connecting directly to learned robotic policies for continuous evaluation. Applications and Research Scope Beyond visual realism, NVIDIA highlights the necessity of new closed-loop benchmarking metrics to evaluate tool-tip reach, pose fidelity, idle stability, counterfactual diversity, and transferability to physical hardware. The real-time simulator enables on-demand generation of rare failure modes, policy evaluation without robotic hardware, and exploratory applications in latency-compensated telesurgery and surgical rehearsal. NVIDIA explicitly notes that Cosmos-H-Dreams is an exploratory research and development platform rather than a diagnostic tool, intraoperative imaging alternative, or live surgical robot controller.

NVIDIA推出Cosmos-H-Dreams：面向手术机器人的实时动作条件生成式模拟器 NVIDIA推出手术机器人实时动作条件生成模拟器Cosmos-H-Dreams，通过自强蒸馏技术和FlashDreams流式推理加速库，在单张RTX PRO 6000 GPU上实现约160 FPS的低延迟闭环交互仿真。 基于Cosmos-H-Surgical-Simulator与Open-H-Embodiment数据集，针对dVRK台式缝合进行专业化因果自回归模型蒸馏。 采用因果预热与自强蒸馏（self-forcing distillation），仅需每潜在帧2次去噪步数，有效抑制长序列误差累积。 搭配FlashDreams推理引擎（流式KV缓存、CUDA Graph及模型编译），推理帧率从约10 FPS提升至约160 FPS。 支持WebRTC浏览器端、Meta Quest（WebXR）控制及学习策略的闭环交互，并与CMR Surgical及剑桥咨询合作集成了Versius手术台控制端。 官方强调该模型属于研发平台，非诊断系统、术中影像替代品或实体手术机器人控制器。 从离线世界模型走向实时交互仿真 手术机器人仿真长期面临软组织形变、缝线操作及反光遮挡等建模难题。早期Cosmos-H-Surgical-Simulator虽能预测未来手术动作视频，但主要用于离线策略评估和合成数据生成。Cosmos-H-Dreams将其转化为因果自回归模型，输入初始RGB图像与实时机器人运动学数据流即可连续生成对应画面。此外，团队已与CMR Surgical和剑桥咨询合作，将其接入Versius手术操控台验证实时性能。 教师-学生蒸馏与自强训练流水线 系统将双臂dVRK动作映射至44维统一动作空间，并在包含掉针、打结失败等分布外失败案例的数据集上微调双向教师模型，时间跨度从12帧逐步扩展到72帧。随后，学生模型先通过模仿缓存轨迹完成因果预热，再利用自强蒸馏基于自身生成的上下文滚动生成，由冻结的教师模型监督分布匹配。这使学生模型在保持少步扩散（低至2步去噪）的同时，避免了部署时的自回归复合误差。 FlashDreams加速引擎与多接口支持 为了满足交互要求，NVIDIA通过FlashDreams加速推理库整合了流式KV缓存、CUDA Graph捕获和模型编译，将生成帧率从原先的约10 FPS提升至单张NVIDIA RTX PRO 6000上的约160 FPS。在接口方面，Cosmos-H-Dreams支持通过WebRTC进行浏览器键盘控制、基于WebXR的Meta Quest手柄遥控，以及与算法策略直接形成闭环交互。 闭环物理AI评测与应用前景 该模拟器为手术策略开发提供了罕见故障生成与可扩展训练环境。未来评估将侧重于工具末端位姿精度、夹爪循环保真度、空闲稳定性及长时漂移等闭环指标，并有望拓展至延迟感知远程手术、手术预演和术中决策支持。NVIDIA同时指出，该系统是研发探索工具，不能替代术中影像或直接控制实体机器人。

Surgical robotics is moving quickly from teleoperation toward increasingly capable vision-language-action policies. But evaluating and training these systems remains difficult. Physical robotic platforms are expensive to operate, experiments are slow to reproduce, and failures can damage instruments or biological material. Conventional simulators provide a safer alternative, but surgical scenes are exceptionally difficult to model: deformable tissue, fine instrument interactions, specular surfaces, sutures, needles, smoke, and occlusions all matter. World foundation models offer a different path. Instead of manually authoring every object and physical interaction, they learn visual dynamics directly from synchronized video and robot kinematics. NVIDIA's Cosmos-H-Surgical-Simulator demonstrated this approach by generating future surgical video from an initial scene and a sequence of robot actions. It enabled faster-than-physical evaluation and synthetic data generation across the Open-H-Embodiment ecosystem. Today, we are introducing the next step: Cosmos-H-Dreams , a real-time, action-conditioned generative simulator for surgical robotics. Cosmos-H-Dreams distills the capabilities of Cosmos-H-Surgical-Simulator into a causal, few-step student model and serves it through FlashDreams , NVIDIA's accelerated streaming-inference library. Running on a single NVIDIA RTX PRO 6000 GPU, the result is an interactive environment that a person or a learned policy can control in a closed loop. 1. From Surgical World Model to Interactive Simulator Cosmos-H-Surgical-Simulator is an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2.5-2B and post-trained on the Open-H-Embodiment dataset. Given a surgical context frame and a future robot trajectory, it generates video showing the likely visual consequences of those actions. This makes it useful for offline policy evaluation and synthetic data generation. A recorded or policy-generated trajectory can be sent to the model, the corresponding rollout can be generated, and the result can be inspected or scored without repeatedly executing the motion on a physical robot. Cosmos-H-Dreams moves the model into the real-time regime. Starting from the multi-embodiment surgical priors learned by Cosmos-H-Surgical-Simulator, we specialize the model for da Vinci Research Kit (dVRK) tabletop suturing and distill it into a causal student that generates the scene autoregressively. The released model receives an initial RGB frame and a live stream of robot kinematics, then produces the next chunk of frames before continuing with the following action block. We have also demonstrated the versatility of Cosmos-H-Dreams by collaborating with CMR Surgical and Cambridge Consultants to integrate it with the Versius surgeon controller, enabling real-time operation on the Versius platform. 2. Distilling Cosmos-H-Surgical-Simulator for Real Time The key challenge is preserving useful surgical dynamics while reducing the cost of generation. Cosmos-H-Dreams uses a teacher-to-student training pipeline designed for long, autoregressive rollouts. 2.1. A Surgical Teacher The bidirectional teacher begins from the Cosmos-H-Surgical-Simulator Open-H checkpoint, which uses a unified 44-dimensional action representation. For the released dVRK tabletop model, the dual-arm dVRK action content, consisting of relative end-effector translation, rotation, and gripper state, is mapped into this common representation. The teacher is then fine-tuned on the JHU dVRK tabletop mixture, including successful demonstrations as well as failure and out-of-distribution episodes such as needle drops, missed throws, and unsuccessful knot ties. These failures are important: a simulator intended to evaluate policies must reproduce the consequences of poor actions, not only ideal demonstrations. To improve stability during long rollouts, the training process progressively increases the teacher's temporal horizon. We start training on a 12-frame horizon and progressively increase this number until 72 frames. At each horizon bump, we initialize the warmed-up model with pretrained weights. 2.2. Causal Warmup The teacher's denoising trajectories are first precomputed and cached. A causal student is initialized from the teacher and trained to imitate these cached trajectories. This warmup stage teaches the student to operate with causal attention and a streaming key/value cache before it begins learning from its own generated history. 2.3. Self-Forcing Distillation Autoregressive models face a familiar problem: during training they may see clean, ground-truth context, while during deployment they must condition on their own imperfect outputs. Small errors can therefore compound over time. Cosmos-H-Dreams addresses this mismatch with self-forcing distillation . During training, the student rolls forward using its own generated context. Distribution-matching supervision from the frozen teacher then guides those self-generated rollouts toward realistic surgical video. This prepares the student for the same conditions it will encounter during interactive inference. The resulting model supports few-step diffusion, with as few as two denoising steps per latent frame, rather than the many-step process used by the full teacher. It combines the teacher's surgical priors with the causal structure needed for streaming. 3. FlashDreams: The Real-Time Inference Engine Model distillation is only part of the real-time story. Cosmos-H-Dreams is served through FlashDreams , an accelerated inference library for autoregressive world and video models. FlashDreams turns the distilled student into a low-latency streaming system through several complementary optimizations, such as streaming KV cache, CUDA Graph capturing, or model compilation. Together, these techniques bring the distilled surgical world model fine-tune from the roughly ten-frames-per-second regime of standard Cosmos-H-Surgical-Simulator inference to interactive operation (~160 frames per second) on a single NVIDIA RTX PRO 6000 . Cosmos-H-Dreams also provides the human-machine interfaces that turn generation into interaction. A browser client can send keyboard commands and receive generated frames over WebRTC. A Meta Quest client can map tracked controller motion into robot actions and display the synthesized scene through WebXR. The same model can also be connected to a learned surgical policy, with generated observations and predicted actions exchanged inside a closed loop. 4. Adapting to Your Own Data While Cosmos-H-Dreams includes a pre-trained checkpoint for tabletop suturing, the system is designed to be extensible to your specific embodiment. To train a real-time student model for your own dataset, we provide a complete recipe for teacher fine-tuning and self-forcing distillation in our step-by-step guide . 5. What Is Next: Toward Closed-Loop Surgical Physical AI Cosmos-H-Dreams opens a new frontier for surgical simulation: environments learned from real robot data that are responsive enough to be inhabited. The immediate next step is to evaluate more than visual quality. A useful surgical simulator must respond correctly to actions, preserve instrument and scene structure over long rollouts, and support conclusions that transfer to the physical robot. This motivates a new family of closed-loop benchmarks: tool-tip reach and pose accuracy, gripper-cycle fidelity, idle stability, counterfactual action diversity, long-horizon drift, and agreement between simulated and real policy outcomes. Real-time world models can also become active partners in surgical policy development. They can generate rare failures on demand, provide scalable environments for reinforcement or imitation learning, and enable rapid evaluation of new policies without tying every experiment to scarce robotic hardware. Further ahead, real-time simulation enables a series of downstream applications such as latency-aware telesurgery, where the world model helps maintain a more stable display; or interactive surgical rehearsal, procedure planning, and intraoperative decision support. Cosmos-H-Dreams is a research and development platform, not a diagnostic system, a replacement for intraoperative imaging, or a controller for a physical surgical robot. Yet it provides a foundation for exploring these possibilities safely. As model fidelity, temporal stability, and hardware efficiency continue to improve, real-time generative simulation can help connect surgeon education, synthetic data generation, policy training, and policy evaluation within one shared Physical AI environment. 6. Get Started Today Explore the models, data, and runtime behind Cosmos-H-Dreams: Cosmos-H-Dreams code and examples: GitHub repository Cosmos-H-Dreams model: Hugging Face checkpoint Cosmos-H-Surgical-Simulator: Hugging Face model and GitHub repository Adapt to your own data: Step-by-step recipe for teacher fine-tuning and self-forcing distillation Open-H-Embodiment: Hugging Face dataset FlashDreams: GitHub repository NVIDIA Cosmos-Predict2.5: GitHub repository Cosmos-Surg-dVRK: World Foundation Model-based Automated Online Evaluation of Surgical Robot Policy Learning ( arXiv paper ) Cosmos-H-Dreams brings action-conditioned surgical world modeling into the real-time loop, creating a new environment for people and policies to practice, explore, generate data, and evaluate what happens next.