Training Cross-Embodiment Robot Navigation Policies with AI Agents and COMPASS

Training Cross-Embodiment Robot Navigation Policies with AI Agents and COMPASS NVIDIA outlines an agent-driven workflow utilizing COMPASS (Cross-Embodiment Mobility Policy via Residual RL and Skill Synthesis) to train navigation policies across different robots and environments. The pipeline uses coding agents with repository skills to handle setup, smoke testing, training, and checkpoint diagnostics while enforcing human approval gates at key development phases. COMPASS adapts navigation behavior from the pretrained NVIDIA X-Mobility base policy by training residual reinforcement learning specialists for specific robot-scene pairs. Coding agents (such as Codex or Claude Code) interact via repository skills ($compass, $compass-doctor, $compass-newembodiment) to manage validation, test runs, and training logs. Workflows support three scene sources: a built-in multi-rack warehouse, synthetic indoor environments from SAGE-10K, and reconstructed real-world spaces via Omniverse NuRec. Hardware requirements include Ubuntu 22.04 or 24.04, at least 32 GB of RAM, an RTX GPU with at least 16 GB of VRAM, Isaac Sim 6.0, and Isaac Lab 3.0. At runtime, the exported policy consumes RGB camera streams, odometry, and navigation goals to publish velocity commands (/cmd_vel), with optional cuVSLAM odometry integration. COMPASS Framework and Agentic Workflow Architecture COMPASS simplifies cross-embodiment navigation by building on the pretrained NVIDIA X-Mobility policy. Rather than learning locomotion and navigation from scratch for each embodiment, COMPASS trains residual reinforcement learning specialists that apply corrections to the base policy's actions for a given robot and environment. These specialist models can later be distilled into a generalized cross-embodiment policy. The development pipeline is packaged into repository skills discoverable by coding agents such as Codex ($compass) or Claude Code (/compass). The agent executes routine repository tasks, smoke testing, and telemetry tracking, but stops at defined human approval gates before scene acceptance, full training, and checkpoint promotion. Environment Setup and Hardware Requirements The reference implementation uses Boston Dynamics' Spot quadruped and requires a system running Ubuntu 22.04 or 24.04 with at least 32 GB of RAM, an RTX-capable GPU with 16 GB or more of VRAM (a GeForce RTX 4080 is the minimum reference card for Isaac Sim 6.0), and Linux driver 580.95.05. The software stack operates in Docker 24+ containers running Isaac Lab 3.0 and Isaac Sim 6.0. Access requires a Hugging Face read token to download gated assets, including the simulation USD files and the x_mobility.ckpt model checkpoint. The agent handles environment validation and smoke testing, generating environment reports without storing or displaying private access tokens. Scene Preparation: Built-In, Generated, and Reconstructed Paths Developers can select from three environment types: the default combined_multi_rack warehouse, procedurally generated indoor environments from the SAGE-10K dataset, or captured real-world spaces processed through Omniverse NuRec. The built-in warehouse offers a validated baseline requiring no additional scene conversion. SAGE-10K scenes require USD inspection in Isaac Sim to verify scale, collision meshes, materials, and occupancy maps before running a one-environment smoke test. Omniverse NuRec transforms stereo RGB captures into simulation-ready assets with collision geometry, allowing real-to-sim fine-tuning and validation before scaling up RL training. Residual Training, Evaluation, and Diagnostic Gates Once a one-environment smoke test passes visual and physical inspection, residual RL is launched via run.py using Spot and the registered environment. Training executes across parallel environments configured by GPU memory, tracking reward values, fall rates, goal progress, and episode terminations. Rather than selecting the final iteration by default, periodic checkpoints are saved and evaluated against the base X-Mobility policy under matched seeds, initial positions, and termination conditions. Standard COMPASS evaluation assesses goal-reached rate, fall-down rate, and travel time. If issues arise, diagnostic skills like $compass-doctor conduct read-only health checks to isolate authentication, simulation, collision, or memory bottlenecks. Runtime Deployment and Odometry Integration The agent coordinates development and evaluation, but does not control the robot during execution. The exported COMPASS policy is packaged as a unified inference model within ROS 2. The compass_inference module processes front-camera RGB images, navigation targets, and odometry-derived robot speed, outputting forward-linear and angular-velocity commands directly to the /cmd_vel topic. In GPS-denied settings lacking reliable platform odometry, NVIDIA cuVSLAM can be added as a separate ROS 2 node to supply camera-based tracking and odom-to-base_link transforms to /chassis/odom. Unregistered robot embodiments follow a dedicated onboarding workflow ($compass-newembodiment) to configure kinematics, action mappings, and preview integration.

利用 AI Agent 训练跨实体机器人导航策略：COMPASS 工作流指南 将机器人导航能力迁移至新形态或新场景通常成本高昂且难以复现。英伟达介绍了基于智能体（Agent）驱动的 COMPASS 框架工作流，利用预训练基础策略与残差强化学习，结合人工把关点，实现从环境验证、场景准备、训练到运行时集成的跨实体导航策略构建。 COMPASS 复用预训练 NVIDIA X-Mobility 基础策略，通过残差强化学习训练专用策略校正动作，并可蒸馏为通用跨实体策略。 工作流程由编码智能体调用仓库技能执行依赖验证、冒烟测试与训练，并在场景采纳、单环境冒烟测试和检查点晋级等关键节点保留人工审批关卡。 支持三种场景源：内置仓库基线、SAGE-10K 生成场景数据集，以及利用 Omniverse NuRec 重建真实环境。 部署时由 compass_inference 处理 RGB 图像、导航目标与里程计信息并输出速度指令，必要时可结合 cuVSLAM 提供视觉里程计。 COMPASS 框架与 Agent 驱动架构 COMPASS（Cross-Embodiment Mobility Policy via Residual RL and Skill Synthesis）旨在通过单一实体的专家演示实现跨形态机器人的移动能力。该框架复用预训练的 NVIDIA X-Mobility 导航策略，利用残差强化学习为特定机器人和场景训练专用策略以修正动作，避免从零开始学习，并可将多实体数据蒸馏为共享策略。在开发过程中，Codex 等编码智能体通过仓库技能（skills）自动化环境配置、依赖检查和故障诊断，但运行时导航由训练好的策略及机器人控制器独立执行，无需智能体介入。 系统环境与开发配置流程 参考工作流以波士顿动力 Spot 四足机器人为基准，推荐配置包括 Ubuntu 22.04/24.04 系统、至少 32GB 内存、至少 16GB 显存的 RTX GPU（最低参考 GeForce RTX 4080）、Docker 24+ 与 NVIDIA Container Toolkit，并使用测试过的 NVIDIA Isaac Lab 3.0 与 Isaac Sim 6.0 软件栈。开发者需通过 Hugging Face 访问受控仓库并配置读取令牌，智能体在引导下完成资产与 X-Mobility 检查点下载，并在每个开发阶段（验证、场景准备、训练、评估、打包）生成供人工审查的日志与凭证。 三类导航场景的准备与验证 工作流支持三类场景来源：内置 combined_multi_rack 仓库可作为快速复现的基础基线；SAGE-10K 数据集提供 10,000 个生成的室内场景，需通过 Isaac Sim 检查几何结构、材质、碰撞网格并生成占用图（Occupancy Map）；Omniverse NuRec 则用于通过双目 RGB 采集数据重建真实部署环境。在进入正式大规模残差训练前，必须执行单环境冒烟测试，确认模型生成有效、相机观测可用且动作无穿模或跌倒等错误，并通过人工确认。 残差训练与指标评估机制 在通过冒烟测试审批后，智能体启动残差强化学习训练任务。训练期间需根据显存调整并行环境数量，监控奖励、目标进展、碰撞跌倒与资源占用，并定期保存检查点。评估环节要求在相同的随机种子、目标、初始状态和终止条件下，对比残差检查点与 X-Mobility 基础策略的表现。COMPASS 标准评估指标包括到达目标率、跌倒率和移动时间，仅在严格满足安全与导航门槛并获得人工批准后才推进检查点打包。 运行时集成与实体扩展 在 ROS 2 运行时环境中，compass_inference 节点接收前置相机 RGB 图像、导航目标及里程计计算出的速度，在 /cmd_vel 上发布线速度与角速度指令。如果部署环境缺乏可靠定位，可引入独立的 cuVSLAM ROS 2 节点提供视觉里程计支持。对于未注册的新机器人形态，则可借助 compass-newembodiment 技能完成接口映射与初始配置；模型导出（如 ONNX、TensorRT）和物理部署仍需针对目标硬件单独进行验证。

Navigation enables a robot to turn perception and motion into purposeful autonomy. Unlike locomotion, which produces stable movement, navigation must be used to continuously localize the robot, interpret changing surroundings, select a route, and avoid obstacles to reach a goal safely. Moving this capability to a new robot or scene can require new data, simulation assets, robot interfaces, training, diagnosis, and evaluation. Repeating that work for every robot-scene pair is expensive and difficult to reproduce. An agent-driven workflow reduces this burden. The developer defines the robot, scene source, and navigation goal. A coding agent uses repository skills to validate dependencies, prepare assets, run smoke tests, launch training, diagnose failures, and compare checkpoints. Human approval gates control scene acceptance, the one-environment smoke test, and checkpoint promotion. Using Spot as the reference robot, this post tutorial applies an agent-driven COMPASS workflow to a built-in scene and a SAGE-10K scene, while showing how NVIDIA Omniverse NuRec supports captured environments. It follows the policy workflow through smoke testing, residual training, checkpoint evaluation, and runtime integration, including optional odometry. What is COMPASS? COMPASS (Cross-Embodiment Mobility Policy via Residual RL and Skill Synthesis) is a unified framework that enables scalable cross-embodiment mobility using expert demonstrations from a single embodiment. It reuses navigation behavior from the pretrained NVIDIA X-Mobility policy. It trains a residual specialist, a reinforcement learning (RL) policy that corrects the base action for a selected robot and environment instead of relearning navigation from the beginning. Data from multiple specialists can later be distilled into a shared cross-embodiment policy. The COMPASS policy architecture that this agent-driven workflow trains and evaluates is shown in Figure 1. Figure 1. COMPASS adapts an X-Mobility base policy into embodiment specialists and distills them into a cross-embodiment policy COMPASS packages this development workflow as repository skills. This tutorial uses Codex during development. The trained policy and robot controller execute navigation at runtime without the coding agent. Reference workflow overview The reference workflow uses the Boston Dynamics Spot quadruped. The built-in warehouse is the primary reproducible path, SAGE-10K extends it to a generated scene, and NVIDIA Omniverse NuRec provides an optional path for a reconstructed target environment. NVIDIA cuVSLAM , a CUDA-accelerated visual odometry and simultaneous localization and mapping library, can provide deployment odometry when the robot does not already supply compatible odometry and transforms. For another environment, use the repository-pinned COMPASS software stack and the following hardware guidance: An Ubuntu 22.04 or 24.04 system with at least 32 GB of RAM, an RTX-capable NVIDIA GPU with at least 16 GB of VRAM, and Linux driver 580.95.05 , the version tested for Isaac Sim 6.0 . The Isaac Sim 6.0 minimum reference GPU is a GeForce RTX 4080. Run the Isaac Sim Compatibility Checker before installation. Docker Engine 24 or later with the NVIDIA Container Toolkit. A Hugging Face account and read token with access to the gated nvidia/COMPASS and nvidia/X-Mobility Hugging Face repositories. The tested tutorial stack: NVIDIA Isaac Lab 3.0 with NVIDIA Isaac Sim 6.0. Step 1: Set up the COMPASS agentic workflow First, prepare the repository and give the coding agent a clear workflow contract before beginning the scene work. You will download the gated assets, make the COMPASS skill discoverable to Codex, run stack checks, and stop at the one-environment approval gate. All $compass blocks are copyable prompts for the Codex chat at the COMPASS repository root, not shell commands. In Claude Code, use /compass for the same workflow. For Codex, first expose the repository skills under .agents/skills , then select COMPASS with /skills or mention $compass in the prompt. Codex supports symlinked skill directories, so the current repository skill can remain in its maintained location. For Claude Code, invoke the same workflow with /compass . mkdir -p .agents/skills ln -s ../../.claude/skills/compass .agents/skills/compass ln -s ../../.claude/skills/compass-doctor .agents/skills/compass-doctor ln -s ../../.claude/skills/compass-newembodiment .agents/skills/compass-newembodiment The coding agent can clone, build, download non-secret assets, and validate the stack. The developer must accept the gated repository terms and enter the Hugging Face token outside the chat. The agent should never request, display, or store the token in logs. Install COMPASS and download the assets Clone the COMPASS repository and follow the COMPASS Handbook quick start with the repository-pinned container. Before the first run, accept access to the gated nvidia/COMPASS and nvidia/X-Mobility Hugging Face repositories, create a Hugging Face read token , and confirm that it can read public gated repositories available to your account. Expose the token only in the current shell. Do not paste it into an agent prompt or commit it to source control. export HF_TOKEN=hf_xxx ./docker/run.sh assets ./docker/run.sh build source ./docker/activate The assets step downloads the registered simulation assets to ./assets/usd/ and the pretrained X-Mobility checkpoint to ./assets/x_mobility.ckpt . A 401 or 403 response usually indicates incomplete repository access or token scope. Resolve authentication before debugging Isaac Lab. Each phase produces reviewable evidence before the next phase begins: Validate: Software and asset inventory, environment report, and smoke-test log Prepare scene : Registered scene configuration, occupancy map, and visual inspection evidence Train: Pinned command and configuration, logs, telemetry, and periodic checkpoints Evaluate : Matched protocol, standard COMPASS metrics, videos, and a promotion recommendation Package: Approved checkpoint, configuration, evaluation record, and artifact manifest Approval criteria are project-specific, but every gate should answer the same question: are the required inputs present, did the expected outputs appear, are there unresolved errors, and is the evidence sufficient to continue? Invoke the COMPASS skill After the container is active, open the coding agent at the repository root and describe the robot, scene, navigation outcome, and approval gates. For the baseline workflow, copy this prompt into the agent chat: $compass Validate the COMPASS environment for Spot. Confirm the pinned repository revision, container, GPU, Isaac Lab and Isaac Sim versions, simulation assets, and pretrained X-Mobility checkpoint. Run a one-environment smoke test, save the validation report, and stop for approval. The $compass skill checks the requested workflow against the repository and runs the relevant validation steps. If a run fails, $compass-doctor performs a read-only health check and reports likely causes without silently changing the environment. Figure 2. A coding agent invokes COMPASS skills to validate the environment, prepare the scene, run smoke tests, and stop at human approval gates Step 2: Choose and prepare a navigation scene This section explains how to choose and prepare one of three scene sources: the built-in COMPASS warehouse, a generated SAGE-10K scene, or a captured environment rendered with Omniverse NuRec . You will learn what each path is for and which registration, occupancy map, and approval checks must be completed before training. Path 1: Use the built-in warehouse Start with the registered combined_multi_rack warehouse for the fastest reproducible baseline. The robot, scene, and occupancy map are already registered, which makes this the best path for validating the installation before introducing a new scene. Copy the following prompt into the coding agent to run the baseline and pause after the smoke test: $compass Train and evaluate Spot in the built-in combined_multi_rack warehouse. Stop after the one-environment smoke test for approval. Figure 3. A Spot quadruped navigates the registered COMPASS combined_multi_rack warehouse Path 2: Use a SAGE-10K scene The SAGE-10K dataset contains 10,000 generated indoor scenes across 50 room types. It is a scene dataset, not a policy or simulator. Each scene provides geometry, materials, layout metadata, and a preview. Living room and warehouse scenes follow the same preparation path, so select one suitable candidate instead of downloading the entire dataset. Figure 4. Spot quadruped in a converted SAGE-10K indoor scene prepared for COMPASS validation The SAGE-10K path includes two human approval gates. First, inspect the converted USD in NVIDIA Isaac Sim and confirm geometry, materials, scale, and collision meshes before registration. After registration and occupancy-map generation, approve the one-environment preview before full training. The occupancy map identifies free and blocked space for valid robot starts and navigation goals. Copy the following prompt into the coding agent to shortlist a scene and pause at both gates: $compass Find suitable SAGE-10K living-room or warehouse scenes for Spot and show the best candidates. After I approve a scene, convert and register it, generate and verify its occupancy map, and stop for inspection. After I approve the scene and map, run a one-environment smoke test and stop again before full training. Figure 5. Codex shortlists compatible SAGE-10K scenes and stops for developer approval Path 3: Introduce a captured environment with NuRec Use Omniverse NuRec when the goal is to fine-tune and evaluate COMPASS in a reconstruction of the intended deployment environment. NuRec converts stereo RGB captures into an Isaac Sim-ready reconstruction with aligned visual geometry, collision meshes, and optional scene augmentation. The documented COMPASS path registers the rendered scene, verifies its supplied occupancy map and origin convention, inspects robot clearance, and runs a one-environment smoke test before training. Figure 6. The NuRec Real2Sim policy (right) navigates around the table toward the goal, compared with the synthetic policy (left) NuRec is an optional path for the purposes of this post. The hands-on training flow continues with SAGE-10K so that the tutorial follows one scene from preparation through evaluation. For a captured environment, use the COMPASS NuRec workflow and the NVIDIA Isaac Sim NuRec guide , including the living room example , for scene preparation, training, evaluation, export, and ROS 2 deployment. Copy the following prompt into the coding agent to prepare a registered NuRec scene and pause before training: $compass Prepare the registered NuRec Real2Sim scene <scene_name> for <supported_robot>. Verify the supplied occupancy map and origin convention, inspect collisions and robot clearance, run a one-environment smoke test, and stop for approval before training. Step 3: Validate the robot-scene integration Once the selected scene is available to COMPASS, run a one-environment preview before scaling training. For a SAGE-10K scene, complete the earlier visual inspection and scene-registration approval gate first. Confirm that Isaac Sim starts, the scene loads, Spot spawns in a valid location, camera observations are available, and the robot responds to policy commands without clipping, falling, or producing unresolved simulation errors. Direct the coding agent to summarize the preview logs and visual evidence, identify any blockers, and stop for human approval. Proceed to residual training only after the scene, robot, observations, and action interface work together as expected. Step 4: Train the residual specialist This section explains how COMPASS adapts the pretrained X-Mobility policy to the selected robot and scene. You will launch residual reinforcement learning, monitor the run, save candidate checkpoints, and preserve the evidence required for evaluation. Launch residual training After the one-environment smoke test is approved, the coding agent can launch the standard residual RL workflow . The following command uses Spot and the built-in warehouse. Replace the environment key with the registered SAGE-10K scene when following the generated-scene path. After the smoke test is approved, the coding agent can launch the standard residual RL workflow . The following example runs the built-in warehouse baseline. Replace the environment key with the registered SAGE-10K scene when continuing the generated-scene path. python run.py \ -c configs/train_config.gin \ -o ./outputs/spot_combined_multi_rack \ -b ./assets/x_mobility.ckpt \ --enable_cameras \ --embodiment spot \ --environment combined_multi_rack Manage the training run Residual training is a long-running process. The coding agent should run it in a persistent session or managed scheduler, write logs and checkpoints to the configured output directory, and report progress without keeping the interactive session open. Before training, record the command, repository revision, scene key, configuration, checkpoint interval, and stopping criteria. If the run is interrupted, verify that the latest checkpoint is complete and confirm the supported resume options before continuing. Monitor training and save checkpoints Set --num_envs according to available GPU memory, using one environment only for the smoke test. During training, monitor reward components, goal progress, contacts and falls, episode terminations, throughput, and GPU memory. Save periodic checkpoints and evaluate them under matched conditions instead of assuming that the final iteration is best. COMPASS also supports distributed multi-GPU training for larger runs. Training time varies with the hardware, scene complexity, environment count, and stopping criteria. Diagnose failures and preserve evidence Use the COMPASS diagnostic workflow to investigate failures before changing the environment or training configuration. Route authentication errors to Hugging Face access checks, scene-loading or collision errors to scene preparation, camera or action-interface errors to the smoke-test stage, and memory errors to environment-count or multi-GPU configuration. Preserve the training configuration, command, repository revision, scene registration, occupancy map, smoke-test evidence, logs, checkpoints, and artifact manifest. Require developer approval before changing dependencies, scene assets, rewards, or training settings. This section shows how to determine whether a residual checkpoint is ready for promotion. You will learn how to compare the pretrained base policy and residual candidates under matched conditions, interpret the standard COMPASS metrics, label any derived evidence, and retain human approval before packaging. Review task performance and safety together. Standard COMPASS evaluation reports goal-reached rate, fall-down rate, and travel time. Additional evidence, such as goal progress, contact behavior, timeouts, or command stability, should be labeled as derived analysis or custom instrumentation. Promote a checkpoint only after the matched evidence satisfies the project’s navigation and safety gates, and a human approves packaging. The following prompt is an example. Adapt it to the robot, scene, checkpoints, and evidence required by your workflow: $compass Compare the pretrained X-Mobility base policy with the available Spot residual checkpoints in the selected scene under matched seeds, goals, initial states, rollout length, and active terminations. Report the standard COMPASS evaluation metrics, save matched videos and the exact evaluation command, clearly label any derived evidence, and stop for human approval before promoting or packaging a checkpoint. Step 6: Connect the policy to the robot runtime This section explains how the trained policy connects to the robot runtime after development is complete. You will learn the reference policy inputs and outputs, when cuVSLAM can provide deployment odometry, and when an unregistered robot requires the new-embodiment workflow . The coding agent coordinates development and validation; it does not control the robot at runtime. Understand the policy inputs and outputs The COMPASS asset step downloads the pretrained X-Mobility checkpoint used for smoke testing and residual training. Training creates the residual checkpoint for the selected robot and scene. Export and deployment package the trained policy for inference; they do not expose the base and residual as two ROS 2 components that developers must connect manually. In the reference ROS 2 integration , compass_inference converts front-camera images, the navigation target or route, and robot speed derived from odometry into the exported policy inputs. It publishes forward-linear and angular-velocity commands on /cmd_vel . Recurrent state and previous action are internal to the inference implementation, not external ROS integration inputs. Validate coordinate frames, update rates, normalization, command limits, stopping behavior, and the physical robot controller for the target deployment. Figure 7. At inference, the COMPASS policy uses RGB input, odometry and a goal point, and optional map and route data to produce linear and angular velocity commands for the robot controller Add cuVSLAM odometry when needed Use the cuVSLAM library when the deployed robot needs camera-based state estimation in a GPS-denied or GPS-intermittent environment and does not already provide compatible, validated odometry and transforms. Its odometry can support the COMPASS navigator, but its map is not an input to the navigation policy. cuVSLAM is not part of COMPASS policy training and does not require an agent skill. Run it as a separate, version-matched ROS 2 component, connect or remap its odometry output to /chassis/odom , provide the required odom-to-base_link transform, and validate calibration, timestamps, topic names, and frame conventions. The optional $cuvslam-onboard and $cuvslam-troubleshoot skills can help configure and diagnose this state-estimation component during development. Example prompt: $cuvslam-onboard Configure cuVSLAM as the odometry source for the COMPASS navigator on <robot and camera rig>. Select a compatible release and tracking mode, validate calibration and timestamps, connect odometry and TF to the expected COMPASS interfaces, and stop for approval before enabling navigation. Extend the workflow to another robot For a robot that is not registered, $compass-newembodiment guides the developer through robot configuration, environment registration, action mapping, and a one-environment visual smoke test. Onboarding a new embodiment is a separate engineering task from training a specialist for an existing robot, but it uses the same validation and approval pattern. This tutorial stops at checkpoint evaluation. Export to ONNX, JIT, or TensorRT, ROS 2 integration, and physical hardware deployment require separate validation for the target robot and runtime. Scene quality, training duration, and checkpoint performance vary with the embodiment, environment, reward design, and available compute, so the workflow does not define a universal success threshold. Get started with COMPASS Start with the reference path, then extend one component at a time: Set up the reference environment. Clone the COMPASS repository , follow the COMPASS Handbook quick start , accept the gated model terms, and download the COMPASS simulation assets and X-Mobility checkpoint . Run the agentic workflow. Invoke $compass in Codex with a supported robot and built-in scene, retain approval after the smoke test, and train and evaluate the specialist under matched conditions. Extend and package deliberately. Use $compass-newembodiment for an unregistered robot. Save the configuration, checkpoint, logs, matched evaluation results, and videos needed for the next engineering decision. To reproduce and extend the workflow, check out these resources: SAGE-10K dataset for generated indoor scenes Omniverse NuRec developer page , COMPASS NuRec workflow , and Isaac Lab NuRec guide for captured-scene reconstruction and navigation policy training NVIDIA Isaac Sim and NVIDIA Isaac Lab documentation for the simulation and robot-learning stack Isaac ROS Visual SLAM documentation for optional cuVSLAM-based deployment odometry