Using Multi-Agent AI Workflows and Omniverse Libraries to Prepare 3D Scenes for Robotics Simulation

Using Multi-Agent AI Workflows and Omniverse Libraries to Prepare 3D Scenes for Robotics Simulation Preparing 3D scenes for physical AI and robotics simulation is often a manual, time-consuming bottleneck. By combining high-level orchestrators like Codex with NVIDIA NemoClaw subagents and NVIDIA Omniverse Libraries, developers can automate the inspection, semantic labeling, sensor configuration, physics integration, and validation of Blender scenes into simulation-ready OpenUSD worlds for NVIDIA Isaac Sim and Isaac Lab. A multi-tier architecture uses Codex or Claude for coordination, NemoClaw subagents (powered by Nemotron models) for reasoning, and NVIDIA Omniverse Libraries for tool execution. The workflow converts Blender assets via Model Context Protocol (MCP) into OpenUSD, preserving scene hierarchies, physics metadata, and sensors in a layered, non-destructive structure. Dedicated subagents handle scene inspection, semantic segmentation, material attributes, sensor placement, rigid body physics via ovphysx, and preflight rendering using ovrtx. SimReady Foundation validation serves as an acceptance gate, automatically fixing safe mechanical errors while surfacing ambiguous issues to humans before handoff to Isaac Sim or Isaac Lab. Recommended hardware profiles span from desktop prototyping on NVIDIA DGX Spark to large-scale deployments on DGX Station, RTX PRO Servers, and DGX Cloud. The Simulation-Ready Bottleneck and Multi-Agent Architecture Robotics development frequently stalls because artists' 3D scenes lack critical simulation data, such as collision meshes, sensor placements, semantic labeling, and physical material parameters. General-purpose agents alone cannot solve this without specialized tooling. The proposed pattern relies on a coordinator (such as Codex powered by OpenAI GPT-6 Astra or Claude) managing specialized subagents built on frameworks like Hermes and deployed through NVIDIA NemoClaw. These subagents invoke NVIDIA Omniverse Libraries to inspect assets, apply changes, and validate outputs against simulation profiles. Scene Inspection and Transition to OpenUSD The pipeline begins by connecting an inspection subagent to Blender through a Model Context Protocol (MCP) server. The subagent inventories scene objects, transforms, lights, and missing metadata into structured data, which the orchestrator evaluates to plan downstream tasks. The workflow then commits scene data to OpenUSD, establishing a layered, non-destructive contract that allows subagents to append semantic labels, physics metadata, and sensor definitions without flattening original artistic assets. Semantic Labeling, Materials, and Sensor Authoring Downstream perception and task setups require more than geometric meshes. A semantic-labeling agent tags prims with functional labels (e.g., bins, obstacles, grabbable items) and flags ambiguous objects for human confirmation. A material subagent annotates surfaces with simulation metadata (such as concrete, rubber, or metal) needed for rendering, sensing, and domain randomization. In parallel, a sensor agent configures cameras and lidars early, checking fields of view, resolutions, and potential occlusions prior to simulator execution. Physics Configuration, Visual Preflight, and Validation Gates Physical properties are integrated through the ovphysx tool, which assigns collision shapes, rigid body behaviors, mass, and friction, auto-repairing basic errors like missing collision meshes. To ensure usability, ovrtx runs visual preflight renders from robot viewpoints to catch lighting artifacts or blocked sensors. Finally, a validation agent tests the USD scene against SimReady Foundation profiles. Safe errors are patched automatically, whereas uncertain labels or conflicting physics settings escalate to human reviewers before delivering the world to Isaac Sim or Isaac Lab. Hardware Infrastructure Recommendations NVIDIA outlines compute tiers for varying deployment scales: NVIDIA DGX Spark (128 GB unified memory) for local prototyping and MCP tool testing; NVIDIA DGX Station powered by Grace Blackwell Ultra superchips (up to 748 GB memory) for complex scenes and local model inference; and NVIDIA RTX PRO Servers alongside DGX Cloud for batch pipelines, synthetic data production, and distributed training workloads.

英伟达提出基于AI智能体的3D场景仿真准备工作流 在机器人与物理AI开发中，为仿真准备3D场景往往耗时费力且容易出错。英伟达介绍了利用Codex/Claude等编排智能体、NemoClaw子智能体以及Omniverse工具库的协同工作流，将Blender创作的场景自动补充语义、传感器与物理属性，并完成验证后交付至Isaac Sim或Isaac Lab。 机器人训练瓶颈常出现在场景缺乏仿真就绪性（SimReady），需要耗费大量人力补充物理与元数据属性。 采用“主智能体协调+NemoClaw子智能体推理+Omniverse库执行”的架构模式，实现自动化操作与人工审核结合。 工作流依托OpenUSD作为统一数据载体，并通过Blender MCP实现场景资产清点与编辑。 全流程包含场景检查、语义标注、材质增强、传感器预设、物理赋能、视觉预检及SimReady验证等8个步骤。 英伟达针对本地原型、桌面超算、企业团队及云端训练提供了DGX Spark、DGX Station、RTX PRO服务器和DGX Cloud等硬件部署推荐。 场景准备的痛点与智能体协作架构 3D美术人员制作的模型通常缺乏机器人仿真所需的标签、碰撞网格、传感器配置及物理材质。该方案提出将整体目标拆解为多智能体工作流：由Codex（基于GPT-6 Astra）或Claude担任主协调智能体，负责任务分解与依赖判断；通过NVIDIA NemoClaw部署基于Hermes等框架的专业子智能体进行逻辑推理；底层调用NVIDIA Omniverse Libraries工具库实施场景修改，形成可重复且安全的工程闭环。 资产检测与USD分层构建 首先，检查子智能体通过Blender MCP（模型上下文协议）服务清点资产层级、材质、灯光与缺失数据。随后，将数据统一转换为OpenUSD格式作为交互契约。利用USD的分层与非破坏性特性，智能体可在不破坏原创意资产的前提下，独立添加语义、物理属性与传感器配置，为后续工具提供共享真值。 语义增强、材质赋予与传感器早期配置 语义标注智能体将普通网格打上货架、箱子、障碍物及可抓取物体等类别标签，并对不确定项进行标记。材质智能体则将单纯的视觉外观补充为机器可读的仿真属性（如金属、纸板、混凝土等）。传感器子智能体提前设定相机与激光雷达（LiDAR）的位姿、视场角及分辨率，并在导入仿真前测试目标可见度与遮挡情况。 物理赋能、视觉预检及SimReady验证闭环 在物理层面上，ovphysx子智能体负责添加碰撞网格、刚体属性、质量与摩擦系数，自动修复安全问题并向人类汇报模棱两可的设计意图。视觉层面由ovrtx子智能体渲染机器人视点与审查视角，验证光照及视野是否正常。最后，验证智能体依据SimReady标准进行审查，生成修复清单或交由人工裁决，通过后方可直接移交Isaac Sim或Isaac Lab进行策略训练。 部署建议与硬件系统支持 为保障多智能体工作流与物理渲染的高效运行，英伟达推荐在不同阶段选用匹配的硬件方案：本地原型开发可采用具备128GB统一内存的NVIDIA DGX Spark；桌面级重度AI与推理工作流可采用搭载GB300芯片的DGX Station；团队级流水线适于部署NVIDIA RTX PRO服务器，而大规模训练与批处理仿真则可依托DGX Cloud提供弹性算力。

Agentic AI workflows can be used to prepare and validate digital twins for physical AI systems. Agents can inspect 3D scenes, author simulation-relevant data in OpenUSD , add physics properties, render preflight views, and validate the result against simulation-ready (SimReady) requirements. This workflow follows that process from a scene in Blender to a simulation-ready OpenUSD handoff for NVIDIA Isaac Sim or NVIDIA Isaac Lab . In practice, however, if you’re building agents for robotics, the workflow can often get stuck. It’s tempting to blame the hard part on the policy, the model, or the training loop, but the bottleneck often happens earlier: the robot does not have a simulation-ready world to train in. The extra work required to get a 3D scene into that state is laborious, time consuming, and frequently out of scope for the robotics simulation engineer. This post walks through an agent workflow for preparing a Blender scene for robotics simulation using NVIDIA Omniverse Libraries . Codex, powered by OpenAI GPT-6 Astra, coordinates the overall task, interprets results, and guides iteration. Specialized subagents built with the Hermes agent harness and deployed through NVIDIA NemoClaw use Omniverse Libraries to inspect the scene, author simulation metadata, configure physics, and render visual preflight views. Together, these components connect reasoning, tool execution, and validation into a repeatable process for delivering a simulation-ready OpenUSD world. Why is preparing a 3D scene for simulation challenging? The scene exists—the assets are there, created by a 3D artist in Blender—but is it usable for simulation? Has all of the following prep work been done? Are objects labeled? Are collision meshes correct? Are materials meaningful for simulation? Are sensors placed and configured? Does the scene export cleanly to USD? Can the robot perceive the target objects? Will the scene pass validation before you burn time debugging it in Isaac Sim or Isaac Lab? This prep work is tedious, repetitive, and easy to get wrong. It is also exactly the kind of work agentic systems should help with, if they have the right tools. The point of using NVIDIA Omniverse Libraries in an agent workflow is to integrate the tools agents need to build SimReady worlds. How can agents help prepare a 3D scene for simulation? A general-purpose agent such as Codex by ChatGPT or Claude Cowork by Anthropic can look at a Blender scene and recognize that it needs to be made simulation-ready. Recognition is useful, but it’s not enough. To help a robotics developer, the agent must be able to act on the scene: Inspect the Blender scene Identify missing simulation metadata Add semantic labels Configure sensors Author collision shapes and physics properties Render visual preflight views Run SimReady validation Automatically repair safe issues Escalate ambiguous decisions to a human A broad request to make a 3D scene ready for simulation becomes a multi-agent engineering workflow. Codex or Claude serves as the main agent, coordinating the overall task of preparing the scene for simulation. NVIDIA NemoClaw provides a reference architecture for building the specialized subagents that perform each job. These subagents can use open source agent harnesses such as Hermes, OpenClaw, or LangChain, configured with different NVIDIA Nemotron models for vision, reasoning, and tool use. In a configuration using Astra and Hermes, Codex uses Astra to translate the developer’s objective into tasks, identify dependencies, and review results from specialized Hermes subagents deployed through NemoClaw. For example, making an object grabbable requires coordinated updates to its semantic label, rigid-body configuration, and collision geometry. Astra helps connect those requirements across subagents and determine which checks are needed before the workflow proceeds. NVIDIA Omniverse Libraries provide the tools the subagents call to act on the scene. OpenUSD operations establish the shared scene structure, ovphysx authors and checks physics properties, ovrtx renders visual preflight views, and SimReady validation evaluates the resulting assets against a target simulation profile. Each subagent owns a specific job and its acceptance criteria. Safe, mechanical issues can be fixed automatically. Decisions that depend on developer intent, such as an uncertain semantic label or physical behavior, are escalated to a human with the relevant context and a proposed next step. The pattern is: Codex or Claude coordinates NemoClaw agents reason Omniverse Libraries act Together, these layers turn the prompt, “Make this scene simulation-ready” into a tool-driven workflow with specialized jobs, a persistent scene state, validation gates, and human review where judgment matters. Figure 1. Reference workflow demonstrating how to use Omniverse Libraries to activate a Blender scene with agents How to use agents to prepare a 3D scene for simulation Begin by specifying the main objective for the orchestration agent, including the input, desired output, destination, and validation criteria. This gives Codex or Claude enough structure to coordinate the overall task, route work across specialized NemoClaw subagents, and decide when the job is actually complete. Input: Blender scene Goal: Prepare it for robotics simulation Output: USD-based simulation-ready world Destination: Isaac Sim or Isaac Lab Validation: Visual preflight + SimReady validation By specifying the main objective, the task changes from simply “make this scene better” to a coordinated agent workflow. Codex or Claude manages the overall request, NemoClaw subagents reason through specialized jobs, and Omniverse Libraries provide the tools that modify, render, validate, and prepare the world. After setting your goal, follow the steps below. Step 1: Inspect the scene through Blender MCP The first subagent connects to Blender through a Model Context Protocol (MCP) server and uses it to inventory the scene. MCP gives the agent a controlled tool interface into Blender: instead of guessing from screenshots or relying on manual exports, the agent can call tools to inspect objects, collections, transforms, materials, cameras, lights, and scene metadata. That scene inventory becomes the shared context the rest of the subagents use. This subagent should answer: What objects exist? What collections and hierarchy exist? What materials are assigned? What cameras and lights are present? What looks like a robot target, obstacle, floor, shelf, or bin? What simulation data is missing? The output should be structured as follows: { "objects": 142, "materials": 37, "missing": [ "semantic_labels", "collision_meshes", "camera_sensors", "physics_materials" ] } This provides a shared starting point for the other subagents. The Hermes inspection subagent returns its structured findings to Codex. Astra uses this inventory and the developer’s robotics objective to identify missing information and plan the next tasks. For example, identifying the robot’s target objects helps determine which assets need movable-body properties and which sensor viewpoints require review. Codex then delegates those tasks to the relevant Hermes subagents deployed through NemoClaw, with clear acceptance criteria and unresolved assumptions flagged for developer input. The workflow in this post uses The Junk Shop by Alex Trevino (original concept by Anais Maamar) for the demo (Figure 2). Codex coordinates NemoClaw to orchestrate which subagent is needed for the requested task. In this instance, NemoClaw is working through Blender MCP to run The Junk Shop scene and inventory the objects, materials, scene structure, and more. Figure 2. The Blender add-on bridges scene data to Omniverse Libraries through gRPC, enabling rendering and physics capabilities to feed directly back into the Blender Viewport Step 2: Move toward USD as the contract Blender is the authoring environment. USD is the simulation handoff because it gives agents and downstream tools a shared, structured representation of the world. Once the scene is authored into USD, subagents can inspect prims, add metadata, validate requirements, and pass the same world forward to Isaac Sim or Isaac Lab without relying on fragile one-off exports. The USD authoring agent uses Omniverse Libraries to preserve hierarchy, transforms, materials, labels, physics metadata, and sensor definitions. A useful rule for agent builders is: if another agent or simulator needs to rely on it later, author it into USD. USD is built for layered, nondestructive scene composition, so agents can add labels, physics metadata, sensor definitions, materials, and validation data without flattening the original creative work. That keeps the workflow from becoming a collection of temporary edits trapped inside one tool and gives every downstream step a shared, inspectable source of truth. Figure 3. USD composes the final scene from discrete sublayers (assets, lighting, and materials), each editable in isolation before being combined Step 3: Add semantic labels Robots do not just need geometry. They need meaning. The semantic-labeling agent turns anonymous meshes into task-aware objects: shelves, bins, floors, obstacles, grabbable items, and robot targets. By authoring those labels into USD, the workflow gives downstream agents and robotics tools a shared vocabulary for perception, validation, synthetic data, and training setup. The semantic-labeling agent tags prims with task-relevant classes: shelf bin box floor obstacle grabbable_object robot_target no_go_zone The agent can infer labels from object names, hierarchy, shape, and context. But it should also flag uncertainty: Tagged 118 prims. 9 labels need review. Figure 4 shows Codex orchestrating NVIDIA NemoClaw. NemoClaw coordinates with subagents through Blender MCP. NVIDIA Omniverse Libraries are the tools the subagents act with to do the task. In this instance, the ovrtx agent inspects the Blender scene through this workflow and applies the semantic segmentation and labels required for the scene to become robotics simulation-ready. Figure 4. Codex orchestrating NVIDIA NemoClaw to complete a task through Blender This is important because labels become the bridge between scene content and robotics workflows: perception, task setup, synthetic data, and validation. Step 4: Make materials simulation-aware A material that looks fine in Blender may still be incomplete for simulation. In a viewport, it may be enough for a shelf to look metallic or a bin to look plastic. In a robotics workflow, those surfaces need material attributes that downstream systems can use for rendering, sensing, physics, domain randomization, and validation. The material agent turns visual appearance into simulation-useful metadata. The material agent should inspect visual materials and author simulation-relevant material metadata. In a warehouse scene, that could mean identifying metal shelving, cardboard boxes, plastic bins, concrete floors, rubber wheels, or glass panels. The goal is not prettier materials. The goal is more useful information for simulation and validation. Figure 5. In Codex, a NemoClaw completes a task through Blender. A Material Subagent uses the MCP to communicate with Blender, identify materials in the 3D scene, inspect their properties, and tag in metadata appropriately for the materials to be machine-readable Step 5: Author sensors early If the robot will need to perceive the world, sensors should not be an afterthought in the training environment. Camera and lidar configuration shapes what the robot can observe, what data gets generated, and whether the training scene reflects the real task. Authoring sensors early lets agents validate placement, field of view, range, polling rate, occlusion, and target visibility before the scene reaches Isaac Sim or Isaac Lab. A sensor subagent can author camera and lidar sensors into the scene with: Position Orientation Field of view Polling rate Range Resolution Target frame This approach allows the workflow to surface useful questions before training starts: Can the robot see the target? Is the sensor blocked? Is the field of view useful? Are the training objects visible from the expected viewpoints? Figure 6 shows NemoClaw orchestrating a subagent to call the tool needed for the job: ovrtx. ovrtx loads a scene containing a configured lidar, warms up the sensor pipeline, renders one point-cloud frame, reads valid point data using the count channel, prints summary statistics, and visualizes the points with intensity-based colors. Figure 6. NemoClaw orchestrates a subagent to call the tool needed for the job: ovrtx Step 6: Use ovphysx for physical readiness At this point, the scene is no longer only visual. It becomes a physically aware digital twin world. The objects are no longer just meshes with materials; they have collision shapes, mass, friction, rigid body behavior, and rules for how they interact. That means a box can be picked up, a shelf can block motion, and a robot can test actions against a world that behaves physically instead of just looking correct. The ovphysx subagent adds or validates the following: Collision meshes Static colliders Rigid bodies Mass properties Friction Restitution Physics materials Movable versus fixed objects Common failures are exactly the tedious issues that are difficult to address later: 46 objects missing collision meshes. 12 grabbable objects marked static. 7 collision meshes too complex. 3 props floating above the floor. The ovphysx agent owns this job. It turns those findings into a repair plan, applies safe fixes automatically, and routes ambiguous cases to a human. For example, the agent can generate simple collision meshes for static props, mark floors and shelves as fixed colliders, assign rigid-body properties to grabbable objects, and flag anything where the physical behavior depends on task intent. The output is not just a cleaner scene; it is a physics-readiness report that downstream agents and validation tools can use. Figure 7 shows NemoClaw orchestrating a subagent to call the appropriate tool needed to make the 3D scene physics enabled: ovphysx. ovphysx is used to add the rigid body properties, colliders, mass, and friction properties to ensure that the scene is simulation-ready when it is transferred to Isaac Sim or Isaac Lab. Figure 7. NemoClaw orchestrates a subagent to call the appropriate tool needed to make the 3D scene physics enabled: ovphysx Step 7: Use ovrtx as the preflight loop Why render before simulation? Because validation can tell an agent that the scene is structurally acceptable, but rendering shows whether the scene is usable. The ovrtx agent can generate robot-camera and review views, check for hidden targets, bad lighting, clipped sensors, broken materials, or unreadable objects, and route issues back to the right fix agent before training time is wasted. The ovrtx agent renders review and robot viewpoints so the workflow can check key points: Are target objects visible? Are labels attached to the objects that can be seen? Are materials rendering correctly? Is the lighting physically plausible? Is the camera blocked or clipped? Does the object scale look plausible? At this point, ovrtx is providing visual QA for the agent pipeline. The Hermes rendering subagent, running within the NemoClaw environment, calls ovrtx to generate review images and returns them with the relevant scene metadata to Codex. Astra can use this evidence to investigate discrepancies and coordinate targeted follow-up tasks. If a labeled target is absent from a robot-camera view, Codex can ask the sensor and scene-inspection subagents to check camera orientation, clipping settings, and possible occluders. After a correction, the rendering subagent produces another view so the result can be checked. This connects visual review to an actionable repair loop. Figure 8. NemoClaw orchestrates the subagents to call ovrtx and run a QA report, checking that the target objects are visible, labels are attached, materials render properly, lighting is physically plausible, and the camera is clear and unclipped Step 8: Run the SimReady validation Finally, the validation agent runs SimReady validation against the target profile. This is the acceptance gate for the agent workflow. SimReady Foundation defines standards and validation profiles for simulation-ready USD content, and the validation agent uses those profiles to check whether the scene is actually ready to move forward. If validation fails, the report becomes a task list for the fix agents. If it passes, the scene is ready for Isaac Sim or Isaac Lab handoff. The report should be actionable: Validation failed: 14 issues - 10 auto-fixable - 4 require review Fix agents can repair safe issues. Ambiguous failures are reported to a human. For example, “I fixed 10 validation issues automatically. Four require review: two uncertain semantic labels, one grabbable object with conflicting physics settings, and one object that may be either an obstacle or a target.” The human approves the intended behavior, then the agents apply the fix and rerun validation. The Hermes validation subagent returns the SimReady report to Codex, where Astra helps determine which repairs and follow-up checks are needed. Codex delegates those tasks to the appropriate Hermes subagents deployed through NemoClaw. Moving a sensor may require another ovrtx visibility check, while changing an object’s role may require updates to both semantic labels and physics properties. The subagents apply approved changes and rerun the relevant checks, and Codex summarizes the changes, validation evidence, and unresolved decisions for the developer. The goal is a scene that meets the simulation contract, not simply a file export. Figure 9 shows NemoClaw orchestrating the subagent needed to call the SimReady Blender addon, which is used to validate the scene on the target SimReady profiles. If the validation reports any failures, the agent flags for human review before proceeding. Figure 9. NemoClaw orchestrating the subagent needed to call the SimReady Blender addon Get started preparing 3D scenes for simulation Robot training does not start when the policy runs. It starts when the world is ready, which takes more than a scene that looks good. It requires a USD world with semantic labels, simulation-aware materials, sensors, physics properties, visual preflight, and validation against a target profile. That is too much tedious glue work to leave entirely to humans, and too concrete to solve with prompts alone. The useful pattern is a set of subagents with real tools: Blender MCP inspects the scene. Omniverse Libraries author the world. USD carries the contract. Semantic labels add meaning. Sensors define perception. ovphysx makes it physical. ovrtx makes it visually testable. SimReady validation makes it acceptable. Isaac Sim / Isaac Lab makes it trainable. The path forward is agentic engineering with real tools: Codex or Claude to orchestrate, NemoClaw to coordinate the subagents, and NVIDIA Omniverse Libraries to allow those agents to act on the scene. Recommended systems include: NVIDIA DGX Spark : Ideal for local prototyping, with 128 GB of coherent unified system memory. Use it to develop NemoClaw subagents, connect to Blender through MCP, author USD, run SimReady validation, and test ovrtx or ovphysx loops from a desktop system. NVIDIA DGX Station : Ultimate deskside AI supercomputer powered by the NVIDIA GB300 Grace Blackwell Ultra Desktop Superchip with up to 748 GB of coherent memory for local AI development, inference, and agentic workflows, configurable with an additional NVIDIA RTX PRO 6000 Blackwell Generation Workstation GPU. Use it when scenes get larger, local models need more memory, more subagents are running in parallel, or simulation and visualization workloads become more demanding. NVIDIA RTX PRO Servers : Best for team-scale pipelines. Use them for shared agent workflows, batch scene preparation, large OpenUSD assets, synthetic data generation, and production-style validation runs. NVIDIA DGX Cloud : Best for cloud-scale development. Use it when the workflow needs elastic compute for large training jobs, batch simulation, or scaled physical AI pipelines beyond local hardware. Ready to get started? Select one scene-prep bottleneck, give it to a subagent, connect it to an Omniverse tool, and add a validation gate. To learn more, check out these resources: Browse more Omniverse samples on the Omniverse Labs GitHub repo Explore NVIDIA Omniverse Libraries for agent-callable tools across USD, rendering, physics, storage, and validation Try SimReady Foundation to understand validation profiles and simulation-ready USD requirements Build toward Isaac Lab or Isaac Sim for robot learning Learn how NemoClaw helps build specialized agent. Become an OpenUSD developer and learn the foundation of agentic 3D workflows. Join us on September 30 at 11:00 am Pacific time for OpenUSD Insider Livestream: Developing a Physical AI Simulation Live with GPT-6 Astra and NVIDIA Omniverse Libraries .