NVIDIA Demonstrates Holoscan Application Development Using CLI, Skills, and AI Coding Agents

NVIDIA Demonstrates Holoscan Application Development Using CLI, Skills, and AI Coding Agents NVIDIA demonstrated an engineer-guided, agentic workflow to build a real-time endoscopic tool segmentation application on the Holoscan platform. Using Codex with GPT-5.6 Sol max mode alongside HoloHub documentation, development skills, and the Holoscan CLI, the project achieved an iterative development loop that reduced latency and optimized token and time expenditures. Development proceeded in iterative steps: goal decomposition, minimal application scaffolding, reproducible benchmarking, latency optimization, and final handoff. The application reused a MONAI deep learning segmentation model and sample video, wrapped inside a new dashboard featuring live telemetry and HoloViz overlays. Optimization lowered mean application-path latency by 33.6% (from 4.891 ms to 3.247 ms) and boosted rendered throughput by 50.5% (from 204.0 FPS to 306.9 FPS). An ablation study showed that combining CLI, skills, and documentation required only 40 minutes and 11 million tokens, outperforming configurations lacking skills or CLI guidance in execution speed, code quality, and token efficiency. Iterative Development Architecture Rather than attempting full generation from a single prompt, the workflow divided the target into small, verifiable engineering iterations. The AI coding agent leveraged the Holoscan CLI via the ./holohub wrapper, HoloHub documentation structured in a progressive disclosure pattern via agents.md, and specialized HoloHub skills such as holohub-app-lifecycle and holohub-debug-build-run. This setup allowed the developer and agent to execute and inspect identical commands across containers and local environments. Application Scaffolding and Repeatable Benchmarking In the initial iteration, the agent inspected reference examples, created standard scaffolding using './holohub create', and integrated video replay, preprocessing, TensorRT inference, segmentation postprocessing, and HoloViz overlays in 40 minutes of agent processing time. The next iteration introduced three distinct operational modes—visual, smoke, and benchmark—integrating Holoscan Data Flow Tracking across 300 tracked frames to capture reproducible latency plots and headless test verdicts. Latency Reduction and Optimization To improve performance without skipping model inference on any frame, the agent optimized dashboard overhead. It reused HoloViz input specifications and static coordinate tensors across frames and queued the 10-value GPU-to-host telemetry copy asynchronously across two pinned buffers. These revisions reduced mean application-path latency from 4.891 ms to 3.247 ms, reduced 95th-percentile latency by 27.4% to 4.554 ms, and increased rendered throughput from 204.0 FPS to 306.9 FPS across five measured trials. Ablation Study on Tooling and Skills An evaluation of prompt configurations highlighted the utility of combined development tooling. Providing CLI, skills, and documentation yielded a complete, properly containerized application using standard Holoscan operators in 40 minutes and 11M tokens. Providing CLI and documentation without skills caused trial-and-error host probing, increasing time to 65 minutes and token consumption to 20M. Supplying documentation alone took 40 minutes and 15M tokens but generated suboptimal code that ignored optimized TensorRT operators and ran 2.6 times slower.

利用 CLI、Skills 与 AI 编码智能体开发 NVIDIA Holoscan 应用 NVIDIA 探索了利用通用 AI 编码智能体结合 Holoscan CLI、开发技能（Skills）和 HoloHub 文档资产，以多轮迭代方式构建实时内窥镜工具分割应用的完整流程与优化效果。 采用工程师引导与智能体执行的循环：工程师设定目标与约束，智能体查阅范例并通过 CLI 完成脚手架生成、编码与测试，工程师评估后推进下一轮。 经过三轮迭代依次实现了端到端最小可用原型、包含三类模式的可重复基准测试，以及低风险性能优化（吞吐量提升 50.5% 至 306.9 FPS，平均延迟下降 33.6% 至 3.247 ms）。 消融实验表明，同时提供 CLI、Skills 与文档组合时效率最高且代码质量最好，消耗 11M Token 及 40 分钟即可生成符合规范的应用。 迭代开发流程与原型构建 开发目标是基于现有的 MONAI 内窥镜工具分割模型构建实时应用。在第一轮迭代中，智能体在限定不修改模型权重的约束下，利用 holohub-app-lifecycle 技能与 ./holohub CLI 生成应用脚手架，串联视频回放、预处理、TensorRT 推理、SDK 分割后处理及 HoloViz 视觉覆盖，历时约 40 分钟完成端到端原型。 引入基准测试与性能调优 第二轮迭代为应用确立了交互式 visual、快速验证 smoke 和无显示导出指标的 benchmark 三种运行模式，通过数据流跟踪实现了稳定的延迟测量。第三轮迭代针对延迟进行优化，智能体保留了逐帧推理设计，通过复用 HoloViz 输入规格、静态坐标张量以及异步双缓冲队列传输 GPU 到主机遥测数据，使渲染吞吐量提升至 306.9 FPS，P95 应用路径延迟降低 27.4% 至 4.554 ms。 资源组合消融实验对比 在相同环境及 Codex（GPT-5.6 Sol max）配置下，仅有文档/范例时虽耗时 40 分钟、消耗 15M Token，但智能体绕过了现成的 TensorRT 算子导致运行速度慢 2.6 倍；若提供 CLI 但缺少 Skills，智能体会使用通用 Bash 试错，耗时 65 分钟并消耗 20M Token；结合 CLI、Skills 与文档的配置兼具规范的代码结构与最低的资源开销。

NVIDIA Holoscan is a platform for building real-time AI applications at the edge, from medical imaging to robotics. HoloHub is its companion repository: a growing collection of reference applications and components that demonstrate what’s possible. We wanted to explore how a general-purpose coding agent could use the same examples, documentation, and development tools available to an engineer in an actual development task. In this post, we walk through building a real-time endoscopic tool segmentation application using an AI coding agent. HoloHub examples and documentation provide implementation patterns, and the development skills guide the agent through the HoloHub development process. The Holoscan CLI , invoked via ./holohub wrapper, provides the shared execution interface. The agent can discover development operations through the CLI, while the engineer can inspect and repeat the same commands. The development workflow proceeds in iterations: The engineer defines a goal and constraints The coding agent inspects relevant examples, implements the application-specific code, and uses CLI to run the required development operations The engineer reviews the code, outputs, and tests, then sets the goal for the next iteration The workflow is agent-agnostic; in this example, we used Codex with GPT-5.6 sol max mode, and the agent processing times mentioned were approximate. Figure 1. AI-assisted development and review workflow Setup and development target The overall objective is an end-to-end endoscopic tool segmentation application: real-time inference with a live visualization of the segmentation masks, along with statistical analysis rendering. We reused the existing MONAI endoscopic tool segmentation model and a Holoscan sample video, and confirmed that the existing application monai_endoscopic_tool_seg worked locally. We focused on a new application reusing the deep learning segmentation pipeline, and adding comprehensive visualization, runtime telemetry and repeatable benchmarking. The agents were additionally provided with: Holoscan CLI with Bash execution permission HoloHub repository with documentation in a progressive disclosure pattern via the agents.md HoloHub development skills, including holohub-app-lifecycle and holohub-debug-build-run The next sections show how the pieces work together in an engineer-guided, agentic development workflow. Iteration 0: Divide the goal Instead of trying to build the entire application with a single prompt, developers should decompose the ultimate objective into smaller, verifiable engineering iterations guided by uncertainty and evidence. This approach ensures that design choices are reviewed in a timely manner. The comprehensive goal can thus be structured into a sequence of reviewable topics: Is the development environment configured correctly to run a similar existing application locally? Can the existing model and video run in a separate end-to-end application? Does the visualization present meaningful information? Can latency be measured repeatedly? Can rendering throughput be improved without feature regressions? Each iteration produces reviewable code, outputs, and tests, which inform the prompts and design choices for the next iteration. Iteration 1: Create a minimal working application The first prompt defined the outcome while constraining model and data reuse. Developer prompt 1: Use $holohub-app-lifecycle to create a separate new Python HoloHub application for displaying endoscopic tool tracking as model outputs https://github.com/Project-MONAI/model-zoo/tree/dev/models/endoscopic_tool_segmentation. Reuse the MONAI endoscopic tool segmentation model, sample data, preprocessing, and inference. Show the model-derived mask, coverage and timeline, and useful uncertainty measurements in a polished HoloViz overlay. Do not train or modify the model weights. Make the sample video work end to end. That left implementation choices to the agent while keeping model reuse, visual evidence, and weight integrity explicit. The agents collected information from the designated sources: read the app lifecycle skill, nearby HoloHub examples, project metadata, and CLI documentation. Different types of actions were taken as expected: Inspected the relevant endoscopy, segmentation, HoloViz, recording, and testing patterns; monai_endoscopic_tool_seg , endoscopy_tool_tracking , surgical_scene_recon were particularly useful references Dry-ran and invoked ./holohub create to generate and register the standard scaffold Implemented the application graph, execution modes, tests and documentation using existing Holoscan operators and assets Built and ran the application via ./holohub run with the application metadata defined by the CLI The resulting application connected video replay, preprocessing, TensorRT inference, the SDK segmentation postprocessor, telemetry, and HoloViz. Inference and mask postprocessing ran for every replayed frame. The overlay reported frame-derived measurements. Figure 2. Endoscopic tool segmentation dashboard The agentic processing time was 40 minutes. The developer could review the live app through the same CLI used by the agent: ./holohub run endoscopy_tool_segmentation_dashboard visual --language python After reviewing the implementation, the visual outputs, and the test cases, we confirmed that the reused model and sample video worked in the new application. Visual review also showed that the overlay needed clearer measurements, so we proceed to define the next iteration. Iteration 2: Make future reviews repeatable by implementing benchmarking The second prompt turned the visual demonstration into a repeatable development artifact: Developer prompt 2: Revise the visual output, add more meaningful statistics, tool area, mask motion, temporal intersection-over-union as a stability indicator, edge entropy, FPS, bounding box position, and remove values that remain unchanged during replay. Add a benchmark mode that records actual latency and plots the results in Python. Export the figures to the build folder and also show them interactively when the environment supports it. In response, the agent revised the dynamic measurements and screenshot readability, then made visual review and benchmarking explicit application modes. The app was implemented to have three named application modes: Mode Contract visual Run the full sample at source pace in an interactive window smoke Quick 60-frame headless recording with a finite verdict benchmark Process 300 frames offscreen and export measurements and plots Table 1. The three execution modes available via ./holohub run With the Holoscan CLI and application lifecycle management, the modes and tests are discoverable and runnable without remembering a detailed container and application script recipe: ./holohub modes endoscopy_tool_segmentation_dashboard --language python ./holohub run endoscopy_tool_segmentation_dashboard benchmark --language python ./holohub test endoscopy_tool_segmentation_dashboard --language python Benchmark mode used Holoscan Data Flow Tracking for the configured path from the video replayer through preprocessing, inference, telemetry, offscreen HoloViz, and the rendered-frame sink. It effectively reused the ideas presented in the existing holoscan flow benchmarking module . The agentic processing time was 20 minutes. Figure 3. Performance benchmark of the endoscopic application, comparing application-path latency over 300 tracked frames The revised benchmark provided more repeatable measurements. With that baseline available, the next iteration could investigate performance without relying on visual checks alone. Iteration 3: Investigate and improve latency Once the application was measurable, the developer issued a third prompt: Developer prompt 3: Check whether the deep-learning model runs on every frame. Investigate ways to reduce latency, including approaches that take advantage of similar neighboring segmentations, and show the new benchmark results. In response, the agent confirmed that inference still ran on every frame. It considered reusing masks across neighboring frames, which could avoid some inference work but would require a policy for deciding when stale output is acceptable. For this engineering iteration, it kept inference on every frame and first removed lower-risk dashboard overhead: Reused HoloViz input specifications and static coordinate tensors while refreshing text and dynamic geometry for every frame. Queued the current 10-value GPU-to-host telemetry copy asynchronously in two pinned buffers and rendered with the previous completed values. We compared the first iteration’s implementation with the final one on the same test system. The optimized version was faster in all five measured trials. The agentic processing time was 30 minutes. Measurement Before After Change Rendered throughput 204.0 FPS 306.9 FPS 50.5% higher Mean application-path latency 4.891 ms 3.247 ms 33.6% lower P95 application-path latency 6.273 ms 4.554 ms 27.4% lower Table 2. Performance comparison between the baseline and optimized endoscopy dashboard implementations Figure 4. Final performance comparison between the baseline and optimized versions of the endoscopy dashboard Final handoff prompt and revalidation After the three engineering iterations, the developer gave the agent a separate handoff prompt. Developer prompt 4: Commit the implementation and keep the benchmark logs. To finalize the progress, we reran the application and the tests, verified the output figures and headless test results. The implementation and benchmark evidence were retained along with the commit hash and dependency versions via ./holohub env-check and ./holohub env-info . The development iterations show what the workflow produces. To better understand the impact of the CLI, skills and documentation, we next compare the same development task under different combinations of these resources. Ablation study We compared resource usage using the same coding agent and sandbox environment across identical settings (note that the study was last conducted on Aug. 1, using Codex 0.146.0 with GPT-5.6 Sol at max reasoning effort). All evaluations were based on the single prompt used in the first iteration (with the skill name removed if unavailable) with the goal of creating a new application. CLI + skills + docs/examples (this blog) The agentic processing time was 40 minutes, with a total cost of 11M tokens. CLI + docs/examples The agents were given agents.md including references to CLI usage guide and docs, as well as the HoloHub codebase. No HoloHub development skills were provided. The agentic processing time was 65 minutes, with a total cost of 20M tokens. While the workflow resulted in a complete application that achieved the implementation goal, the process was less efficient. The agents correctly located similar apps within the HoloHub repository, but tended to use generic Bash tools that required more trial-and-error probing of the development environment. For instance, they often applied generic linting tools before CLI-based linting, or attempted to install Python dependencies and run inference scripts directly on the host (should have consistently worked in a container). docs/examples The agents were given the HoloHub codebase but without the agents.md and explicit guidance on the CLI usage. No HoloHub development skills were provided. The agentic processing time was 40min, with a total cost of 15M tokens. No CLI guidance was provided, the coding agents still gained understanding of the CLI (from the overall codebase examples) and used it as the main dev tooling. However the coding quality is suboptimal compared to the other two settings: The 3rd party model config and code were embedded in the application code incorrectly A Dockerfile was created without leveraging the existing HoloHub base image already has all the required dependencies Existing optimized Holoscan operators such as TensorRT inference and format converter were ignored in the implementation, as a result this version is 2.6x slower then the other two settings While follow-up prompts could potentially address these issues, the combination of CLI, skills, and documentation/examples provided the best developer experience with the lowest resource overhead. Cost to first end-to-end app CLI + skills + docs/examples (this blog) docs/examples CLI + docs/examples Agentic processing time 40 min 40 min 65 min Token usage 11M 15M 20M Outcome Separate application using standard Holoscan operators Application required rework, used custom PyTorch/MONAI inference instead of Holoscan InferenceOp/TensorRT Reviewable application, with more agentic host-side validation and retries Table 3. Agentic processing time and token usage across three scaffolding configurations A co-development loop for engineers and agents This post follows a small, verifiable development loop. The result is an engineering prototype built on an existing model, sample video, and Holoscan components. It runs end to end, provides several application modes and automated tests, preserves the model weights, and records reproducible benchmark evidence. The main takeaway is the development loop shared by the developer and the agents: ./holohub supplies consistent operations, the skill encodes project-specific sequences and checks, and examples and documentation provide engineering context. The agent and the developer use the same CLI commands. The developer can focus on defining the objectives, setting the constraints, evaluating trade-offs, and deciding whether the evidence is sufficient. References HoloHub: https://github.com/nvidia-holoscan/holohub/tree/holoscan-sdk-4.5.0 Holoscan CLI: https://github.com/nvidia-holoscan/holoscan-cli/tree/v4.5.0 HoloHub skills source: https://github.com/nvidia-holoscan/holohub/tree/main/skills HoloHub skills at Nvidia catalog: https://build.nvidia.com/skills?filters=library%3Alibrary_holoscan&q=holohub