NVIDIA Introduces DIN Deploy for Building Local C++ AI Applications with TensorRT RTX

NVIDIA Introduces DIN Deploy for Building Local C++ AI Applications with TensorRT RTX NVIDIA has released Do Inference Now (DIN) Deploy, an open-source collection of C++ samples pairing ONNX Runtime with the TensorRT RTX execution provider to run hardware-accelerated AI models locally on Windows and Linux. DIN Deploy separates model conversion from deployment using Python export scripts for ONNX models alongside native C++ CLIs built on ONNX Runtime. Supported workloads include automatic speech recognition with Whisper, Parakeet, and Nemotron, segmentation with Meta SAM 2.1, and image generation using FLUX.2-klein-4B. The repository leverages ONNX Runtime graphics interop with Vulkan and DirectX, as well as post-training quantization via NVIDIA Model Optimizer. Benchmarks on DGX Spark demonstrate substantial GPU speedups over CPU across ASR transcription rates and SAM 2.1 frame rates. Architecture and Cross-Platform Integration DIN Deploy splits AI deployment into Python exporters, which download checkpoints from Hugging Face and convert them into ONNX format, and native C++ command-line interfaces built on ONNX Runtime (ORT). The implementation relies largely on standard ORT session and tensor APIs, isolating vendor-specific code such as CUDA kernels to optional accelerated paths. The framework supports Windows and Linux (including Arm64 variants) via CMake presets, and enables graphics interoperability using Vulkan and DirectX through ORT 1.25. Supported Tasks and Performance The sample library covers multiple AI domains: offline and streaming speech recognition through OpenAI Whisper, NVIDIA Parakeet TDT, and NVIDIA Nemotron ASR; interactive computer vision segmentation via Meta SAM 2.1; and text-to-image generation via FLUX.2-klein-4B. On DGX Spark benchmarks, GPU acceleration achieved 58.5x real-time speed on Whisper Large v3 Turbo (versus 3.8x on CPU) and 38.3 FPS on SAM 2.1 Hiera Base Plus (versus 0.5 FPS on CPU). The FLUX.2 sample also demonstrates drop-in post-training quantization via NVIDIA Model Optimizer without requiring application code changes.

英伟达推出开源C++示例库DIN Deploy，加速本地AI应用落地 英伟达发布开源C++示例项目DIN Deploy，结合ONNX Runtime与TensorRT RTX执行提供商，帮助开发者在Windows和Linux系统上快速将模型检查点部署为硬件加速的本地原生应用。 DIN Deploy采用Python导出ONNX与C++原生CLI分离的架构，绝大多数代码使用通用的ONNX Runtime API。 支持语音识别（Whisper、Nemotron、Parakeet）、图像分割（SAM 2.1）和图像生成（FLUX.2）等多种AI任务。 在DGX Spark测试中，GPU加速显著优于CPU，如Parakeet-tdt达到206.41倍实时速度，SAM 2.1达38.3 FPS。 提供跨平台CMake预设，支持x86-64及Arm64，并集成了Vulkan与DirectX图形交互能力及量化支持。 从模型导出到C++部署架构 DIN Deploy通过Python脚本将Hugging Face模型检查点导出为ONNX格式，应用端则采用基于ONNX Runtime的C++原生CLI实现，将转换与部署逻辑解耦。大部分示例代码直接调用ONNX Runtime会话与张量API，CUDA等特定供应商接口仅用于可选加速路径。此外，FLUX.2示例还结合了ONNX Runtime 1.25引入的图形交互能力，配合Vulkan和DirectX完成采样前后处理。 支持任务与性能表现 该项目涵盖自动语音识别（离线Whisper与流式Nemotron、Parakeet）、Meta SAM 2.1交互式图像与视频分割，以及FLUX.2-klein-4B文本生成图像。在DGX Spark硬件测试中，GPU相比CPU提升巨大：Whisper Large V3 Turbo达到58.5倍实时（CPU为3.8倍），Parakeet-tdt达到206.41倍实时（CPU为14.44倍），SAM 2.1帧率达38.3 FPS（CPU仅0.5 FPS）。FLUX.2示例还展示了通过NVIDIA Model Optimizer进行量化且无需修改业务代码的无缝替换能力。 平台适配与上手指南 仓库提供了面向Windows和Linux（包括Arm64）的CMake预设，CMake构建过程中默认会自动下载ONNX Runtime和TensorRT RTX。开发者在配置构建后，即可导出模型并运行CLI，或直接将推理管线代码集成到现有的原生软件中。

Adding AI models to local applications requires a portable model format, a reliable runtime, and acceleration that works across target systems. [Do Inference Now (DIN) Deploy](https://github.com/NVIDIA/din-deploy) is an open-source collection of practical C++ samples that bridges that gap. It combines ONNX Runtime with the NVIDIA TensorRT RTX execution provider to help developers move from a model checkpoint to a native, hardware-accelerated application on Windows and Linux. The same ONNX Runtime API can also be accessed through [WinML 2.0](https://github.com/microsoft/WindowsAppSDK-Samples/tree/main/Samples/WindowsML/cpp-cmake/WinMLEpCatalog). ## From ONNX export to C++ implementation Each DIN Deploy sample starts with a Python exporter that downloads a model checkpoint from Hugging Face and converts it into an ONNX artifact. The application side is a native C++ CLI built on ONNX Runtime (ORT). That split keeps model conversion separate from deployment logic, and developers can take an exported model into a local application without requiring a model-specific runtime. Most sample code uses ONNX Runtime session and tensor APIs in C++. Vendor-specific code, including CUDA APIs and kernels, appears only in optional accelerated paths. Execution providers that support the required ONNX Runtime tensor APIs can run the shared code. ORT’s [copy tensor API](https://onnxruntime.ai/docs/api/c/struct_ort_1_1_env.html#a61ca063195ceb79fe2daed1def02fe6e) keeps data locality manageable without dedicated vendor API usage in shared code. For preprocessing and postprocessing around exported model inference, the FLUX.2 sample uses ONNX Runtime’s [graphics interop capability](https://onnxruntime.ai/docs/api/c/struct_ort_graphics_interop_config.html), introduced in version 1.25, with Vulkan and DirectX for sampling. The repository provides CMake presets for Windows and Linux, including Arm64 variants. DirectX is available only on Windows. ## AI tasks supported by DIN Deploy For automatic speech recognition (ASR), DIN Deploy supports offline and streaming pipelines. OpenAI Whisper covers offline transcription across model sizes, while NVIDIA Parakeet TDT and NVIDIA Nemotron ASR Streaming provide streaming pipelines. The samples show how to move audio through a native application and return transcription results while using GPU acceleration where it is available. Meta SAM 2.1 samples support interactive masking for images and video. They turn model outputs into segmentation masks that native applications can use for selection, tracking, and other computer-vision workflows. Table 1 compares GPU and CPU performance for selected DIN Deploy workloads measured on DGX Spark. | **Model** | **GPU (DGX Spark)** | **CPU (DGX Spark)** | |---|---|---| | `openai/whisper-large-v3-turbo` | 58.5x | 3.8x | | `nvidia/nemotron-3.5-asr-streaming-0.6b` | 39.01x | 3.24x | | `nvidia/parakeet-tdt-0.6b-v3` | 206.41x | 14.44x | | `facebook/sam2.1-hiera-base-plus` | 38.3 FPS | 0.5 FPS | *Table 1. DGX Spark GPU and CPU performance for selected DIN Deploy workloads. Results for audio are reported as multiples of real time (higher is faster)* The FLUX.2-klein-4B sample provides prompt-driven image generation. The sample includes graphics-API interops with Vulkan and DirectX, allowing applications to integrate GPU-resident resources with a cross-vendor shader interface. It also shows how post-training quantization (PTQ) with NVIDIA Model Optimizer produces a quantized ONNX model. Quantization is hardware-dependent, but due to ONNX interfaces remaining unchanged, the quantized model is a drop-in replacement requiring no application-code changes. ## Get started with DIN Deploy Start with the repository’s CMake presets for Windows, Linux, x86-64, and Arm64. After configuring and building the project, export a model to ONNX and run the CLI with TensorRT RTX, or copy the code into your own application and use the pipeline implementations. CMake downloads ONNX Runtime and TensorRT RTX by default. Learn more about [TensorRT for RTX](https://developer.nvidia.com/tensorrt-rtx), [NVIDIA Local AI](https://developer.nvidia.com/local-ai), and the [DIN Deploy repository](https://github.com/NVIDIA/din-deploy), and the [NVIDIA blog series on model quantization](https://developer.nvidia.com/blog/model-quantization-concepts-methods-and-why-it-matters/).