Deploy Open Models to Native C++ Inference with NVIDIA TensorRT Model Connect

Deploy Open Models to Native C++ Inference with NVIDIA TensorRT Model Connect NVIDIA TensorRT Model Connect is an open collection of reference implementations designed to streamline bringing open AI models from Hugging Face checkpoints into native C++ applications using NVIDIA TensorRT. Deployment is split into two phases: packaging a model bundle with a Python CLI command and running native inference in C++ without requiring Python or PyTorch at runtime. The framework provides a task-level semantic API for standard inputs and outputs alongside a module-level API for direct tensor and pipeline control. Developers can integrate custom GPU kernels into targeted model components using TVM FFI while TensorRT runs the remainder of the pipeline. Model Connect is built as an AI-native project using coding agents under human review, deploying nightly releases to track new open model architectures quickly. Two-Phase Native C++ Deployment Model Connect eliminates the need to rebuild preprocessing, engine construction, and runtime orchestration for each model family. Developers use a Python CLI command to create an artifact bundle from a Hugging Face ID or local checkpoint, which a native C++ application subsequently loads. The production runtime executes natively without PyTorch or a Python interpreter, offering performance that can surpass torch.compile on supported workloads. Dual APIs and Kernel Customization The framework provides a semantic C++ API to handle tasks using prompts, audio, or images, as well as a module-level API for granular control over named tensors and TensorRT components. To accommodate specialized or newly released operators, TensorRT Model Connect uses TVM FFI, allowing developers to substitute specific model segments with custom GPU kernels without altering the surrounding inference pipeline. AI-Native Development and Release Model To keep up with rapid changes across open model architectures, Model Connect operates as an AI-native project where coding agents generate code, tests, and documentation under human direction and review. Supported by automated validation release gates, the project maintains nightly releases to shorten the turnaround time between upstream model releases and working TensorRT reference implementations.

NVIDIA推出TensorRT Model Connect：两行命令实现开源模型到原生C++推理部署 NVIDIA发布了TensorRT Model Connect参考实现集合，旨在简化开源AI模型在原生C++应用中的部署流程，使用户仅需两步即可将Hugging Face模型或本地权重转换为高性能TensorRT引擎并运行原生推理，且生产运行环境无需依赖Python。 部署仅需两阶段：通过Python CLI工具构建包含引擎与资产的bundle文件，随后在原生C++中使用简洁代码加载运行。 提供语义级（Semantic）和模块级（Module-level）两种C++ API，兼顾高层任务输入和底层命名张量精细控制。 支持通过TVM FFI集成自定义GPU内核，允许在保持TensorRT整体管线运行的同时无缝替换特定模型部分。 采用AI原生方式构建与维护，通过代码智能体结合人工审查加速模型覆盖，并通过自动化验证保障每夜构建质量。 两阶段部署与无Python生产运行时 Model Connect将部署流程解耦为构建与运行两个阶段。第一阶段使用Python CLI命令（如trtmc build）将Hugging Face模型ID或本地权重打包成包含TensorRT引擎与模型运行时资产的bundle；第二阶段在C++原生应用中通过pipeline加载该bundle并直接生成结果。该方案自动处理权重映射、引擎构建及预处理与后处理，应用可在无需PyTorch或Python解释器的环境中原生执行。 双层API设计与自定义内核扩展 项目提供两套C++ API以适配不同深度需求：语义级API直接面向文本、图像和音频等任务级输入输出，由框架处理编排；模块级API则允许开发者直接操作命名张量和特定组件以自定义管线。此外，借助TVM FFI的跨语言接口能力，开发者能够将模型特定部分替换为自定义GPU内核，而无需为新内核重构整个应用的运行时。 AI原生开发与性能表现 为跟进开源模型生态的快速迭代，Model Connect采用AI原生模式开发，由代码智能体在人类指导与审查下并行生成实现代码、测试与文档，并借助自动化验证门禁实现每夜构建发布。在经支持与验证的工作负载中，其推理速度可超越torch.compile，在降低部署门槛的同时保障了TensorRT的高性能输出。

Open AI models are evolving faster than ever, but bringing them into native applications can still require model-specific conversion, preprocessing, post-processing, and runtime code. NVIDIA TensorRT Model Connect open collection of reference implementations helps to address this challenge. TensorRT Model Connect shows you how to run supported models with NVIDIA TensorRT in native C++ applications. You can use, inspect, modify, and extend the implementations. Model Connect is designed to support the open model ecosystem wherever TensorRT runs. This post explains what NVIDIA TensorRT Model Connect is and how to deploy a model from Hugging Face model ID to native C++ inference in two commands. It also covers the two API levels TensorRT Model Connect provides, how to integrate custom GPU kernels, and how the project is built to keep pace with the open model ecosystem. Figure 1. Out-of-framework deployment bottleneck Figure 2. NVIDIA TensorRT Model Connect stack How to deploy a model from model ID to native C++ inference in two commands Getting a model into production should not require deep compiler expertise. Model Connect splits deployment into two phases with a single artifact between them. 1. Build the bundle (Python CLI) For a supported model, the first phase is building a deployment bundle from a Hugging Face model ID or local checkpoint: trtmc build Qwen/Qwen3-0.6B -o qwen3-0.6B.bundle The bundle contains the TensorRT engines and the model-specific assets needed at runtime. 2. Load and run (C++) In the second phase, a native C++ application then loads the bundle and works with task-level inputs and outputs: #include <trtmc/pipeline.h> auto pipeline = trtmc::load("qwen3-0.6b.bundle"); auto result = pipeline->generate("Explain why native inference matters.", {.max_new_tokens = 20}); std::cout << result.text << std::endl; Model Connect handles checkpoint mapping, TensorRT engine construction, preprocessing, runtime orchestration, and post-processing. You start with a complete working implementation instead of rebuilding this integration for every model family. You can use Python to prepare the model, but the deployed application runs natively without requiring PyTorch or a Python interpreter in its production runtime. Two API levels, one starting point Model Connect provides two levels of C++ APIs. With the semantic API, you can work with familiar inputs and outputs, such as prompts, images, and audio, while Model Connect handles model-specific preprocessing, execution, and post-processing. If you need more control, the module-level API allows you to work directly with named tensors and individual TensorRT components to customize the inference pipeline. Both APIs use the same Model Connect implementations, so you can start with a simple task-level interface and customize the pipeline only when needed. Extend TensorRT Model Connect with custom kernels TVM FFI provides a language-agnostic interface for invoking GPU kernels without tightly coupling the calling system to the kernel’s implementation framework or runtime. Using TVM FFI through TensorRT Model Connect, you can replace a targeted portion of a model with a custom GPU kernel while TensorRT continues to execute the rest of the inference pipeline. This makes it easier to integrate specialized or newly developed kernels without rebuilding the application around a separate runtime. See the Bring Your Own Kernel tutorial for a worked example. Reference implementations for the open model ecosystem Model Connect is not a new inference framework or a replacement for TensorRT. It is a bridge between the end-to-end inference experience for open models and the ability of TensorRT to translate a computation graph into an accelerated engine on GPU. Each model’s implementation serves three purposes: Running a supported open model in a native TensorRT-enabled application Learning from a complete, inspectable implementation of the model and its inference pipeline Extending the implementation for a related architecture, custom checkpoint, or application requirement This provides the broader ecosystem with a clearer path to TensorRT deployment from a model ID. Application developers can begin with working code. Community contributors can reuse existing patterns to add support for new models instead of starting from zero. The goal is straightforward: wherever TensorRT is available, you should have a consistent Model Connect path for supported open models. Built AI-natively to keep pace with open models The open model ecosystem changes quickly. New architectures and checkpoints appear continuously, so a reference library must evolve just as quickly. Model Connect is built as an AI-native software project . Coding agents generate implementation code, tests, integrations, and documentation under human direction and review. This enables the project to develop and validate multiple model implementations in parallel while maintaining a consistent architecture and user experience. Model Connect uses nightly releases to shorten the path from a new model, user report, or contribution to an available implementation. Automated validation remains the release gate.  The faster cadence helps new model support, fixes, and UX improvements reach you sooner. Delivering the complete TensorRT workflow Model Connect is built on TensorRT, so performance remains central. For supported and validated workloads, Model Connect can deliver faster inference than torch.compile, and each implementation is continuously tested and optimized as the project evolves. Performance should not come at the expense of usability. Model Connect brings the complete workflow together: find the model ID, build the model, load it from C++, and adapt it when needed. You get an accessible path to high-performance TensorRT inference while retaining the ability to inspect, customize, and optimize the underlying inference pipeline. Get started with NVIDIA TensorRT Model Connect Visit the NVIDIA/TensorRT-Model-Connect GitHub repo to find supported implementations and build a model bundle. Use an implementation as-is, adapt it for your application, or contribute support that helps the next developer bring another open model to TensorRT. Want to use an AI-native quick start that doesn’t require a complicated setup? Open a terminal in any folder you can access, then paste the following prompt into a coding agent. You should have a complete deployment in minutes. /goal Clone https://github.com/NVIDIA/TensorRT-Model-Connect.git into a new TensorRT-Model-Connect directory in the current workspace. Detect the current GPU compute capability, modify the repository development Docker image, build and start the container, install TensorRT-Model-Connect, compile the CLI, TensorRT backend, and all native model DSOs only for that SM, then build and run an end-to-end Qwen/Qwen3-0.6B smoke test. Do not commit or push changes. Report the result of the test, show exact command, input and output of the inference run. For more about model coverage, architecture details, and a full developer guide, see the TensorRT Model Connect documentation .