NVIDIA Dynamo-Triton Accelerates HSTU Generative Recommender Inference with PyTorch AOTI and FlexKV

NVIDIA Dynamo-Triton Accelerates HSTU Generative Recommender Inference with PyTorch AOTI and FlexKV NVIDIA has introduced an end-to-end inference workflow in its recsys-examples repository for serving Hierarchical Sequential Transduction Unit (HSTU) generative recommenders. The pipeline integrates PyTorch Ahead-of-Time Inductor (AOTI) compilation, FlexKV-backed KV caching, and Dynamo-Triton to significantly cut latency when evaluating long-sequence user histories. Generative recommenders model user context, historical items, actions, and candidates as sequential tokens rather than using disjoint retrieval and ranking pipelines. The workflow pairs PyTorch AOTI compilation with NV Embedding Cache and FlexKV-backed GPU/host KV caching to minimize redundant attention calculations and manage large embedding tables. On an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU at batch size 8, KV caching with PyTorch AOTI delivered speedups of up to 4.47x for a three-layer model and 5.93x for an eight-layer model over uncached AOTI. A five-stage deployment pipeline validates compiled C++ artifacts before serving them via Dynamo-Triton's torch_aoti backend. Generative Recommendation and Serving Bottlenecks Generative recommender systems reframe personalized recommendation as sequence modeling over event streams composed of user context, historical item and action tokens, and candidate items. While this captures recency and complex interaction patterns, inference becomes computationally prohibitive when evaluating long sequences repeatedly. Because user histories remain mostly static between requests, recalculating key-value states for every interaction causes latency bottlenecks. AOTI Compilation and Multi-Tiered Memory Architecture The serving pipeline addresses computational overhead using PyTorch Ahead-of-Time Inductor (AOTI), which exports models into native C++ packages to bypass Python runtime overhead. To handle memory footprints, the NV Embedding Cache retains frequent embeddings in GPU memory while holding full tables in CPU memory. A KVCacheManager utilizes paged GPU memory and FlexKV-backed host storage with LRU-style eviction, allowing HSTU attention kernels to reuse historical key-value states and process only new tokens. Deployment Pipeline and Latency Benchmarks Production serving is structured into five stages covering custom operator builds, AOTI export, FlexKV service initialization, C++ replay validation, and Dynamo-Triton deployment under the torch_aoti platform. Single-GPU benchmarks using the KuaiRand-1K configuration on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU showed that at dynamic batch size 8 with full GPU KV-cache hits, per-request latency dropped to 0.423 ms for a three-layer model and 0.678 ms for an eight-layer model, yielding speedups up to 5.93x over uncached execution.

NVIDIA 推出基于 Dynamo-Triton 与 PyTorch AOTI 的 HSTU 生成式推荐部署方案 NVIDIA 在 recsys-examples 代码库中发布了针对分层序列转导单元（HSTU）生成式推荐模型的端到端推理工作流。该方案结合了 PyTorch AOTI 预编译、基于 FlexKV 的 KV 缓存以及 NV 嵌入缓存，在 NVIDIA RTX PRO 6000 Blackwell 显卡上实现了高达 5.93 倍的推理延迟加速。 结合 PyTorch AOTI 与 GPU KV 缓存，在 100% 缓存命中率及动态批处理大小为 8 时，3 层 HSTU 模型加速达 4.47 倍，8 层模型加速达 5.93 倍。 PyTorch AOTI 将模型提前编译为可供 C++ 原生运行时加载的制品，消除 Python 运行时开销，并通过 Dynamo-Triton 部署。 FlexKV 与 KVCacheManager 实现了针对用户长历史序列的分页缓存、LRU 逐出与多级存储，避免重复计算注意力状态。 NV 嵌入缓存将高频嵌入存放在 GPU 显存、完整表格保留在 CPU 内存中，有效降低了超大嵌入表的显存占用。 HSTU 架构与生成式推荐的推理瓶颈 生成式推荐（GR）将推荐任务重构为针对用户行为事件流的序列建模，统一处理用户上下文、物品及交互行为 token。尽管其在建模长期历史与非平稳数据方面表现优异，但在生产环境中直接提供长序列与大嵌入表的在线推理会导致延迟过高，尤其是在面对重复访问的用户时存在严重的重复计算问题。 结合 AOTI、KV 缓存与显存优化的技术栈 该推理方案通过 PyTorch AOTI 将模型编译为打包好的 .pt2 制品，供 Dynamo-Triton 的 C++ 运行时直接调用；NV Embedding Cache 仅在 GPU 显存中保存热点数据以节省显存。同时，KVCacheManager 和 FlexKV 提供分页 GPU/主机双层 KV 缓存机制，在用户历史相对稳定时仅计算新 token，大幅减轻了序列处理计算量。 KuaiRand-1K 基准测试表现 在 NVIDIA RTX PRO 6000 Blackwell 工作站显卡上的测试显示，在序列长度达 8,320 token 的配置下，PyTorch AOTI 相比 Python 后端展现出明显优势。在逻辑批处理大小为 8 且 GPU KV 缓存完全命中的情况下，3 层 HSTU 模型的单请求延迟低至 0.423 毫秒，8 层模型达到 0.678 毫秒，有效化解了深度长序列推荐系统的延迟痛点。

Generative recommender (GR) systems are emerging as a powerful new approach for large-scale personalization. Instead of treating recommendation as a set of isolated retrieval, ranking, and prediction stages, GRs reformulate recommendation as sequence modeling over user behavior. A user’s interactions, context, candidate items, and actions become tokens in a high-cardinality event stream, and the model learns to generate or score the next relevant items from that sequence. This approach is especially attractive for modern recommendation workloads, where user histories can be long, item catalogs are constantly changing, and personalization quality depends on modeling rich sequential behavior. But it also introduces a serving challenge: GR models need low-latency inference despite long histories, large embedding tables, and sequence-heavy model architectures. NVIDIA Dynamo-Triton (formerly NVIDIA Triton Inference Server) now supports an end-to-end Hierarchical Sequential Transduction Unit (HSTU) GR inference workflow through the NVIDIA recsys-examples repository. The workflow combines HTSUs , PyTorch Ahead-of-Time Inductor compilation , FlexKV-backed KV caching , native C++ validation , NV embedding cache , and Dynamo-Triton deployment . The result is a practical path for serving HSTU ranking models with strong latency performance. At dynamic batch size 8 on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU, Dynamo-Triton with PyTorch AOTI achieved best-case speedups of up to 4.47x for the three-layer HSTU model and 5.93x for the eight-layer model under a 100% GPU KV-cache hit rate, relative to the same AOTI configuration without KV caching. This post shows you how to move an HSTU generative recommender from PyTorch development to production inference with NVIDIA Dynamo-Triton, PyTorch AOTI, and FlexKV. You will learn how to export and ahead-of-time compile the model, validate the resulting deployment artifact in Python and native C++, and serve it through Dynamo-Triton without rewriting the model for a separate runtime. It also examines how GPU-backed KV caching reduces repeated computation and presents benchmarks demonstrating up to 5.93x lower latency, highlighting the practical performance benefits of this deployment workflow. Why use HSTUs for generative recommendation? HSTUs were introduced for GR workloads that operate over high-cardinality, nonstationary event streams. In a traditional recommender system, retrieval and ranking are often built from a collection of specialized models and feature pipelines. GRs instead model recommendation as a sequential prediction problem, allowing the model to reason over user context, item history, action history, and candidate items in one sequence-aware architecture. In the NVIDIA HSTU ranking example, the model input is built from categorical tokens. Contextual tokens represent user-side information, item tokens represent items, and optional action tokens represent user interactions with those items. Figure 1. Traditional deep learning recommendation model (left) versus generative HSTU workflow (right) The HSTU preprocessing path retrieves embeddings, interleaves item and action embeddings when action tokens are present, appends contextual information, and applies positional encoding. HSTU blocks then process the sequence, and a prediction head produces multitask ranking outputs. This structure is well-suited for recommendation systems where recency, order, and repeated interaction patterns matter. However, it also means inference can become expensive when each request repeatedly processes long historical sequences. Production systems need to preserve HSTU modeling benefits while reducing redundant computation during serving. Why is serving large sequential recommenders challenging? Serving large sequential recommenders is different from serving a small dense ranking model. The serving stack must handle jagged sequence inputs, large categorical embedding state, long histories, and request patterns where the same user may return repeatedly with only a small amount of new information. Recomputing the full key-value state for a user history on every request wastes work and increases latency. This is where KV caching becomes important. A KV cache stores reusable key-value data from prior sequence computation, allowing the model to avoid recomputing cached portions of the user history. For recommender inference, this is particularly useful when a user’s long-term history remains mostly stable while new candidate items or recent actions arrive. Figure 2. HSTU serving The NVIDIA HSTU inference workflow includes a KVCacheManager that uses GPU memory and host storage for KV data caches. The GPU cache is organized as a paged KV data table and supports lookup, allocation, append, and eviction. When GPU cache space is constrained, older users can be evicted according to an LRU-style policy. Host-side storage provides another tier for cached KV data, and the workflow includes a FlexKV-backed backend for the KV-cache runtime. The HSTU attention kernel can consume KV data from the paged cache, and the exported inference path includes cache-aware custom operations for lookup, allocation, onboarding, appending, and offloading. This allows the serving path to preserve the model’s sequence semantics while reducing redundant computation. PyTorch AOTI for native inference The Pytorch AOTI (Ahead-of-Time Inductor) workflow starts from a PyTorch model and exports it using torch.export and PyTorch AOTI. AOTI compiles the model ahead of time into a package that can be loaded by a native C++ runtime. This reduces Python runtime overhead and provides a deployment-friendly artifact for the Dynamo-Triton PyTorch AOTI backend. Figure 3. PyTorch AOTI workflow The exported model package contains the AOTI model archive plus metadata and embedding table files. In the NVIDIA example, the embedding implementation combines DynamicEmb inference embedding tables and NV Embedding Cache that reduces GPU memory usage by storing only the popular embeddings in GPU memory while keeping the entire table in CPU memory. The export path writes layer metadata and embedding table data alongside the compiled .pt2 archive so the model can be loaded without unnecessary duplicate embedding table copies. The workflow validates the same exported artifacts in multiple ways. Python export scripts generate the package and replay tensors. Native C++ executables load and replay the exported model for correctness and performance validation. The Dynamo-Triton deployment then uses the same AOTI package and replay path, which helps keep development validation and production serving aligned. Dynamo-Triton deployment path Dynamo-Triton provides the production serving layer for the exported HSTU model. The AOTI deployment uses the Dynamo-Triton PyTorch backend with platform: "torch_aoti" . This allows Dynamo-Triton to load and serve the ahead-of-time compiled PyTorch model package. The full workflow includes the following five stages: Build the required custom operators and runtime libraries Export the HSTU ranking model with PyTorch AOTI Start the FlexKV-backed KV-cache service Validate the exported artifacts with native C++ replay Serve the exported KV-cache AOTI model with Dynamo-Triton This approach is important because recommender serving requires more than just a fast model kernel. Dynamo-Triton brings model repository management, request handling, backend integration, metrics, and deployment structure. AOTI brings a lower-overhead compiled model artifact. NV Embedding Cache lowers GPU memory requirements by keeping only the hot parts of embedding tables in GPU memory. FlexKV reduces recomputation for long user histories by caching attention blocks. Together these form a serving stack aimed at realistic generative recommender inference. Figure 4. HSTU GR inferencing stack with Dynamo-Triton Benchmarking HSTU serving latency This benchmark compares HSTU serving latency across Dynamo-Triton backends, model sizes, batch sizes, and KV-cache states. The goal is to quantify the performance benefits of the production HSTU serving stack. It compares the Dynamo-Triton PyTorch AOTI backend with the Python backend, measures the additional latency reduction from GPU KV caching, and evaluates how those benefits scale across model depths and batch sizes. Ultimately, it shows developers the performance they can expect when moving from uncached Python-based inference to compiled, cache-aware HSTU deployment with Dynamo-Triton. The benchmark results in recsys-examples use the KuaiRand-1K ranking configuration on a single GPU. The model structure includes three-layer and eight-layer HSTU variants, hidden size 512, four attention heads, BF16 model weights, BF16 KV cache, maximum history sequence length of 8,192 in total (4,096 item plus action pairs) history stream, maximum candidate sequence length of 100, and six contextual features. The effective sequence length before alignment is 8,298 tokens, exported with a maximum aligned sequence length of 8,320. The benchmark protocol reports latency per logical request. For the Dynamo-Triton AOTI benchmark, each Dynamo-Triton call contains one logical batch, and latency per logical request is calculated by dividing end-to-end pass time by the number of Dynamo-Triton calls times the logical batch size. Dataset loading, validation, rebatching, user-ID generation, server startup, warmup, and post-warmup sleep are excluded from the measured time. The hardware used for the Dynamo-Triton backend comparison and batch-size results is an NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPU. Dynamo-Triton backend comparison At Dynamo-Triton batch size 2, PyTorch AOTI improves latency over the Dynamo-Triton Python backend even without KV-cache hits. With hits from the GPU KV cache of 20 GB, the latency improvement is significantly larger. Figure 5. Dynamo-Triton backend comparisons for three-layer and eight-layer HSTU These results show two different gains. First, AOTI reduces serving overhead compared with the Python backend. Second, KV cache hits reduce model work by reusing cached sequence state. The cache benefit is especially visible on the deeper eight-layer model, where avoiding recomputation has more impact. Batch-size scaling with AOTI and KV cache The PyTorch AOTI backend results by batch size show that KV caching becomes increasingly effective as logical batch size grows. Figure 6. Three-layer HSTU, with Dynamo-Triton batch-size scaling Figure 7. Eight-layer HSTU, with Dynamo-Triton batch-size scaling At batch size 8, the three-layer HSTU model reaches 0.423 ms latency per logical request with GPU KV-cache hits. The eight-layer HSTU model reaches 0.678 ms. These are strong results for long-sequence ranking inference and demonstrate the value of combining compiled model execution with cache-aware serving. What are the benefits of accelerating HSTU GR inference? Recommendation systems operate under tight latency budgets. Additional ranking latency can affect page-load times, feed responsiveness, and ad-serving deadlines. Meanwhile, increasingly sequence-aware and personalized models can require more inference compute, particularly as user histories grow. The HSTU serving workflow combines complementary technologies to address this challenge. HSTU provides the generative recommendation architecture, PyTorch AOTInductor produces an ahead-of-time compiled deployment artifact, FlexKV-backed KV caching enables reuse of previously computed attention state, and NVIDIA Dynamo-Triton provides the production serving environment. This approach is especially valuable when successive requests share an unchanged prefix of a user’s interaction history. Rather than recomputing attention over that portion of the sequence, the model can reuse its cached key-value state and compute only what is required for newly appended tokens. The potential savings increase with longer sequences and deeper models, where repeated computation across multiple HSTU layers would otherwise add significant latency. Get started accelerating HSTU GR inference You can reproduce and extend this workflow from the NVIDIA/ recsys-examples GitHub repo. The HSTU overview introduces the GR model structure, including contextual tokens, item tokens, action tokens, embedding tables, HSTU blocks, and prediction heads. The AOTI inference guide walks through building the required images and libraries, preparing KuaiRand-1K data, training a checkpoint, exporting the KV-cache AOTI model, validating it with C++ replay, packaging the Dynamo-Triton runtime image, and replaying requests through the Dynamo-Triton server. To learn more, check out these related resources: HSTU generative recommender overview Dynamo-Triton plus FlexKV plus PyTorch AOTI integration HSTU inference benchmarks Dynamo-Triton HSTU documentation NV embedding cache Acknowledgments This post is a cross-functional effort across several NVIDIA teams. We would like to thank J, Runchu Zhao, Yulu Liu, Lin Hu, Zhuofan Li, Jacob Subag, and Tomer Bar-On for their contributions.