NVIDIA Releases VSS Blueprint 3.3 with Automated Agent Composition and Adaptive Video Sampling

NVIDIA Releases VSS Blueprint 3.3 with Automated Agent Composition and Adaptive Video Sampling NVIDIA has released the Metropolis Blueprint for Video Search and Summarization (VSS) 3.3, adding features to reduce the development, change, and runtime operational costs associated with visual AI agents. The new Build Vision Agent skill allows developers to generate multi-workflow deployments from natural-language prompts by applying minimal deltas over four validated foundation profiles. Adaptive Efficient Video Sampling (EVS) drops unchanged visual tokens using cosine similarity before they reach the vision-language model, reducing input tokens by 80% on a 60-minute summary. Benchmarking on an NVIDIA RTX PRO 6000 Blackwell GPU showed Adaptive EVS increased concurrent real-time streams by 46% and lowered alert contextualization latency by 17%. Automated Agent Composition VSS 3.3 introduces the Build Vision Agent skill (vss-build-vision-ai), which enables coding agents to translate high-level natural language prompts into complete system deployments. The skill selects the closest starting profile from four validated foundation options—base, alerts, lvs, or search—and computes the minimal required delta rather than generating configurations from scratch. Shared components such as Kafka, Redis, Elasticsearch, and Video IO and Storage are unified across capabilities to prevent duplicated infrastructure. In a demonstration on a dual-GPU RTX PRO 6000 Blackwell host, the skill configured and deployed an orange juice bottling line monitoring agent in under 30 minutes. Runtime Optimization with Adaptive EVS To address the operational expense of vision-language models repeatedly processing static camera backgrounds, VSS 3.3 integrates Adaptive EVS into its real-time VLM microservice. The feature computes cosine similarity between frame patches and drops unchanged tokens before they reach the language model, while token retention levels dictate event-aware batching. Running Cosmos 3 Super FP8 on an NVIDIA RTX PRO 6000 Blackwell host, Adaptive EVS lowered alert contextualization latency from 1,021 ms to 844 ms, raised concurrent real-time VLM streams from 13 to 19, and cut input tokens for 60-minute summaries by 80%. Integration and Setup Developers can deploy VSS 3.3 by cloning the blueprint repository, symlinking skills into a compatible coding agent's directory, and providing natural-language instructions detailing camera sources and required workflows. The skill produces an architecture diagram and self-contained configuration files, such as override.env and resolved.yml, without altering the core repository tree. Adaptive EVS operates locally inside the real-time VLM container and can be enabled and calibrated through session, pruning rate, and similarity threshold parameters.

NVIDIA 发布 VSS Blueprint 3.3，通过智能体技能与自适应采样降低视觉 AI 成本 NVIDIA 推出面向视频搜索与摘要的 Metropolis Blueprint 3.3（VSS 3.3），通过引入新的 Build Vision Agent 技能与自适应高效视频采样（Adaptive EVS）技术，分别解决视觉 AI 智能体在开发集成复杂与运行时 VLM 显存/算力消耗高昂两大痛点。 Build Vision Agent 技能（vss-build-vision-ai）允许开发者通过自然语言 Prompt 在 30 分钟内生成并部署多工作流视觉 AI 智能体。 构建技能基于 4 个基础配置（base、alerts、lvs、search）计算最小增量并复用 Kafka、Elasticsearch 等共享基础设施，避免重复建设。 运行时新增 Adaptive EVS 技术，通过余弦相似度动态剪枝未变化的视频帧 Patch，并进行事件感知批处理以节约 Token。 在 RTX PRO 6000 Blackwell 测试中，Adaptive EVS 使 60 分钟视频摘要的 VLM 输入 Token 减少 80%，单 GPU 并发流提升 46%。 开发侧：通过自然语言技能自动编排工作流 生产级视觉 AI 系统通常需要整合告警、检索和摘要等多个微服务与共享组件。VSS 3.3 引入的 vss-build-vision-ai 技能可配合 Claude Code 等编码智能体，根据用户的自然语言描述自动映射工作流。该技能从 base、alerts、lvs 或 search 四种已验证的底层配置切入，仅计算最小增量并复用 Kafka、Elasticsearch 和 VIOS 等基础设施，最终生成独立的 Docker Compose 部署方案，降低架构构建与后期变更成本。 运行侧：Adaptive EVS 技术大幅降低 GPU 计算负载 针对监控场景中大量画面静止却持续消耗 VLM 算力的问题，VSS 3.3 集成了实时自适应高效视频采样（Adaptive EVS）。该功能通过逐 Patch 对比相邻帧的余弦相似度来裁剪冗余视觉 Token，并依据保留率进行事件批处理。测试显示，在搭载 Cosmos 3 Super FP8 的 RTX PRO 6000 Blackwell 上，该技术使告警情境化延迟由 1,021 毫秒降至 844 毫秒，并发视频流从 13 路增至 19 路，生成 60 分钟视频摘要耗时减半且 Token 消耗减少 80%。

Vision-language models have made it possible to build visual AI agents that understand video at production scale. The harder problem is turning that capability into a maintainable system that combines ingestion, stream processing, event detection, retrieval, summarization, and reporting. The NVIDIA Metropolis Blueprint for Video Search and Summarization (VSS) and its agent skills help developers build visual AI agents faster. VSS connects vision-language models (VLMs) such as NVIDIA Cosmos , LLMs such as NVIDIA Nemotron , retrieval-augmented generation (RAG), and Model Context Protocol (MCP) tools to turn live and recorded video into natural-language search, visual Q&A, verified alerts and automated reporting. VSS Blueprint 3.3 also reduces costs from both sides: faster application composition with the new Build Vision Agent skill ( vss-build-vision-ai ), and cheaper VLM processing at runtime with Adaptive Efficient Video Sampling (EVS). This post covers both sides of that cost reduction. On the development side, a single prompt builds and deploys a bottling-line overflow agent in under 30 minutes, with a few dollars of coding-agent usage. On the runtime side, Adaptive EVS delivers 80% fewer VLM input tokens for a 60-minute summary and 46% more concurrent streams on the same GPU. To learn more, join us live on Oct. 1 at 9 a.m. PT, where we will build a visual AI agent from a single prompt. Video 1: One prompt builds and deploys a visual AI agent for an orange juice bottling line, then the agent raises and verifies an overflow alert. The spill is synthetically generated for the demo Why visual AI agents are expensive A production visual AI agent usually spans more than one workflow. A smart city application may need vehicle detection, collision alerting, searchable incident clips, hourly summaries, and an operator report. A warehouse application may need people and forklift tracking, near-miss alerts, SOP checks, and follow-up Q&A. Each workflow is useful by itself, but real deployments become valuable when those workflows work together. That creates 3 recurring cost drivers: Development cost: Teams must choose and connect microservices, shared services such as Kafka, Redis, Elasticsearch, and Video IO and Storage (VIOS), model endpoints, environment variables, and APIs without duplicating infrastructure. Operating cost: Visual AI workloads can lead to heavy token usage, every additional stream, frame window, prompt, and visual token can increase GPU usage, queueing delay, and end-to-end summarization latency. Change cost: Teams must move from proof of concept to production, add capabilities, and keep configuration, documentation, and operations aligned. VSS 3.3 updates for building and running video analytics AI agents VSS Agent Skills let coding agents such as Claude Code, Codex, or any agentskills.io-compatible agent deploy and operate VSS from natural-language requests. VSS 3.3 adds two updates that reduce cost on both sides of a deployment: Build Vision Agent skill (vss-build-vision-ai): Combines VSS workflows such as alerting, search, and summarization into one application for use cases such as SOP compliance or traffic management, and extends a running deployment without rebuilding the whole stack. Adaptive EVS: This new feature reduces redundant VLM processing by pruning the visual tokens for parts of a frame that did not change compared to the previous one, and by batching VLM work around the moments when something happens. EVS already ships in vLLM and the Cosmos NIM microservices at a fixed pruning rate. The adaptive version in VSS 3.3 is integrated into the real-time VLM microservice and decides which tokens to keep per patch and per frame. Figure 1. One prompt becomes a composed, validated, running visual AI agent, here for a warehouse, then answers questions about its video The Build Vision Agent skill reduces development and change costs by composing and extending deployments. Adaptive EVS lowers operating cost by reducing VLM tokens and GPU time spent on unchanged video. Reduce development cost with the Build Vision Agent skill Earlier VSS skills handled individual operations such as deployment, camera setup, summarization, search, alerts, and analytics. VSS 3.3 organizes them as deployment skills, operation skills, tools, and benchmarks, with vss-build-vision-ai composing the rest. Developers describe the application they want, and the Build Vision Agent skill translates that intent into a deployment plan spanning profiles, microservices, configuration, and runtime operations. Rather than generate a deployment from scratch, the skill starts with the closest of four validated developer profiles, each a complete, tested stack for one workflow (see Table 1, below). The skill calls that starting profile the Foundation and changes only what the request requires. Profile Capability base VLM dense captioning and Q&A on clips alerts Real-time VLM alerting, or RT-CV detection with behavior analytics and VLM alert verification lvs Long video summarization search Object and video embeddings with agentic search Table 1: The four developer profiles a build can start from. The skill selects one as the Foundation and computes the smallest delta on top of it The skill then computes the smallest delta: adds or removes only exact service keys, keeps only the services that a requested capability actually reaches, and converges shared roles onto one instance. Two capabilities that both need a detector get one detector. Two that both need Kafka and Elasticsearch share one message bus and one Elasticsearch deployment, each writing its own indices. When the rules cannot settle a choice, the skill asks one structured question instead of guessing. Figure 2: Every build is a delta on a validated Foundation: add only what a capability needs, prune what nothing reaches, and converge shared roles to one instance What the skill automates Maps the application goal to the required VSS workflows and microservices. Combines workflows such as alerting, search, and summarization in one deployment plan. Reuses shared infrastructure, including VIOS, Kafka, Redis, Elasticsearch, HAProxy ingress, and MCP services. Generates the deployment as a self-contained build: _builds/<name>/override.env (the Foundation, the effective Compose profiles, and only the settings you changed), compose.yml, and resolved.yml, one flattened Compose file produced with docker compose config that deploys on its own. The repository’s deploy/docker/ tree is never modified. Shows an architecture diagram for review before anything is written or deployed, then runs validation, deployment, and readiness checks so the developer can verify the stack before building application logic on top. Asks whether to deploy an agent harness. The default is NemoClaw , a host-side sandbox with the VSS skills installed; answering no produces a headless stack driven by the VSS CLI. Extends a running deployment through a smaller delta that reuses existing services. This shortens discovery, makes composition repeatable, and avoids duplicate ingestion, storage, messaging, and analytics infrastructure across workflows. Build a bottling line visual AI agent with VSS For an orange juice bottling line, the team wants an agent that watches filler and capper cameras, alerts on overflows or spills, searches past incidents, and generates shift reports. Manual development would connect event detection, alert verification, storage, search ingestion, summarization, and reporting. With the Build Vision Agent skill, development begins with the desired outcome. Sample prompt Build a VSS vision agent for an orange juice bottling line. Use two RTSP cameras on the filler and capper. Detect bottle overflows and juice spills, verify each alert with the VLM, make alert clips searchable, and generate a shift report for the line supervisor. What the agent assembles VIOS-backed ingestion for RTSP cameras and recorded clips. Real-time detection, tracking, captioning, or VLM-based alerts, depending on the Foundation profile. Behavior analytics or rules for overflow, spill, and line-stoppage events. VLM verification that confirms alerts and explains its reasoning. Natural-language search across validated clips and indexed video. Summarization and reporting for operator handoff and incident review. Shared messaging, storage, APIs, and observability. On a two-GPU RTX PRO 6000 Blackwell host, the skill combines search and alerts by reusing existing services and adding only an alert bridge and real-time VLM. FP8 Cosmos 3 Nano shares the detector’s GPU, avoiding duplication. A recorded alerts build reached a live, previewable deployment in under 30 minutes. The result is a reusable pattern: ingest video once, share evidence across workflows, and give operators natural-language search, alerts, summaries, and reports. How adaptive EVS reduces operating cost Once the bottling line agent from the example above is running, its cameras keep producing video around the clock, and most of each frame never changes: the filler, the guards, the floor. Only the bottles moving through it do, and an overflow is rare. Every frame window still becomes visual context for the VLM, so the model spends most of its compute re-reading regions that look exactly like the frame before. That VLM processing is one of the main runtime cost drivers in a deployed visual AI agent, and it is the cost Adaptive EVS goes after. Figure 3: Four scenes where most frames repeat the one before. Adaptive EVS spends VLM tokens on the events, not the waiting What Adaptive EVS changes in the pipeline Dynamic pruning. Each patch is compared with the prior frame using cosine similarity; unchanged patches are dropped before reaching the language model. Event-aware batching. Token retention indicates activity: clips above about 70% are batched as events, those below about 30% are dropped or flushed, and the rest run normally. Figure 4: Most frames repeat the one before them. Adaptive EVS keeps only the patches that changed, drops the quiet frames, keeps the event, and the VLM’s answer stays the same on a fraction of the tokens Performance impact On an NVIDIA RTX PRO 6000 Blackwell running Cosmos 3 Super FP8, Adaptive EVS: Cut alert contextualization latency 17%, from 1,021 ms to 844 ms, while similarly reducing token usage. Increased concurrent real-time VLM streams 46%, from 13 to 19. Summarized a 60-minute video in about half the time with 80% fewer VLM input tokens. Results vary with scene motion, chunk length, and similarity threshold; benchmark representative footage before choosing production defaults. Adaptive EVS is most useful when VLMs read many frames and produce short responses, as in dense captioning, long-video summarization, and alert verification. It offers less benefit for long outputs from few frames, runs inside the RT-VLM container rather than against remote endpoints, and is optional. Enable it in override.env, then benchmark accuracy, throughput, and latency on representative footage: VIA_EVS_SESSION=true VLM_VIDEO_PRUNING_RATE=0.5 # 0.0 to 1.0; higher prunes more VLLM_EVS_SIMILARITY_THRESHOLD=0.2 Cost impact: What changes for teams Lower integration effort through natural-language composition of multi-workflow applications. Less duplicate infrastructure across alerting, search, summarization, reporting, and Q&A. Higher GPU efficiency by pruning unchanged visual regions. Lower summarization and incident-review latency by skipping uneventful video. Easier extension through incremental deltas that reuse a running deployment. Together, these changes reduce upfront development work and the GPU work required to process video with VLMs across environments and applications. Getting started With VSS 3.3 Clone the VSS Blueprint repository and check out the branch containing the 3.3 skills. Install the VSS skills in your coding agent’s standard skills directory. Describe the desired agent, including video sources, workflows, and deployment constraints, or say “build a vision agent” for guidance. Review the architecture diagram and _builds/<name>/override.env, especially GPU placement, model endpoints, ports, storage, and security boundaries. For RT-VLM workloads, enable Adaptive EVS, tune the pruning rate, and benchmark accuracy, throughput, and latency on representative video. Deploy on a trusted, isolated network with authentication, TLS, rate limiting, and external controls. Example setup commands git clone https://github.com/NVIDIA-AI-Blueprints/video-search-and-summarization.git cd video-search-and-summarization Ask your coding agent to install the skills: Read skills/README.md and every SKILL.md under skills/. Install each skill for this host using the standard skills directory, symlinking rather than copying so a git pull keeps them current. After the skills are installed, you can start with a prompt such as: Build a VSS vision agent that combines alert verification, natural-language video search, and hourly summarization for my warehouse cameras. Reuse existing Kafka and Elasticsearch services where possible, and produce a deployment plan before running Docker Compose. Going further Clone the VSS Blueprint repository Install the VSS Agent Skills Try the Build Vision Agent skill with the sample prompt above on your own cameras