NVIDIA AI Red Team Outlines Key Defenses to Secure Enterprise AI Agents

NVIDIA AI Red Team Outlines Key Defenses to Secure Enterprise AI Agents Evaluations by the NVIDIA AI Red Team reveal recurrent vulnerabilities in AI agents, including missing access controls, arbitrary code execution, absent egress restrictions, and exposed credentials. The team emphasizes implementing deterministic, environmental controls outside the model's control plane rather than relying on prompt-based guardrails. Most tested agents lacked strict user-level access controls, allowing internal users to hijack the agents' legitimate enterprise credentials. Exposing command execution tools or broad file-write privileges enables remote code execution via prompt injection or malicious package installations. Prompt-level defenses and LLM-as-a-judge approaches consistently fail against techniques like social engineering frames and crescendo attacks. Robust security requires architectural controls: strict authorization, isolated sandboxes, default-deny egress policies, and short-lived secrets stored off-environment. Agent Access Control and Least Privilege The NVIDIA AI Red Team observed that AI agents frequently hold individual user credentials while remaining accessible to any internal network user. This setup allows unauthorized users to exploit or extract legitimate credentials. Security must begin with enforcing strong access controls that restrict agent interaction strictly to authorized individuals and match agent capabilities to the invoking user's permissions. Restricting Command Execution and File Manipulation Command shells are commonly supplied to agents to support developer tasks like package management and testing. However, attackers can leverage direct or indirect prompt injection to execute arbitrary commands, spawn reverse shells, or achieve persistence. Even without direct shells, writable file paths like shell startup scripts or MCP configuration files can trigger code execution. Red team guidance recommends eliminating command tools where possible, locking write permissions to non-executable workspaces, and sandboxing necessary execution environments. Network Egress and Secrets Management Unrestricted outbound network connections enable data exfiltration, reverse shells, and SOCKS proxies. Implementing default-deny egress rules forces interactions through the agent itself, adding significant operational friction for attackers. Furthermore, placing plaintext secrets in environment variables or configuration files leaves them vulnerable to extraction via shell inspection or chat output; systems should instead supply ephemeral, scoped tokens retrieved on demand from dedicated secret managers. Deterministic Defenses Over Model Guardrails System prompts and LLM-as-a-judge validators routinely succumb to adversarial tactics, including benign framing, multi-turn crescendo attacks, and weaponized package installations. Because probabilistic models cannot guarantee policy compliance under adversarial manipulation, effective agent security requires deterministic environmental controls, hardened runtimes, restricted package sources, and limited persistent storage.

NVIDIA红队提出安全部署AI代理的四项关键控制措施 NVIDIA AI红队在过去六个月评估各类AI代理后指出，单纯依靠提示词工程或“LLM担任裁判”等概率性防御极易被绕过。团队总结了常见攻击面，并强调必须在模型控制平面之外实施确定性架构控制，以防止恶意代码执行、数据外泄和凭证泄露。 AI代理常见故障模式包括缺乏访问控制、工具允许任意代码执行、缺少出站网络控制以及明文暴露机密。 攻击者常利用身份假冒、渐进式引导（Crescendo攻击）或包安装等看似合法的开发工作流绕过基于LLM的审查。 防范代码执行风险需隔离运行环境，严格限制文件写入路径，并对命令行输入进行清理与规范化。 安全架构必须强制实施默认拒绝出站网络流量，并禁止向代理的运行环境暴露持久性机密凭证。 实施严格的代理访问控制 评估发现许多代理持有特定用户凭证，却对内网中所有授权用户开放，导致攻击者能够滥用或盗取凭据。防御的第一道防线是限制仅特定经授权的用户可调用代理，并按照最小权限原则将代理权限与当前调用者严格匹配。若代理直接拒绝非授权用户的请求，攻击难度会大幅提升。 严格限制代码执行与文件写入权限 命令行和Bash工具虽然通用，但攻击者可通过直接或间接提示注入控制命令输出，甚至借助文件写入工具篡改~/.bashrc、~/.gitconfig或配置文件以实现特权提升与远程代码执行（RCE）。团队建议尽可能避免提供命令行工具，在操作系统层禁止写入不可执行工作区之外的目录；若必须执行命令，须在强隔离沙箱（如Docker或VM）内运行，配合严格的命令白名单并对参数进行规范化。 默认拒绝网络出站连接 允许出站网络连接为攻击者回传数据以及建立反向Shell或SOCKS代理提供了通道。团队建议在代理触及的所有网络边界强制执行默认拒绝策略，仅为工作任务保留极简的端点白名单。严格的出站控制能迫使攻击交互必须通过代理进程完成，极大增加攻击者的操作负担和失败率。 隔离机密信息与弃用软防御 针对非代理容器的环境变量注凭证方法在具有代码执行权限的代理中不再安全，攻击者可通过执行env命令或对话套取明文凭证。机密应存放在专用机密管理器中，由需使用的进程在内存中按需临时获取，或使用短期作用域令牌。NVIDIA红队强调，系统提示词和LLM裁判极易被语境伪造或工作流伪装突破，只有独立于模型控制平面之外的确定性架构控制才能切实降低滥用风险。

Knowledge workers are increasingly integrating AI agents into their workflows. Agents that function as “digital coworkers” offer clear benefits. For example, they can review a bug report, implement and test a fix, push a patch, and ping a human for review. By handling routine tasks, agents have the potential to deliver large productivity gains. On the other hand, connecting a large language model (LLM) to live tools and corporate data through an agentic harness risks turning a helpful assistant into privileged software with a poorly understood attack surface. Over the past six months, the NVIDIA AI Red Team has assessed multiple AI agents—from simple interactive coding tools to always-on autonomous digital assistants. When an agent proved exploitable, we typically saw the same key failure modes, regardless of the framework or harness used, including: Lack of access control to the agent. Agent tools that enable arbitrary code execution. No network egress controls. Secrets exposed to the agent in plaintext. In this post, we examine these failure modes and describe the controls that succeed under adversarial pressure. While our examples focus on chat-connected agents—the majority of those we encountered in our evaluations—the patterns generalize to any agent. Implement agent access control The most common failure mode in current AI deployments is a lack of access control to the agent. We discovered multiple agents that held credentials for individual users and were accessible to any authorized user within the internal network. While this opened the door to misuse of the agents’ legitimate credentials, it also often enabled us to collect those credentials and use them outside of the intended context of the agent, as shown in later examples. Recommendations: Use strong access controls as the first line of defense against adversarial activity. Restrict each agent to explicitly authorized users; agents that did not respond to unauthorized users were significantly harder to test. Match the agent’s permissions to those of the user invoking it, following the principle of least privilege. Limit code execution Many harnesses expose a Bash shell or command-execution tool. These are often used because of their generality. They support a wide range of routine tasks, without requiring a separate tool per function. However, when model output controls command execution, an attacker who can influence that output—through direct input, or indirect prompt injection—may be able to run commands in the execution environment. This can enable them to achieve malicious outcomes such as data exfiltration or execution persistence on the host. Common mitigations to this risk of arbitrary command execution often involve using LLM-as-a-judge review agents to block harmful or malicious commands from running, the use of allowlists for acceptable commands, or simply trusting that the model “knows better.” All provide limited defense to adversarial manipulation. Many common agentic command-line tasks support common development workflows and test-driven development. This means they generally require the ability to execute commands such as pytest or npm install , meaning LLM-as-a-judge patterns are often predisposed to accept the execution of these commands. When influenced by attacker-controlled input, however, these commands are equivalent to arbitrary command execution. In some cases, obtaining full remote code execution (RCE) with a reverse shell is as simple as asking the agent to write and execute a Python script or install a remote package, as shown in our previous blog post on this topic . Even without a command-line tool, the ability to interact with the environment in which the agent is running through file read and write tools also often exposes unexpected paths to code execution and privilege escalation. An attacker who can write content to system files such as ~/.bashrc or ~/.zshrc , or configuration files such as ~/.gitconfig , hooks.json , MCP.json , or skills files, can achieve code execution when a different process executes the relevant file, even if command-line execution is not directly available . The locations and files that agents can write to should be strictly controlled and be limited to non-executable locations. Recommendations: Treat arbitrary code execution as the single highest-impact risk in an accessible agent. Avoid command-line tools wherever possible. Block writes outside a non-executable workspace at the OS level. If a command-line execution tool is required, use a strict least-privilege allowlist of executable commands, and run the tool in an isolated execution environment with strong network egress controls, as we will detail next. Exercise caution when processing arguments or strings such as filenames, document titles, and other external data via the command line. Ensure that they are sanitized and normalized before use to prevent issues such as path traversal or command injection. Deny network egress by default Outbound network connections permit data exfiltration and the creation of direct connections, such as reverse shells and SOCKS, through which attackers can interact directly with the agent’s runtime environment. When network egress controls were enforced and appropriately least-privileged, we ran all of our interactions through the agent process, slowing down our pace of execution and making impact less reliable. Maintaining the agent’s state and alignment, navigating output filters, and restarting and manipulating sessions after the agent began refusing requests all added operational burden. Recommendations: Apply a default-deny network-egress policy, with a least-privilege allowlist of endpoints scoped to the minimal set required for tasks that the agent is expected to perform. Enforce these restrictions at every network boundary the agent touches using environmental controls that are not accessible to the agent . Keep secrets out of the agent’s reach Agents often require access to secrets to perform their intended function: platform tokens, API keys, version control system (VCS) access tokens, and in some cases even OAuth refresh tokens. While conventional security advice suggests injecting secrets as environment variables in memory to prevent them from being written to disk, this is reasonable when only your code runs in a container. When an agent with command execution shares that environment, inducing it to run env , printenv , or /proc/self/environ enables direct inspection of them. Command-line tools are worth highlighting as particularly high-risk. CLIs cache credentials on disk in predictable places and print them back readily. We observed tokens in git repositories, .env files, bash history, .netrc files, OAuth 2.0 refresh tokens, and environment variables in the execution environment. Even when we could not establish a reverse shell, we could often exfiltrate credentials through the chat interface. Using a “frog-boiling” approach (see below), we led agents into surfacing several secrets exposed inside their own execution environments. Egress control prevented direct exfiltration over the network or direct examination in the file system, but the credentials were still exposed within the environment and reachable by the LLM, which delivered them to us via the chat interface. Recommendations: Never make persistent secrets accessible to an agent. Store secrets in a dedicated secrets manager. Retrieve secrets on demand only within the memory of processes that require them. Keep secrets out of the agent’s context window and execution environment. When a task requires a credential, use a short-lived, narrowly scoped token, and revoke the token as soon as the task is complete. Use deterministic controls as your first line of defense The most frequent attempt at mitigation we encounter is a system prompt telling the model to avoid risky or dangerous behaviors, sometimes reinforced by a second model judging inputs or outputs (the LLM-as-a-judge pattern). These are all enforced by an LLM, and inherit the same probabilistic, unreliable behavior as the LLM itself. Three general techniques reliably defeat these sorts of controls. We show each on multiple systems. Simply presenting the agent with a context in which malicious activities appear legitimate was remarkably effective. We often suggested to the agent that we were “debugging” or “admin users,” after which it would regularly comply with our requests. One agent even went so far as to write and execute a reverse shell for us: Figure 1. A “social engineering” attack succeeds against a Slack-connected agent, causing it to create a reverse shell for an attacker In other cases, it was possible to directly manipulate agent memory and AGENT.md files by instructing the agent to use file editing tools, which also allowed the construction of “debugging” and “authorized user” frames. Boiling the frog “Frog-boiling” (sometimes called a Crescendo attack ) progressively nudges the agent into desired behavior across multiple interactions, using previous conversation history to establish credibility and the benign nature of requests. The secret extraction proceeded by attempting to execute legitimate-looking workflows in a way that induces errors, and then finally “discovering” that the underlying cause of the errors was related to secrets, convincing the agent to reveal them to us. Misdirection through legitimate workflows Misdirection attacks such as package installation (first described in From Prompts to Pwns at Black Hat 2025) remain extremely effective. By inducing an agent to take an apparently benign action which has code execution as a side effect, it is typically straightforward to bypass any agent resistance. Coding agents routinely install libraries. Creating a malicious library as described in this post and then asking the agent to install it via pip install git+https://… appears to be a standard request; however, the weaponized package creates arbitrary code execution during the installation process. Recommended controls A consistent finding is that defenses in the same control plane as the LLM, particularly prompt-based defenses, are routinely subverted. Controls must be enforced outside of the model’s control plane. Recommended controls, in rough order of importance, are: Use access control on the agent. Only specific, authenticated users should be able to interact with the agent. Run arbitrary command execution only in a sandbox environment such as Docker, NVIDIA OpenShell , or a virtual machine. The environment must be properly hardened against escape. The environment must not be able to configure itself by writing or editing environment or agent configuration files. Default-deny network egress with a least-privilege allowlist of specific network resources required by the task, at every boundary the agent touches. Do not expose secrets at rest or to the environment. While injecting secrets as environment variables is standard in non-agentic applications, this is insecure for workloads that execute arbitrary code. Secrets should be stored in a secret manager, accessed on-demand, and limited to the process that requires them. Where possible, a token broker that provides least-privilege, ephemeral tokens should be used. Permit package installation only from validated package repositories. Block arbitrary URL and VCS-based installs by default. Least-privilege tools, MCPs, skills, etc. Only the tools the job requires; scrutinize anything that executes, writes, or reaches the network. Least-privilege persistent storage. Avoid volume mounts; where you can’t, scope them tightly and never mount anything writable into a path that is later executed. Use recent/frontier models , particularly for LLM-as-a-judge patterns, which can be more robust to adversarial manipulation. Conclusion Our AI Red Team’s experience in securing AI agents highlights the continued requirement for deterministic “hard” controls in defending AI agents. Fully autonomous systems with enterprise credentials are inherently risky and must be carefully secured. While frontier models make adversarial manipulation more difficult, with sufficient time and expertise, nearly all of them can still be subverted. Common flaws we observed include: weak access control (permitting any user to access the agent), execution and file-write tools that create opportunities for RCE, insufficient network egress controls permitting data exfiltration and reverse shells, and secrets in plaintext inside the execution environment accessible by the agent. Prompt-based guardrails, including the LLM-as-a-judge pattern, do not close any of these gaps. Architectural controls do: access control to the agent, hardened sandboxes with least-privilege access to enterprise data, default-deny network egress controls, and secrets kept out of the agent’s reach. When properly configured and enforced, these controls are highly effective in reducing adversarial abuse of AI agents. To learn more about designing secure agents from first principles, see the “How to Govern Autonomous Agents in Enterprise AI Factories” Technical Blog , which guides you through the first steps of implementing our Secure Agent Workspace Reference Design . To learn more about agent security, don’t miss the NVIDIA presentation at Black Hat USA: Cost-Effective, Private, Frontier-Grade: AI Agent Exploitation with a Fine-Tuned OSS Model To read more from the NVIDIA AI Red Team , see our other posts .