NVIDIA NeMo Gym Enhances Reliability and Efficiency of Siemens EDA AI Agent for Chip Design

NVIDIA NeMo Gym Enhances Reliability and Efficiency of Siemens EDA AI Agent for Chip Design Siemens and NVIDIA applied the open-source NVIDIA NeMo Gym evaluation environment to the Siemens Fuse EDA AI Agent, optimizing standard-cell characterization workflows and substantially increasing verification accuracy while decreasing computational overhead. Evaluating final engineering artifacts—such as Liberty (.lib) files—rather than intermediate steps prevents silent error propagation in multi-stage chip design workflows. The evaluation environment uses a multi-factor composite verifier assessing coverage, validity, relative validation, integrity, and tolerance accuracy. Refining skill guidance and MCP tool schemas raised composite benchmark scores from approximately 55% to 95%. Agent interaction turns decreased by approximately 66% (from 155 to 53) and average output tokens dropped by about 64% (from ~81K to ~29K). Challenges in Standard-Cell Characterization Standard-cell characterization requires generating accurate timing, power, and noise models across varying operating conditions whenever a semiconductor foundry releases a new process design kit (PDK). The Solido Characterization Suite agent automates tool setup, SPICE simulation, distributed computing, and Liberty file generation via Model Context Protocol (MCP) interfaces. In multi-step pipelines, undetected early setup errors often propagate silently into final artifacts rather than triggering explicit failure states. Artifact Verification and Evaluation Loop NeMo Gym integrates into the characterization flow by establishing a controlled evaluation loop that tracks full rollout trajectories, thinking traces, and tool actions. Rather than verifying isolated tool calls, a domain-specific verifier evaluates end-to-end output Liberty files. The verifier calculates a composite reward score based on five gated and continuous metrics: cell and PVT corner coverage, file validity, relative validation against reference runs, integrity against truncated or reused data, and the proportion of timing and power values within acceptable tolerance. Optimization and Performance Improvements Evaluations conducted on the open-source GF180MCU PDK across 36 PVT corners using Claude Opus 4.8 enabled engineers to inspect trajectories and pinpoint failure modes. By clarifying agent skills, simplifying overlapping shell and file interfaces, and establishing explicit JSON schemas for tool inputs, the team improved execution efficiency. The mean composite score increased from approximately 55% to 95%, while agent turns dropped from 155 to 53 and output tokens declined from roughly 81,000 to 29,000.

西门子与英伟达利用 NeMo Gym 提升芯片设计 AI 智能体可靠性与工具调用效率 英伟达与西门子团队合作，利用开源库 NVIDIA NeMo Gym 构建评估与优化环境，解决了 AI 智能体在芯片标准单元特征化多阶段流程中早期错误静默传播的问题，大幅提升了工具调用的准确率与执行效率。 标准单元特征化因阶段依赖性高，前期设置错误极易静默扩散至最终的 Liberty (.lib) 文件中。 NeMo Gym 通过构建端到端环境与多维度门控验证器（覆盖率、有效性、相对验证、完整度及容差准确率），直接对最终工件进行校验。 在开源 GF180MCU PDK 基准测试下，基于 Claude Opus 4.8 的智能体综合验证得分从约 55% 提升至约 95%。 通过优化 MCP 工具描述与输入 JSON 架构，智能体交互轮数减少约 66%（155 降至 53），输出 Token 减少约 64%（约 8.1 万降至 2.9 万）。 特征化流程与静默错误挑战 标准单元特征化需要为逻辑单元在各种输入转换、输出负载及 PVT（工艺、电压、温度）条件下生成精确的时序、功耗和噪声模型。西门子的 Solido Characterization Suite 智能体负责编排底层工具与大规模 SPICE 仿真，但由于多阶段相互依赖，前期工具配置微小偏差不会导致显式崩溃，而是生成看似合理却存在阈值错误或缺失单元的无效工件。因此，传统针对单个工具调用的检查无法保障最终交付质量。 NeMo Gym 评估环境与验证器设计 团队将 NeMo Gym 接入自动化流程，建立包含任务、状态、工具和验证器的受控闭环。为防止智能体出现截断、重格式化或复用已有数据等作弊行为，团队设计了严格的门控复合奖励函数。该函数以覆盖率、有效性、相对验证和完整性为布尔门控，并乘以容差内时序与功耗数值的比例；只要任何一项门控未通过，总奖励直接归零。 优化实施与性能基准提升 在开源 GF180MCU PDK（测试子集为 2 个单元覆盖 36 个 PVT 角点）的约 25 次运行优化中，团队基于轨迹诊断简化了重叠的文件与 Shell 接口，并采用明确的 JSON 架构规范模型上下文协议（MCP）工具输入。优化后，智能体的基准综合评分从约 55% 跃升至约 95%，执行轮次从 155 轮缩减至 53 轮，平均输出 Token 从约 8.1 万降至 2.9 万，显著降低了多阶段工程流程的开销并提高了可靠性。

AI agents offer remarkable productivity gains in complex chip design workflows, but their tendency to generate plausible yet incorrect artifacts makes reliability a major barrier to production adoption. In multistage engineering workflows, an undetected error early in setup can silently propagate through downstream stages and final artifact generation. Evaluating and refining these agents against verified final outputs, rather than isolated intermediate steps, is essential for building trustworthy automation. NVIDIA NeMo Gym , an open library for evaluating and improving models and agents using environments, addresses this challenge. The NeMo Gym agentic evaluation environment is designed to diagnose failure modes and track error propagation across complex, multistage workflows. By establishing a domain-specific verifier to inspect final outputs against verified reference artifacts, NeMo Gym enables engineers to trace complete rollout trajectories and identify root causes behind subtle errors. This post walks through how NeMo Gym was used to evaluate and improve the reliability of the Siemens Fuse EDA AI Agent when calling characterization tools. The Siemens Fuse team used NeMo Gym to scale up evaluations and verify the resulting Liberty (.lib) files against known references and tracing—both common and rare failures through each rollout. Those findings guided changes to the agent’s skill and Model Context Protocol (MCP) tool descriptions, increasing the measured benchmark composite score from approximately 55% to ~95% across a benchmark of representative standard-cell characterization tasks, while reducing turns and output tokens. Siemens’ work provides a practical example for chip design teams and EDA developers on how to leverage the NVIDIA AI software stack and open libraries to improve agent reliability and chip design workflows. Automating standard-cell characterization Standard-cell characterization is the process of generating accurate timing, power, and noise models for logic cells across operating conditions. It is foundational to chip design and required whenever a semiconductor foundry issues a new process design kit (PDK). Each new PDK carries SPICE transistor models; design rules; and process, voltage, and temperature (PVT) definitions that connect the manufacturing process to design tools. When a foundry updates a PDK, existing characterization datasets may no longer accurately describe the updated devices. Timing and power models must then be regenerated and validated for every affected cell across combinations of input slew, output load, and PVT conditions. The Solido Characterization Suite agent automates this process. The agent orchestrates tools from the broader Solido Characterization Suite through Siemens Fuse EDA AI Agent, which exposes MCP servers and skills to common coding harnesses including Claude Code, Codex, and Cursor. MCP interfaces make these accelerated tools accessible to engineering agents, while skills guide their use in characterization workflows. Faster simulation helps agents complete engineering iterations sooner, while NVIDIA NeMo Gym helps developers measure and improve how reliably and efficiently agents orchestrate those tools. The Solido Characterization Suite agent must perform tool setup, large-scale SPICE simulation, distributed compute, Liberty file generation, debugging, and final validation (Figure 1). Errors often produce plausible outputs instead of clean failures, leaving incorrect thresholds, missing corners, or incomplete cells hidden. Because these stages depended on one another, an early setup error could propagate into every step that followed. Defining a rigorous evaluation protocol was a key step in preparing the Solido Characterization Suite agent for production workflows. Figure 1. Agentic characterization workflow from SPICE inputs through pilot validation, scale-out, Liberty generation, and waveform comparison. Early setup errors propagate to the final library Measuring and improving agent reliability with NeMo Gym Part of the NVIDIA NeMo collection of open libraries for agent specialization and optimization, NeMo Gym integrates with the standard-cell characterization flow inside a controlled evaluation loop (Figure 2). It defines the task, tools, state, and verifier for each rollout so that attempts started from a clean state and produced comparable results. It also records thinking traces, input and output messages, and tool actions. This enabled engineers to run the following optimization loop: Run baseline rollouts with a versioned task and verifier in a custom NeMo Gym environment Inspect trajectories, failed tool calls, and verifier diagnostics Identify ambiguous instructions, unproductive actions, and incorrect tool selection Refine the agent’s skills and MCP tool descriptions Repeat the same tasks under the same evaluation conditions Compare verified outcomes and efficiency per successful result Figure 2. NeMo Gym checks the Liberty files produced by the Solido Characterization Suite agent and Solido tools, then feeds failure analysis into the next skill and tool-description revision. The entire process is driven by a coding agent with a human in the loop Reward design and artifact verification As the team analyzed agent trajectories, end-to-end evaluation uncovered failures that intermediate checks missed. For example, final Liberty verification exposed missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance. These findings show why engineering agents must be evaluated against final artifacts rather than individual tool calls. To make the work repeatable, the team implemented verifiers that provide a final composite reward score based on the following metrics: Coverage : Whether the agent produced every expected cell and PVT corner. Validity : Whether the delivered Liberty files are well-formed and usable by the validation tools. Relative validation: Whether the output introduced failures beyond those already present in the reference. Accuracy : The fraction of expected timing and power values that falls within the defined tolerance. Integrity : Whether the characterization workflow produced the complete result rather than reusing, truncating, or reformatting existing values. The reward score is defined as: \(R=\mathbf{1}\left[\text{Coverage}\right]\times\mathbf{1}\left[\text{Validity}\right]\times\mathbf{1}\left[\text{Relative Validation}\right]\times\mathbf{1}\left[\text{Integrity}\right]\times\frac{X_{\text{tol}}}{X_{\text{total}}}\) Here, \(X_{\mathrm{tol}}\) is the number of values in the output that are within tolerance, and \(X_{\mathrm{total}}\) is the total number of output values. Their ratio represents the fraction of output values within tolerance. Each indicator \(\mathbf{1}[\cdot]\) equals \(1\) when its corresponding gate passes and \(0\) when it fails. The reward is zero if any gate fails. Using this verifier, the team systematically evaluated and tuned the agent’s performance across multiple iterations. Optimizing the agent The team used the open source GF180MCU PDK to benchmark the Solido Characterization Suite agent on Liberty-file generation. The dataset includes 229 standard cells across 36 PVT corners, using Solido AFS for SPICE simulation. A custom agentic environment was built using NeMo Gym. For each run, the characterization flow was restricted to two cells across 36 PVT corners, as this smaller subset was sufficient for tuning the agentic workflow. Each run took approximately 1.5 to 2 hours, and the team conducted approximately 25 rollouts using Claude Opus 4.8 running in Claude Code as the fixed model and agent harness. Across successive evaluation rounds, the team refined the agent’s characterization skills and MCP tool descriptions. To improve tool selection, they simplified overlapping file and shell interfaces and defined tool inputs with explicit JSON schemas, making characterization tools easier for the agent to call. Following these refinements, the mean composite verification score across evaluated rollouts increased from approximately 55% to 95%. The improved workflow also required fewer agent turns and output tokens. Agent turns fell from 155 to 53, a reduction of approximately 66%, while output tokens fell from approximately 81K to 29K, a reduction of approximately 64%. Model Benchmark composite score Turns Avg. output of tokens Claude Opus 4.8 ~55% → ~95% 155 → 53 ~81K → ~29K Table 1. Development benchmark results before and after harness optimization Developing reliable agents in other domains using the same method The same evaluation method applies to engineering agents in EDA, CAE, and other engineering domains: Evaluate the final engineering artifacts and not just whether individual tool calls succeeded. Write a domain-specific grader/verifier that detects missing, truncated, reformatted, or reused results to prevent agent cheating. Collect large numbers of rollouts to target improved reliability through optimized skills and MCP tool interfaces. Get started with NeMo Gym Use the NeMo Gym Quickstart guide to run an initial evaluation. Then apply the approach outlined in this post to one representative task: define a valid final artifact, build a domain-specific verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent evaluation conditions. For more details about how Solido Fuse brings generative and agentic AI to Solido Characterization Suite agent, watch the Self-Verifying, Long-Running Agentic Characterization Workflow Demo . Learn more about the collaboration between NVIDIA and Siemens . Acknowledgments We would like to acknowledge Gunjan Shah of NVIDIA, along with Tim Wheler, Lakshminarayanan Ramesh, and Carl Hofmeister of Siemens for their key contributions to developing the agent and establishing the evaluation methodology.