NVIDIA Introduces SkillEvaluator to Measure and Benchmark AI Agent Skills

NVIDIA Introduces SkillEvaluator to Measure and Benchmark AI Agent Skills NVIDIA has launched SkillEvaluator, an open-source evaluation framework designed to benchmark how packaged 'skills' affect AI agent performance. Based on an August 2026 snapshot evaluating more than 300 verified skills across over 30 NVIDIA products, the tool measures performance lift across two test harnesses through static checks, similarity screening, and isolated sandboxed executions. Evaluations follow a three-tier system: Tier 1 static checks, Tier 2 distinctiveness screening, and Tier 3 isolated sandboxed comparison runs via Harbor. Across published skills, installing verified skills raised Correctness from 46 to 87 (+41 points) and Effectiveness from 39 to 78 (+39 points). Evaluations across Claude Code and OpenAI Codex showed overall Skill Lift averages of +34 and +29 points, respectively (+42 and +36 excluding Security). External platforms including OpenClaw (ClawHub) and Nous Research (Hermes Agent via SkillSpector) are piloting SkillEvaluator workflows. Skill Lift varied significantly more across products (+2 to +46 points) than across harnesses (~5 point variance), and token reductions were not universally automatic. Three-Tier Evaluation Methodology Before a skill is verified and published, SkillEvaluator assesses it through three standalone stages. Tier 1 conducts static analysis for schema and frontmatter validation, code quality, prompt injection, data exfiltration, secret and PII detection, license conformance, and script linting. Tier 2 applies embedding similarity to detect duplicate guidance within a skill and catalog-wide coverage overlaps. Tier 3 performs live agent evaluations against generated task datasets in isolated sandbox environments powered by Harbor, measuring the exact differential impact between runs with and without the skill installed. Live Benchmark Results and Baseline Scores In the August 12, 2026 benchmark snapshot, macro-averaged scores across Codex and Claude Code showed baseline without-skill performance between 39 and 46 out of 100 for Correctness (46), Discoverability (42), Effectiveness (39), and Efficiency (43). Security maintained a high baseline of 97. When verified skills were installed, Correctness jumped 41 points to 87, Discoverability rose 40 points to 82, Effectiveness increased 39 points to 78, and Efficiency gained 35 points to 78, leading to an average Skill Lift of +39 points across non-security dimensions. Limitations and Harness Variations NVIDIA noted known evaluation constraints: Discoverability and Efficiency evaluate how the skill is handled, meaning non-zero baselines without skills reflect general clean execution, productive tool usage, and leaving skills unloaded for unrelated tasks. Furthermore, 85% of evaluated skills were tested on a single attempt per task (15% on two), and confidence intervals were not reported. Across harnesses, Claude Code achieved a +34 point overall lift (+42 excluding Security), while OpenAI Codex recorded a +29 point lift (+36 excluding Security). Partner Pilots and Operational Takeaways SkillEvaluator integrations are underway with OpenClaw on ClawHub, presenting comparative run data in an Evals tab, and Nous Research, which uses SkillSpector for pre-installation security scans running in roughly 1.4 to 1.5 seconds. NVIDIA observed that skill lift relies far more on the product and task domain (ranging from +2 to +46 points) than the choice of agent harness. Additionally, token efficiency varied widely: while the jetson-optimize-memory skill reduced token consumption by 76.9% and execution time by 53.7%, the cuopt-install skill increased tokens by 120.3% and runtime by 20.8%.

NVIDIA推出开源工具SkillEvaluator：评估与基准测试AI智能体技能性能 NVIDIA发布开源评估工具SkillEvaluator，用于通过静态检查与真实任务运行来衡量技能（Skills）对AI智能体表现的影响，并公布了覆盖30多个NVIDIA产品的300多个已验证技能的基准测试结果。 SkillEvaluator采用三层评估机制，包括静态安全检查、嵌入相似度去重，以及基于Harbor沙盒的对照实时评估。 在Codex和Claude Code两大测试框架中，验证技能使正确性、可发现性、有效性及效率均获得超过35分的Skill Lift提升，总体均分提升31分（排除安全性后为39分）。 OpenClaw与Nous Research已开展试点，将该工具集成到各自技能中心与安装流程中。 测试发现产品本身的领域与任务设计对技能提升幅度的影响远大于智能体框架本身，且技能并不必然节省Token与执行时间。 背景与三层评估方法论 智能体的效率受制于所获上下文，专用技能封装了指令、示例与工具指引，以加速任务求解。在技能发布前，SkillEvaluator执行三层评估：Tier 1进行结构验证、提示注入检测、数据外泄及PII扫描等静态安全检查；Tier 2利用嵌入相似度排查重复指引与覆盖重叠；Tier 3则通过开源框架Harbor在隔离沙盒中运行实时任务，分别测试安装与未安装技能时的表现并对比得分。 基准评估数据与提升表现 根据基准快照，未安装技能时，智能体在正确性、可发现性、有效性和效率上的基准均分在39至46分之间（满分100），安全性则高达97分。加载已验证技能后，正确性由46分升至87分（+41分），有效性由39分升至78分（+39分），可发现性与效率分别提升40分和35分，安全性保持98分微升。跨框架方面，Claude Code的整体提升为34分，OpenAI Codex为29分。 测试局限性说明 基准测试存在一定局限：未安装技能时，可发现性与效率基准分（42与43分）主要反映智能体的常规工具使用及在无关任务时不加载技能的表现，而非零分起步；此外，85%的技能仅经过单次尝试评估，15%进行了两次，由于实时运行存在波动且汇总未附带置信区间，单项技能分数会有所起伏。 合作伙伴试点与落地实践 OpenClaw正在ClawHub上为官方组织试点SkillEvaluator，在Evals标签页直观展示安装与未安装技能的对比数据。Nous Research则在Hermes Agent中结合SkillSpector进行可选的预检扫描，可针对PII、Unicode走私、脚本规范等提供行级反馈，每次扫描耗时约1.4至1.5秒，并通过了29项测试。 核心实证发现 基准评测得出了三项实践经验：一是详尽定义任务和边界的高质量评估数据集直接决定了评估信号的精度；二是产品本身对提升幅度的影响超过智能体类型，产品间Skill Lift差异在+2至+46分之间，而框架间平均仅差约5分；三是引入技能不代表自动节省Token，例如jetson-optimize-memory将Token减少了76.9%并缩减了53.7%耗时，但cuopt-install却使Token增加了120.3%、耗时增加20.8%，反映出进一步优化的空间。

AI agents are only as effective as the context they receive. Even with capable models and well-documented NVIDIA libraries, agents can spend extra steps finding the right tools, burn tokens on dead ends, or struggle with specialized tasks. Skills package the instructions, examples, and tool guidance for agents to move faster from intent to solution. To measure whether these skills improve agent trajectories and outputs, NVIDIA built an open evaluation layer for agent skills. NVIDIA SkillEvaluator is an open source tool for measuring how skills affect agent performance through static checks and real-world task runs with and without each skill. NVIDIA verified Skills are packaged, signed capability descriptors that tell an agent exactly what an NVIDIA product does, when to invoke it, and how to call it. The verified part is the measurement that determines it is ready. This post shares the first benchmark results for more than 300 verified skills across over 30 NVIDIA products. Each skill was evaluated on two independent harnesses. For each harness, Skill Lift was calculated by comparing scores from runs with and without the skill installed. NVIDIA publishes plugins for Claude Code , Codex , and Cursor , and the same skills are available through Skills.sh , ClawHub , and Hermes Hub . Methodology: How skills are evaluated Before a skill is published, it goes through three tiers of evaluation. Each answers a different question, and each can run on its own. Tier 1—Safety and structure: Runs static checks for schema and frontmatter validation, quality scoring, security scanning for prompt injection and data exfiltration, secret and PII detection, license checks, and script linting. Tier 2—Distinctiveness: Uses embedding similarity to identify duplicated guidance inside a single skill and overlapping coverage across the catalog. Tier 3—Live evaluation: Runs a live evaluation with an agent against generated tasks, once with the skill installed and once without, inside an isolated sandbox, and measures the difference. Tier 3: How live evaluations work Tier 3 evaluations use Harbor , an open-source framework for running agent evaluations in repeatable, isolated environments. SkillEvaluator handles the Harbor setup. It turns evaluation cases into tasks, runs the agents in sandboxes, collects the results, and calculates the skill’s impact. Every result comes from a controlled comparison. For each evaluation case, an agent harness runs twice: once with the verified skill installed and once without it. Each run executes in its own isolated sandbox with the same prompt, model, task inputs, and grading criteria. Within each harness, the only experimental variable is whether the skill is installed. This comparison is repeated across two agent harnesses. The difference between the with-skill and without-skill scores is the skill’s contribution, reported as Skill Lift in points. A simple example First, generate an evaluation dataset for a skill: skillevaluator create-eval-dataset ./my-skill --full This creates evals/evals.json . Each case includes an ID, prompt, and expected output, plus optional assertions. With –full, the dataset includes explicit, implicit, contextual, and negative cases. After reviewing the cases, run the live comparison: skillevaluator tier3 evaluate ./my-skill \ --agents codex \ --env-mode docker SkillEvaluator converts the cases into a Harbor task bundle, runs each case with and without the skill, grades both runs, and produces the scores and Skill Lift. This keeps the workflow simple for users while Harbor manages the underlying execution and isolation. See the Tier 3 Live Evaluation documentation for the complete workflow. Live evaluation results All figures in this section use the August 12, 2026 snapshot of benchmarks.json at commit 738d79e . Scores are macro-averaged across published skill–harness results, giving each skill–harness pair equal weight. Skill Lift is the with-skill score minus the without-skill score, measured in points. The catalog is evaluated continuously, so current numbers remain available in benchmarks.json in the nvidia/skills repository. Without-skill baseline evaluation results In the benchmark snapshot, the following scores represent averages from without-skill runs across skills in the NVIDIA/skills repository evaluated using Codex and Claude Code. They show how agents performed the same tasks without the relevant skill installed. Table 1 defines the five scoring dimensions. Average baseline scores ranged from 39 to 46 out of 100 across Correctness, Discoverability, Effectiveness, and Efficiency, indicating substantial room for improvement. Security was the exception, with an average baseline score of 97. For Security, the primary objective was to verify that installing a skill did not introduce a regression. Dimension What It Measures Baseline score without a skill (score out of 100) Correctness Is the final answer correct? 46 Discoverability Whether the right skill loads when it’s relevant, and stays unloaded when it isn’t? 42* Effectiveness Did the agent reach the user’s goal and follow the expected workflow? 39 Efficiency Did the agent get there without wasted steps or redundant tool calls? 43* Security Whether the run avoids unsafe operations, secret leakage, and unauthorized access 97* Table 1. The five scored dimensions and how agents perform without a skill loaded. See asterisk explanation in the verbiage below Known limitations Correctness, Effectiveness, and Security measure the outcome of the run, so their baselines show what the agent can do without the relevant skill installed. Discoverability and Efficiency also measure how the skill is used: whether the agent finds it, reads it before acting, and avoids unnecessary steps. Without the skill, those actions are unavailable. However, the agent can still earn credit for productive tool use, clean execution, and correctly leaving the skill unloaded for unrelated tasks. These scoring components help explain why the baselines are around 42 and 43 rather than zero. Most skills were evaluated at a single attempt per task. Of the skills with published results, 85% ran one attempt, and 15% ran two. Live agent runs vary between runs, so individual skill scores vary. Catalog-wide averages aggregate thousands of trials, but this post does not report confidence intervals. With-skill and without-skill evaluation results The same evaluations used skills from the NVIDIA/skills GitHub repository. Table 2 shows the average with-skill scores and Skill Lift relative to the without-skill baseline across Codex and Claude Code. Verified skills improved agent performance across both evaluated harnesses, with the largest gains in Correctness, Discoverability, Effectiveness, and Efficiency. Skill Lift is reported in points, not percent change. Security was already high at baseline, so its measured gain is smaller. Discoverability and Efficiency should be read as indicators that the skill is being activated and used correctly when present. Dimension Without-skill score With-skill score Skill Lift Correctness 46 87 +41 Discoverability 42 82 +40 Effectiveness 39 78 +39 Efficiency 43 78 +35 Security 97 98 +1 All dimensions (average) — — +31 Excluding Security (average) — — +39 Table 2. Average Skill Lift from verified skills On Correctness and Effectiveness—the two dimensions measured consistently in both conditions—mean scores rose from 46 to 87 and from 39 to 78, gains of 41 and 39 points. These scores are not pass-probability estimates; they show higher average performance on the evaluated specialized tasks. Discoverability and Efficiency show Skill Lift values of 40 and 35 points. Both are scored against the skill itself, so read them as evidence that the agent activates and uses the relevant verified skill correctly once installed rather than as a measure of unaided agent behavior. That property matters: every skill in an environment competes for the agent’s attention, and a skill that loads when it is not relevant can reduce agent performance. Skill Lift by harness Scope Claude Code OpenAI Codex All dimensions +34 +29 Excluding Security +42 +36 Table 3. Skill Lift varies by harness. Claude Code shows a higher Skill Lift across all dimensions Both harnesses show consistent, meaningful gains. The difference between them is expected, given different default system prompts, context handling, and tool-calling implementations. The verified Skill supplies structured grounding that neither harness produces on its own. Partner pilots use SkillEvaluator OpenClaw is piloting SkillEvaluator for official organizations on ClawHub. The integration runs Tier 3 evaluations and displays with-skill and without-skill results in an Evals tab, enabling developers to review evaluation signals where they discover and adopt skills. Nous Research tested SkillEvaluator in Hermes Agent with an optional advisory scan by SkillSpector in the skills install flow. The integration checks for PII, Unicode smuggling, script linting, license, and security issues, and surfaces file-line findings before installation. The workflow is covered with 29 passing tests, with each skill scan taking approximately 1.4-1.5 seconds. Key findings The evaluation results highlight three practical findings for teams building and testing agent skills. Better evaluation datasets produce better skills Teams that clearly define important tasks, expected outputs, and out-of-scope requests produce sharper evaluation signals, and it happens before any agent runs. A skill can only be measured as precisely as its evaluation set describes the job. The product matters more than the agent Skill Lift varies far more across products than across harnesses. Claude Code and Codex differ by about 5 points on average. But per-product Skill Lift ranges from roughly +2 to +46. The domain, the task, and the evaluation design mattered more than the harness. Token savings are not automatic SkillEvaluator tracks token usage separately from Efficiency. In two single-attempt examples, one skill from the NVIDIA verified skills repository , jetson-optimize-memory reduced tokens from 617,306 to 142,540 (76.9%) and execution time from 474.9 to 220.0 seconds (53.7%). Conversely, cuopt-install increased tokens from 25,227 to 55,582 (120.3%) and execution time from 34.0 to 41.1 seconds (20.8%), identifying an opportunity for further optimization. SkillEvaluator revealed whether a skill improves token and execution efficiency or needs further optimization. Get started To get started, visit the SkillEvaluator docs , or review our verified skills from the NVIDIA verified skills catalog at GitHub . Acknowledgments We’d like to thank Roshni Malani, Meghana Puvvadi, Subodh Prabhu, Mohit Gupta, Yogesh Dangi, Yashraj Basaravaj Patil, Keshav Pradeep, Siddharth Itagi, Alejandro Sanabria Portala and Pranita Maske for contributing to this work.