What happened
NVIDIA SkillEvaluator is an open-source tool for measuring how skills affect agent performance through static checks and real-world task runs with and without each skill. The tool evaluates skills across 30+ NVIDIA products and shows significant Skill Lift in Correctness, Discoverability, Effectiveness, and Efficiency, with average gains of 31 points overall and 39 points excluding Security.
NVIDIA SkillEvaluator is an open-source tool for measuring how skills affect agent performance through static checks and real-world task runs with and without each skill. The comparison is built around the same agent work being run with and without the relevant skill, so the measured difference stays tied to the skill under review. Static checks and real-world task runs therefore serve as the two parts of the measurement described by the tool. Together, they provide the basis for measuring how skills affect agent performance.
The tool evaluates skills across 30+ NVIDIA products and shows significant Skill Lift in Correctness, Discoverability, Effectiveness, and Efficiency, with average gains of 31 points overall and 39 points excluding Security. The reported dimensions make the results readable across the products being evaluated. Correctness, Discoverability, Effectiveness, and Efficiency describe the listed forms of Skill Lift, while the averages summarize the stated overall gains and the stated result excluding Security. The product coverage and the two averages are part of the same evaluation picture.
The evaluation results highlight three practical findings for teams building and testing agent skills: better evaluation datasets produce better skills, the product matters more than the agent, and token savings are not automatic. These findings connect the measurement process to the practical work of building and testing skills. They describe how the quality of the evaluation dataset shapes the resulting skill, how product differences outweigh harness differences, and why token usage must be checked separately rather than assumed to improve. Each point follows from the findings presented with the evaluation results.
The tool's ability to measure the impact of verified skills on AI agent performance and the significant Skill Lift in various dimensions. This measurement is the central reason to use the tool: it turns the effect of a verified skill on agent performance into an evaluation result. The listed Skill Lift dimensions give that result several distinct ways to be read, while keeping attention on the impact of the skill itself. Its value in this context is the ability to examine the stated performance effect.
The tool's three-tier evaluation process, which includes static checks, distinctiveness analysis, and live task runs in isolated environments. The static checks, distinctiveness analysis, and live task runs represent the tool's three stated layers of evaluation. The live runs take place in isolated environments, and the combination with the static checks and distinctiveness analysis keeps the process connected to both the skill itself and its observed task performance. This is the process used to produce the measurements described above.
Read the primary source: developer.nvidia.com ↗
Why it matters
The evaluation results highlight three practical findings for teams building and testing agent skills: better evaluation datasets produce better skills, the product matters more than the agent, and token savings are not automatic.
Better evaluation datasets produce better skills, as teams that clearly define important tasks, expected outputs, and out-of-scope requests produce sharper evaluation signals. Clear task definitions establish what the evaluation is meant to examine, expected outputs establish what a successful result should contain, and out-of-scope requests establish what should not be included. Together, those elements make the evaluation signal sharper, which is why dataset quality is one of the practical findings for teams building and testing agent skills.
The product matters more than the agent, as Skill Lift varies far more across products than across harnesses. The comparison is therefore centered on the product context in which a skill is used. Skill Lift can vary more between products than between harnesses, so product-specific evaluation remains important when teams interpret the results. This finding keeps the focus on the stated source of variation instead of treating the harness as the dominant explanation.
Token savings are not automatic, as the tool tracks token usage separately from Efficiency and reveals whether a skill improves token and execution efficiency or needs further optimization. The separate token measure prevents teams from treating every performance gain as a token gain. It shows whether a skill improves token and execution efficiency, or whether further optimization is needed, while the Efficiency dimension remains a separate part of the evaluation. The practical point is to inspect both results rather than infer one from the other.
The tool's ability to measure the impact of verified skills on AI agent performance and the significant Skill Lift in various dimensions. The ability to measure the impact of verified skills gives teams a way to discuss the reported Skill Lift using the dimensions named in the tool's results. That keeps the evaluation tied to observable comparisons and to Correctness, Discoverability, Effectiveness, and Efficiency, without reducing the discussion to a single outcome. It is the stated performance impact that makes the measurement matter.
The tool's three-tier evaluation process, which includes static checks, distinctiveness analysis, and live task runs in isolated environments. This process links the static review, distinctiveness analysis, and live task runs into the evaluation described by the tool. The isolated environments provide the setting for the live task runs, while the three stated layers keep the assessment focused on the skill and its performance. The process matters because it is the structure used to examine the reported results.
What to watch next
The tool's ability to measure the impact of verified skills on AI agent performance and the significant Skill Lift in various dimensions.
The tool's ability to measure the impact of verified skills on AI agent performance and the significant Skill Lift in various dimensions. This measurement is the central reason to use the tool: it turns the effect of a verified skill on agent performance into an evaluation result. The listed Skill Lift dimensions give that result several distinct ways to be read, while keeping attention on the impact of the skill itself. Its value in this context is the ability to examine the stated performance effect.
The tool's three-tier evaluation process, which includes static checks, distinctiveness analysis, and live task runs in isolated environments. The static checks, distinctiveness analysis, and live task runs represent the tool's three stated layers of evaluation. The live runs take place in isolated environments, and the combination with the static checks and distinctiveness analysis keeps the process connected to both the skill itself and its observed task performance. This is the process used to produce the measurements described above.
The evaluation results highlight three practical findings for teams building and testing agent skills: better evaluation datasets produce better skills, the product matters more than the agent, and token savings are not automatic. These findings connect the measurement process to the practical work of building and testing skills. They describe how the quality of the evaluation dataset shapes the resulting skill, how product differences outweigh harness differences, and why token usage must be checked separately rather than assumed to improve. Each point follows from the findings presented with the evaluation results.
The tool's ability to track token usage separately from Efficiency and reveal whether a skill improves token and execution efficiency or needs further optimization. The separate tracking makes token use a question that can be answered directly in the evaluation. Teams can compare the token result with the Efficiency result, then see whether token and execution efficiency improve or whether additional optimization is needed. This keeps token savings from being treated as automatic and preserves the distinction stated in the findings.
The tool's integration with various NVIDIA products and its ability to evaluate skills across 30+ NVIDIA products. That coverage keeps the evaluation connected to the products in which the skills are used. It also means the stated results can be considered alongside the product-focused finding that Skill Lift varies more across products than across harnesses. The integration and the 30+ product scope are therefore key parts of what the tool can track.
