SkillEvaluator

SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent
artifacts, starting with agent skills: deterministic quality gates, semantic
overlap detection, synthetic eval dataset generation, and live agent evaluation.

Agent skills are folders of instructions and supporting files that extend AI
agents, as defined by the Agent Skills specification.
SkillEvaluator is part of the
NVIDIA Verified Skills pipeline.

Three-tier overview

Tiers are independent entry points; nothing requires running earlier ones first.

SkillSpector provides specialized
security scanning for Tier 1 validation.
Harbor, the open-source agent
evaluation framework, powers the sandboxed agent runs in Tier 3 live
evaluation. Full tier guides live in the
documentation.

Quickstart

Install all SkillEvaluator evaluation extras with
uv, then run the built-in deterministic validation
gates. This first result needs no API key, Docker daemon, or repository clone:

uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator validate ./my-skill \
  --checks schema,pii,license,quality,unicode,lint \
  --no-dedup

./my-skill is any directory containing a SKILL.md. The command checks its
schema, PII, license, quality, Unicode safety, and scripts. The scoped check
list keeps this first run keyless; the complete Tier 1 security scan also uses
external tools described in the
installation guide.
If your shell cannot find the command after installation, run
uv tool update-shell and open a new terminal.

LLM provider setup

No OpenAI or Anthropic key yet? Create a free API key at
build.nvidia.com — NVIDIA Build offers free
inferencing, and NVIDIA Build defaults to the open-source Nemotron model
nvidia/nemotron-3-nano-30b-a3b for a quick try. Prefer a different model?
Pick any free model on build.nvidia.com and set
SKILL_EVAL_LLM_MODEL. Once that key is set, the same provider works
seamlessly across Tier 1 LLM checks, Tier 2, and Tier 3 (chat plus embeddings
with one credential):

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
skillevaluator models --limit 10

Other supported provider setups are:

• OpenAI: SKILL_EVAL_LLM_PROVIDER=openai and OPENAI_API_KEY.
• Anthropic: SKILL_EVAL_LLM_PROVIDER=anthropic and ANTHROPIC_API_KEY.
• Amazon Bedrock: SKILL_EVAL_LLM_PROVIDER=bedrock plus the standard AWS
  credential chain and region.
• Local or hosted OpenAI-compatible endpoint: set
  SKILL_EVAL_LLM_PROVIDER=openai-compatible,
  SKILL_EVAL_LLM_BASE_URL, SKILL_EVAL_LLM_MODEL, and
  SKILL_EVAL_LLM_API_KEY.

When exactly one of NVIDIA_API_KEY, OPENAI_API_KEY, or ANTHROPIC_API_KEY
is present, SkillEvaluator can auto-select that provider. Anthropic and Bedrock
do not provide embeddings, so Tier 2 also needs a separate OpenAI, NVIDIA
Build, or OpenAI-compatible embedding provider. See
Providers & Credentials
for model defaults, endpoint overrides, and fully local setup.

Run deeper evaluations

similarity-check needs an embeddings provider. context-optimization-check
also needs a chat provider to check one skill for repeated guidance:

skillevaluator context-optimization-check ./my-skill
skillevaluator similarity-check ./skills

Install Semgrep, SkillSpector, and Gitleaks before a full run; missing Tier 1
scanner evidence makes validation incomplete. Then verify the selected agent
runtime and use validate --full:

skillevaluator doctor --agents codex --env-mode docker
skillevaluator validate ./my-skill \
  --full \
  --agents codex \
  --env-mode docker

--full runs Tiers 1, 2, and 3 and enables autopilot. If the skill has no
accepted evaluation source, autopilot creates one initial case at
evals/evals.json; if the file already exists, SkillEvaluator reuses it. For a
broader four-bucket dataset, generate and review it first:

skillevaluator create-eval-dataset ./my-skill --full

Tier 2 needs chat and embedding providers. Tier 3 also needs the evaluator
provider, the selected agent's credential, and a Docker, local, or cloud
sandbox. Live model calls and managed sandboxes can incur charges; local mode
avoids managed sandbox charges, not hosted model charges. It is experimental
and only for trusted skills and workspaces; use Docker or cloud for untrusted
code. Start with one agent and a small dataset.
See the Tier 3 guide
before scaling a run. Tier 3 results are advisory within validate; Tier 1 and
Tier 2 determine its exit status.

Documentation

Read the complete documentation at
docs.nvidia.com/skills/skillevaluator
for installation, the quickstart, provider configuration, tier guides, results
and CI integration, the CLI reference, and contributor guidance.

Installation and third-party software

Follow the installation guide
to choose the full installation or a smaller per-tier setup.

This project will download and install additional third-party open source
software projects. Review the license terms of these open source projects before
use.

Contributing

Contributions are welcome. Read CONTRIBUTING.md, include tests
for behavior changes, and run the checks before opening a pull request:

make lint && make test && make build

Project governance is described in GOVERNANCE.md. Participation
is governed by the Code of Conduct.

Support

Support level: Experimental. SkillEvaluator is community-supported on a
best-effort basis with no SLA or NVIDIA enterprise support entitlement. Report
reproducible bugs and feature requests through
GitHub Issues; see
SUPPORT.md for details.

Security

Report suspected vulnerabilities using the private process in
SECURITY.md. Do not disclose security issues in a public GitHub
issue.

Releases

Release changes are recorded in CHANGELOG.md and
GitHub Releases.

License

Apache License 2.0 — see LICENSE, NOTICE, and
THIRD_PARTY_NOTICES.md.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.