{"full_name":"NVIDIA/SkillEvaluator","owner":"NVIDIA","name":"SkillEvaluator","shard":"1","url":"https://github.com/NVIDIA/SkillEvaluator","description":"Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.","homepage":"https://docs.nvidia.com/skills/skillevaluator/","topics":["agentic-ai","evaluation","agent-evaluation","agent-security","skills","agent-skills","benchmark","claude-code","codex","evaluate","security-scanner","skill-eval","skill-evals","skill-evaluation","skill-evaluator"],"tag_pages":["agent-evaluation","agent-security","agent-skills","agentic-ai","benchmark","claude-code","codex","evaluation","security-scanner","skills"],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/1728152?v=4","created_at":"2026-06-24T02:32:26Z","pushed_at":"2026-11-16T09:23:37Z","metrics":{"stars":2090,"forks":286,"watchers":3,"issues":97,"prs":509,"releases":11,"issues_open":10,"issues_closed":87,"prs_open":15,"prs_closed":22,"prs_merged":472,"discussions":3},"growth":{"day":{"stars":804,"forks":132,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[771,976,1106,1286,2090],"forks":[97,115,128,154,286],"watchers":[0,0,1,1,3],"issues_open":[20,20,13,10,10],"prs_open":[47,31,3,15,15]},"boards":{"daily":18,"weekly":138,"daily_rate":4,"weekly_rate":3,"monthly_rate":21,"daily_forks":18,"daily_watchers":189},"same_owner":[{"full_name":"NVIDIA/DALI","name":"DALI","shard":"1","boards":{}},{"full_name":"NVIDIA/Isaac-GR00T","name":"Isaac-GR00T","shard":"1","boards":{}},{"full_name":"NVIDIA/Megatron-LM","name":"Megatron-LM","shard":"1","boards":{}},{"full_name":"NVIDIA/Model-Optimizer","name":"Model-Optimizer","shard":"1","boards":{}},{"full_name":"NVIDIA/NemoClaw","name":"NemoClaw","shard":"1","boards":{}},{"full_name":"NVIDIA/OpenShell","name":"OpenShell","shard":"1","boards":{"monthly":107,"monthly_rate":96}},{"full_name":"NVIDIA/SkillSpector","name":"SkillSpector","shard":"1","boards":{"daily":148,"weekly":170,"monthly":91,"monthly_rate":110}},{"full_name":"NVIDIA/TensorRT-LLM","name":"TensorRT-LLM","shard":"1","boards":{}},{"full_name":"NVIDIA/apex","name":"apex","shard":"1","boards":{}},{"full_name":"NVIDIA/cosmos","name":"cosmos","shard":"1","boards":{}},{"full_name":"NVIDIA/cuda-samples","name":"cuda-samples","shard":"1","boards":{}},{"full_name":"NVIDIA/cudf","name":"cudf","shard":"1","boards":{}},{"full_name":"NVIDIA/cudnn-frontend","name":"cudnn-frontend","shard":"1","boards":{}},{"full_name":"NVIDIA/cutlass","name":"cutlass","shard":"1","boards":{}},{"full_name":"NVIDIA/garak","name":"garak","shard":"1","boards":{}},{"full_name":"NVIDIA/nvidia-docker","name":"nvidia-docker","shard":"1","boards":{}},{"full_name":"NVIDIA/open-gpu-kernel-modules","name":"open-gpu-kernel-modules","shard":"1","boards":{}}],"issues":[{"number":462,"title":"[FEA]: Recognize Node scripts (.mjs, .js) for skill type and the script checks","comments":5,"created_at":"2026-11-12T04:11:52Z"},{"number":510,"title":"[BUG]: `nv_build` with no `llm-model` returns HTTP 404 for our NVIDIA Build key, so `RUBRIC_EVAL` and the LLM security verification do not run","comments":3,"created_at":"2026-11-13T17:32:56Z"},{"number":461,"title":"[FEA]: Name the Tier 1 token count an estimate, and offer a tokenizer or a set ratio","comments":3,"created_at":"2026-11-12T04:11:36Z"},{"number":495,"title":"[BUG]: Tier 1 security on a plugin scans only its bundled skills; hooks and bin/ are never scanned and skill references to them stay INCOMPLETE","comments":2,"created_at":"2026-11-13T09:39:04Z"},{"number":42,"title":"ux: add consolidated Tier 1 scanner preflight and guided bootstrap","comments":2,"created_at":"2026-08-05T09:51:32Z"},{"number":508,"title":"[FEA]: Letter-grade quality score for plugins, hooks and mods","comments":1,"created_at":"2026-11-13T15:33:30Z"},{"number":329,"title":"[FEA]: Move the rest of Tier 1 configs out of code and into YAML files","comments":1,"created_at":"2026-11-03T15:47:50Z"},{"number":41,"title":"roadmap: scale Tier 2 overlap detection for registry-sized catalogs","comments":1,"created_at":"2026-08-05T09:51:32Z"}],"discussions":[{"number":490,"title":"Field note: what the Tier 1 Discoverability score reads, and a question about Tier 3","comments":0,"category":"General","created_at":"2026-11-13T08:26:36Z"},{"number":205,"title":"Research collaboration inquiry: LLM judge calibration for Skill Lift","comments":0,"category":"General","created_at":"2026-10-16T18:36:33Z"},{"number":114,"title":"Support GitHub Copilot CLI as a Tier 3 evaluation agent","comments":0,"category":"General","created_at":"2026-08-28T12:11:47Z"}],"latest_release":{"name":"SkillEvaluator 0.9.0","tag":"v0.9.0","published_at":"2026-11-13T11:03:05Z","body":"# SkillEvaluator 0.9.0\n\nSkillEvaluator 0.9.0 expands how you build evaluation datasets, compare skill versions, and understand why an agent passed or failed. It adds controls for evaluation budgets, reusable baselines, and clearer evidence across terminal, HTML, JSON, Markdown, and benchmark reports.\n\nFor PyPI users upgrading from 0.7.2, this release also includes the changes shipped in the GitHub-only 0.8.0 release.\n\n## Highlights\n\n- **Build and review better evaluation datasets.** Declare deterministic checks for script execution, workspace files, and final responses. Generate datasets grounded in skill instructions, author intent, or traces; import production traces as redacted review candidates; and promote expectations only after explicit human review. Preview legacy dataset migrations before writing a new file.\n- **Track skill improvements over time.** Compare retained skill versions with `--compare-to` and `compare-versions`, browse run history, and reuse a compatible no-skill baseline with `--reuse-baseline`. Version comparisons remain separate from Skill Lift against a no-skill baseline.\n- **Control evaluation spending.** Set token, judge-call, attempt, and priced-cost budgets. Reports retain partial results when limits stop a run. Token usage is shown per case and agent, with missing counters and reused-baseline usage identified.\n- **Understand the evidence behind each result.** Reports show met, unmet, unchecked, and inapplicable expectations; dataset coverage and sufficiency; skill activation; failed-arm coverage; and concrete next steps. Skill Lift is the headline comparison, with pass@k shown as supporting reliability. HTML charts work offline.\n- **Audit graders and judge decisions.** Qualify the shipped verifier with `tier3 validate-grader`, inspect retained judge requests and replies, use opt-in judge retry and fallback policies, and ask questions about a completed run with `tier3 feedback`. Import externally computed custom metrics with their provenance.\n- **Extend repository CI.** Opt-in GitHub and GitLab setup adds advisory Tier 2 and Tier 3 evaluations alongside required Tier 1 checks. Capability-gated onboarding supports independent chat, judge, and agent models, reviewed scanner baselines, retained benchmark cards, and optional trusted feedback.\n\n## Notable fixes\n\n- Align Skill Lift, dimension applicability, verdicts, and publication status across report formats. Failed baselines are labeled incomplete rather than “not run,” and missing evidence stays visible.\n- Improve Tier 2 handling of large repeated-content clusters, malformed LLM verdicts, collection coverage, and shell-script chunking. Failed catalog rebuilds preserve the previously committed registry.\n- Reduce security and PII false positives while retaining real coverage gaps and scanner failures. Dependency advisories now appear as structured findings in reports and SARIF; stable finding IDs support tracking across runs.\n- Improve agent setup retries, cancellation, stop-on-pass concurrency, UTF-8 dataset loading, retained feedback, and Docker setup. Upgrade the Tier 3 backend to Harbor 0.25.0.\n\n## Upgrade considerations\n\n- **Dataset generation now defaults to `auto` grounding.** Provide substantive author guidance or usable matching traces. Skill-only starter flows need `--grounding skill`, including with `--no-llm`. Older datasets that require skill use may need outcome-only expectations to pass strict CI validation; the no-skill baseline receives the same checks.\n- **Some grading outcomes change.** Accuracy evaluates task outcomes without penalizing a no-skill baseline for not identifying the skill. Standard judge scores are computed from verdicts. A scored attempt fails when goal accuracy or security is below 1.0; a scored dimension below 0.40 fails the Tier 3 verdict even when another dimension has no score. Re-evaluate before comparing scores with older releases.\n- **Baseline timeout tolerance is opt-in.** `--tolerate-baseline-timeouts` appl","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}