tokens&
For enterprises
Submit
Sign in
tokens&

Build better AI stacks, claim useful opportunities, and give AI infrastructure companies a source-labeled adoption readout they can trust.

For buildersFor enterprises

Product

  • For builders
  • Category rankings
  • Startup credits and perks
  • Agent Skills
  • Platform
  • Submit project, tool, product, or perk

Enterprise

  • Start free company workspace

Community

  • Community
  • Newsletter
  • Events
Xin

© 2026 tokensand, LLC. All rights reserved.

  • Terms
  • Privacy
  • Security
  • Data Processing
  • Status
Agent Skills/NVIDIA RAG evals
NVIDIAModelsSKILL.mdVerified source

Agent Skill

NVIDIA RAG evals

Run RAGAS quality benchmarks against filesystem RAG corpora and deployed RAG endpoints.

Install this skillView repository

Vendor-authored source · Apache-2.0 AND CC-BY-4.0 license.

Raw SKILL.mdInstall the Tokens& Agent Pack

Skill specification

Declared by NVIDIA in the package front matter. Trigger conditions are what the coding agent matches on before it loads the skill.

View package fields
NVIDIA RAG evals SKILL.md front matter fields
Skill namerag-eval
Trigger conditionsFilesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
Allowed toolsRead Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Write Edit
CompatibilityRepository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/eval (eval deps live in scripts/eval/pyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai/mixtral-8x22b-instruct-v0.1).
Declared licenseApache-2.0
Version2.6.0

Install rag-eval

In a terminal with Node.js, npm and Git, run the command for your agent. The Skills CLI installs the complete package directory, including referenced files within it. Review its install prompt, then start a new agent session. A skill package does not set up an MCP server connection.

Claude Code

.claude/skills/rag-eval/SKILL.md

Project skills are committed with the repo. Use the user directory for a personal install across every project.

Project install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'claude-code'
Install for all projects instead

Personal install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'claude-code' --global

Codex

.agents/skills/rag-eval/SKILL.md

Codex reads `.agents/skills/` as its primary location, which is also the cross-platform default other clients honour.

Project install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'codex'
Install for all projects instead

Personal install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'codex' --global

Cursor

.agents/skills/rag-eval/SKILL.md

Cursor also loads `.agents/skills/`, `.claude/skills/`, and `.codex/skills/`, so one committed copy can serve several clients.

Project install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'cursor'
Install for all projects instead

Personal install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'cursor' --global

Gemini CLI

.agents/skills/rag-eval/SKILL.md

Gemini CLI reads `.agents/skills/` first when both directories exist.

Project install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'gemini-cli'
Install for all projects instead

Personal install

npx skills add 'https://github.com/NVIDIA/skills/tree/main/skills/rag-eval' --skill 'rag-eval' --agent 'gemini-cli' --global

SKILL.md

View raw source

Published by NVIDIA under Apache-2.0 AND CC-BY-4.0. Rendered from the package in github.com/NVIDIA/skills/tree/main/skills/rag-eval.

Read full skill instructions

On-disk RAG evaluation (corpus/ + train.json)

Purpose

Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).

For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.

When not to use

Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).

Prerequisites

  • Repo cloned; run commands from repo root (imports and paths assume this).
  • Python 3.11+ and uv; eval deps: uv sync --project scripts/eval.
  • Reachable RAG server and ingestor (defaults often localhost:8081 / 8082).
  • `NVIDIA_API_KEY` for RAGAS (see credential hygiene); optional `RAG_EVAL_JUDGE_MODEL`.
  • Dataset roots passed to --dataset-paths each contain `corpus/` and `train.json`.

Instructions

  1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in `references/dataset-and-conversion.md`. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there.
  2. Run eval — uv run --project scripts/eval python scripts/eval/evaluate_rag.py with --dataset-paths, --host, and --port. See `references/benchmark-execution.md` for command examples, outputs, and errors. Use `references/evaluate-rag-cli.md` for flag-level detail.
  3. Tune quality — Adjust --top_k / --vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in `references/benchmark-execution.md` when comparing retrieval/generation configs for RAGAS scores.
  4. Analyze results — Use `references/result-analysis.md` for scripts; scan rag_*_evaluation_summary.json for headline RAGAS metrics.

Examples

Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: `references/benchmark-execution.md#credential-hygiene-nvidia_api_key`.

Minimal eval (key already in environment):

uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081

Pretty-print summary JSON:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

More examples (skip ingestion, quality sweeps): `references/benchmark-execution.md`.

Limitations

  • Evaluator behavior is fixed to the filesystem contract and evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.
  • Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
  • Scores depend on retrieval quality, judge model availability, and NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references).
  • Large procedural detail lives under `references/` to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

Troubleshooting

Error / signalLikely causeWhat to do
Immediate exit mentioning NVIDIA_API_KEYMissing or invalid keySet key via secure channel; see credential hygiene in `references/benchmark-execution.md`.
train.json must be a JSON arrayWrong JSON shapeTop-level array of objects; validate per `references/dataset-and-conversion.md`.
Fewer rows in evaluation_data.json than train.jsonPer-query failuresCheck stderr: network or stream JSON errors; see error table in benchmark-execution.
Empty generated_contexts everywhereRetrieval gapVerify collection, ingestion, top_k / vdb_top_k, and ingestor_server_url without /v1 suffix.

Full signal table: `references/benchmark-execution.md#common-error-cases-and-signals`.

Gotchas

  • Run from repo root: paths and imports in scripts/eval/evaluate_rag.py assume this; a wrong directory silently breaks imports.
  • `--ingestor_server_url`: pass http://host:port without /v1—the code appends /v1/ automatically. Including /v1 causes 404s on ingestor calls.
  • Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. APP_VECTORSTORE_URL, embedding model).
  • `--model` / `--llm_endpoint`: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
  • Stale collections: a previous run's ingested data persists unless you use --force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.
  • Empty context metrics: if all generated_contexts are empty, RAGAS scores only nv_accuracy and leaves the other two metrics blank—this is not a silent success.

Source of truth

PieceLocation
Driverscripts/eval/evaluate_rag.py (CORPUS_DIRECTORY = corpus, EVAL_DATA = train.json)
Human README (always in-repo)scripts/eval/README.md
Full CLI (flags, defaults)scripts/eval/evaluate_rag.py --help; `references/evaluate-rag-cli.md`
Dataset / conversion`references/dataset-and-conversion.md`
Runs, outputs, errors`references/benchmark-execution.md`
Result analysis scripts`references/result-analysis.md`
Latency / throughputrag-perf skill, docs/performance-benchmarking.md

Agent playbook

  1. Run eval — uv sync --project scripts/eval then uv run --project scripts/eval python scripts/eval/evaluate_rag.py with required --dataset-paths, --host, and --port (and env NVIDIA_API_KEY). Argument --ingestor_server_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint.
  2. Quality tuning — See `references/benchmark-execution.md`: --top_k/--vdb_top_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens.
  3. Data conversion — Follow `references/dataset-and-conversion.md`.
  4. Analyze results — `references/result-analysis.md`; quick scan: python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json.
  5. Error triage — `references/benchmark-execution.md#common-error-cases-and-signals`.

More NVIDIA Agent Skills

All Agent Skills

AI-Q Blueprint deployment

Install, run, validate, troubleshoot, and stop a local or self-hosted NVIDIA AI-Q Blueprint environment.

Agents

CUDA-Q onboarding guide

Install CUDA-Q, validate simulators and hardware targets, and build reproducible quantum applications.

Models

cuOpt installation

Select and verify a compatible cuOpt Python, C, or REST server installation for an NVIDIA GPU environment.

Models

cuOpt numerical optimization API

Solve linear, mixed-integer, and quadratic programs with the cuOpt Python API and result diagnostics.

Models

cuOpt optimization formulation

Translate business constraints and objectives into verifiable cuOpt mathematical programs before implementation.

Models

cuOpt routing API for Python

Build vehicle-routing and fleet-optimization models with constraints, objectives, and solution validation.

Models

Triage errors — Use the error signal table and the Troubleshooting section below.
Ingestor 404 on upload
Bad ingestor base URL
Pass http://host:port only — code appends /v1/.