Eval Audit

Install and use Eval Audit

1. Install the Skill

   Run the upstream command in your project, then follow the author’s setup guide and any requirements below.

   npx skills add ai-evals-course/evals-skills --skill eval-audit

2. Give your agent a task

   Provide the relevant files, context and constraints. For example:

   Audit these evaluation datasets, judge instructions and reports for leakage, missing validation and stale checks.

Über diesen Skill

Audit an LLM evaluation pipeline for missing error analysis, unvalidated judges, data leakage and stale checks.

Warum wir es empfehlen

A focused audit of product evaluation artifacts, including judge validation against human labels.

Wobei es hilft

• Review an existing LLM evaluation system and prioritize evidence-backed improvements.

How it works and limitations

The result is an audit and prioritized recommendations, not proof that a judge is calibrated. Other tasks in the repository cover building and validating evaluators.

Vor der Nutzung

• Provide evaluation traces, judge prompts, labeled data or other pipeline artifacts; an observability integration is optional.