Eval Audit

Install and use Eval Audit

1. Install the Skill

   Run the upstream command in your project, then follow the author’s setup guide and any requirements below.

   npx skills add ai-evals-course/evals-skills --skill eval-audit

2. Give your agent a task

   Provide the relevant files, context and constraints. For example:

   Audit these evaluation datasets, judge instructions and reports for leakage, missing validation and stale checks.

Acerca de este Skill

Audit an LLM evaluation pipeline for missing error analysis, unvalidated judges, data leakage and stale checks.

Por qué lo elegimos

A focused audit of product evaluation artifacts, including judge validation against human labels.

Para qué sirve

• Review an existing LLM evaluation system and prioritize evidence-backed improvements.

How it works and limitations

The result is an audit and prioritized recommendations, not proof that a judge is calibrated. Other tasks in the repository cover building and validating evaluators.

Antes de usarlo

• Provide evaluation traces, judge prompts, labeled data or other pipeline artifacts; an observability integration is optional.