Eval Audit

Install and use Eval Audit

1. Install the Skill

   Run the upstream command in your project, then follow the author’s setup guide and any requirements below.

   npx skills add ai-evals-course/evals-skills --skill eval-audit

2. Give your agent a task

   Provide the relevant files, context and constraints. For example:

   Audit these evaluation datasets, judge instructions and reports for leakage, missing validation and stale checks.

このスキルについて

Audit an LLM evaluation pipeline for missing error analysis, unvalidated judges, data leakage and stale checks.

おすすめする理由

A focused audit of product evaluation artifacts, including judge validation against human labels.

役立つこと

• Review an existing LLM evaluation system and prioritize evidence-backed improvements.

How it works and limitations

The result is an audit and prioritized recommendations, not proof that a judge is calibrated. Other tasks in the repository cover building and validating evaluators.

使う前に

• Provide evaluation traces, judge prompts, labeled data or other pipeline artifacts; an observability integration is optional.