Install and use Eval Audit
Install the Skill
Run the upstream command in your project, then follow the author’s setup guide and any requirements below.
npx skills add ai-evals-course/evals-skills --skill eval-auditGive your agent a task
Provide the relevant files, context and constraints. For example:
Audit these evaluation datasets, judge instructions and reports for leakage, missing validation and stale checks.
About this Skill
Audit an LLM evaluation pipeline for missing error analysis, unvalidated judges, data leakage and stale checks.
Why we picked it
A focused audit of product evaluation artifacts, including judge validation against human labels.
What it helps with
- Review an existing LLM evaluation system and prioritize evidence-backed improvements.
How it works and limitations
The result is an audit and prioritized recommendations, not proof that a judge is calibrated. Other tasks in the repository cover building and validating evaluators.
Before you use it
- Provide evaluation traces, judge prompts, labeled data or other pipeline artifacts; an observability integration is optional.
Sources & references
Original sources are linked below. Practical starting points are editorial suggestions from SkillHey.