An eval suite is a versioned collection of test cases, grading methods and thresholds that together measure how an AI system behaves. It plays the role a test suite plays in software: it runs on every meaningful change and tells you whether the system still meets its requirements.
What goes in one
- Test cases: inputs drawn from real usage, known failure modes, red-team findings and policy edge cases, each tagged with a category and severity.
- Graders: exact match or code execution where possible, rubric-based LLM judges where not, and human review for categories where automated grading has not been calibrated.
- Metrics: pass rate per category, harmful compliance and over-refusal rates, faithfulness, latency, and confidence intervals for each.
- Thresholds and gates: the numbers a release must meet, wired into CI as regression checks.
- Version control: cases, grader prompts, grader models and thresholds all pinned, so a score change can be traced to its cause.
How big should it be?
Size each category for the precision you need. The 95% margin of error on a pass rate is about . With and items, that is roughly points, so a 400-item category can detect drops of about 4 points or more between versions with a paired test, but not 1-point changes. Rare, severe categories need targeted cases rather than random samples, because random traffic will contain almost none of them.
Offline and online
The same graders should run offline on the fixed suite and on a sample of production traffic (online evaluation). Failures found online become new offline cases, so the suite grows toward what users actually send.
Why it matters
Public benchmarks say how a model does on someone else's task. An eval suite says whether your system, with its prompts, tools, retrieval and guardrails, does what your policy requires. It is the evidence behind a ship decision and the record that shows an auditor what was tested.