The template#
eval-plan.mdmarkdown
# Evaluation plan: [system name] v[version]
Owner: [name, team] Approver: [risk/security owner] Date: [yyyy-mm-dd]
## 1. System under test
- Purpose and users: [...]
- Models: [base model(s), versions, decoding settings]
- Tools / permissions: [tool → scope → max impact]
- Data reached: [sources, sensitivity, tenancy]
## 2. Unacceptable outcomes (threat model)
| ID | Outcome (business terms) | Actor | Severity |
|----|--------------------------|-------|----------|
| U1 | [e.g. refund paid to wrong party] | [external sender] | Critical |
## 3. Eval categories
| Cat | Linked outcome | Metric | Grader | Baseline | Min detectable Δ | n required | Gate |
|-----|----------------|--------|--------|----------|------------------|------------|------|
| C1 | U1 | pass rate | action assertion | [..] | [..] | [..] | block |
## 4. Data
- Sources: production sample [%], incidents [#], adversarial generation [method], expert gold [#]
- Held-out slice not visible to prompt engineers: [yes/no, size]
- PII handling: [...]
## 5. Graders
- Per category: type, validation result (Se, Sp, flip rate), human-review routing
## 6. Statistics
- Test: paired (McNemar) vs. production; seeds per case: [k]
- Multiple comparisons: [Bonferroni over strict gates / FDR for warnings]
## 7. Gates and cadence
- Pre-merge smoke: [n, time budget]
- Pre-release full: [n, seeds]
- Production online eval: [sampling rate, alert thresholds]
## 8. Ownership and escalation
- Category owners, on-call, who can override a gate and how it is loggedWorked example: refund-capable support agent#
| Cat | Outcome | Grader | Baseline | Min Δ | n | Gate |
|---|---|---|---|---|---|---|
| C1 Refund limit | Refund > $500 without approval | Action assertion | 99.6% | 1 pt | 1,200 | Block |
| C2 Indirect injection | Tool call triggered by email content | Action assertion + provenance | 97.1% | 2 pts | 1,700 | Block |
| C3 PII egress | Other customer's data in reply | PII grader (Se 0.96, Sp 0.995) | 99.6% | 1 pt | 1,500 | Block |
| C4 Policy adherence | Wrong refund decision | Rubric judge, expert-validated | 88.4% | 3 pts | 1,000 | Warn |
| C5 Over-refusal | Eligible refund refused | Rubric judge | 92.0% | 3 pts | 800 | Warn |
Sample sizes come from the power calculation in our sample-size note, adjusted for three strict gates (Bonferroni). For C1 and C3, at baselines above 99%, the binomial variance is small, so a 1-point change is detectable with fewer cases than near 95%. C4 and C5 are warnings because their judge was validated only to within 3 points of the expert ceiling.
Where plans usually go wrong#
- Categories named after data sources rather than outcomes. Nobody can own "batch 7".
- No minimum detectable Δ. Without it, nobody knows whether 200 cases is enough, and it usually isn't.
- Unvalidated judges in blocking gates. Use deterministic assertions for blocking where possible.
- Safety without helpfulness. Pair every safety category with an over-refusal category, or the cheapest fix will be refusing more.
- No override log. Gates get bypassed under deadline pressure; record who, when and why.