LLM evaluation: the complete guide to evaluating LLMs and agents

What LLM evaluation actually means

LLM evaluation is the practice of running a model-based system against a fixed, versioned set of inputs, scoring every output against explicit criteria, and comparing those scores across versions. The thing being evaluated is rarely just a model. In production it is a system: a prompt template, a retrieval pipeline, a set of tools, guardrails, post-processing and, increasingly, a multi-step agent loop.

That distinction matters because the two kinds of evaluation answer different questions. Model evaluation asks how capable a model is in general, which is the job of public benchmarks such as MMLU-style knowledge tests or coding suites. System evaluation asks whether your application does the right thing for your users under your policies. A model can top a leaderboard and still approve a refund it shouldn't, cite a document that doesn't support the claim, or follow an instruction hidden in an attachment. Only system evaluation catches those.

A benchmark tells you which engine to buy. An eval suite tells you whether the car you built is safe to drive.

The anatomy of an eval suite

Every useful eval suite has four parts, and they should be versioned together so that a score from last quarter means the same thing today.

1. Cases. Inputs plus whatever context the system needs: conversation history, retrieved documents, user attributes, tool state. For agents, a case is often a whole scenario with a simulated user and a sandboxed environment.
2. Graders. Functions that turn an output into a score. They range from exact-match and schema checks to model-based judges and human review.
3. Thresholds. What score counts as acceptable for each category, and how much regression you will tolerate between versions.
4. Metadata. Category, severity, source (synthetic, production incident, expert-written), owner and the policy version each case tests.

Organise suites by failure mode, not by data source. "Refunds above the approval limit" is a category someone can own and fix. "Batch 7 from the contractor" is not.

Choosing what to measure

Start from what would hurt. A short threat-modelling session with product, security and legal produces a better metric list than any template. For most production systems, categories fall into five groups:

Two rules keep metric lists honest. First, every safety metric needs a paired helpfulness metric, otherwise the easiest way to improve it is to refuse more. Second, prefer metrics that map to a decision. If no score on a metric would change whether you ship, delete it.

Building eval datasets that don't lie

Eval datasets fail in predictable ways: too small, too easy, too similar to the prompt engineer's own examples, or contaminated by the data the model was tuned on. A durable dataset mixes four sources:

• Production samples, stratified by intent and user segment, with personal data removed. This keeps the suite anchored to what users actually ask.
• Incidents and near misses. Every bug report, escalation and red-team finding becomes a permanent case. This is the highest-value data you will ever have.
• Synthetic and adversarial generation. Attacker models and templated perturbations expand coverage of rare and hostile inputs. Always have a human review a sample of what was generated.
• Expert-written gold cases for high-stakes domains, with reference answers and rationales. These also become the ground truth used to calibrate model graders.

Hold out a slice that prompt engineers never see. Once people tune against every case, the suite turns into a training set and stops being a test.

LLM-as-a-judge: when to trust it

Model-based graders make evaluation scale, and they are also the most common source of false confidence. A judge prompt that looks reasonable can systematically prefer longer answers, its own model family's style, or whichever answer comes first.

Treat every judge as an instrument that needs calibration:

1. Label a sample of 200–500 outputs per category with domain experts, using the same rubric the judge will use.
2. Measure agreement (Cohen's κ or Krippendorff's α) between the judge and the experts, and between experts themselves. Human–human agreement is your ceiling.
3. Use the judge only where its agreement is close to that ceiling. In our grader reliability research, judges reached κ above 0.9 for PII and policy-violation detection but fell below 0.5 for subtle medical harm and long-horizon agent plans.
4. Route low-confidence or high-severity cases to human review automatically, and feed the disagreements back into the next judge version.

Evaluating agents, not just answers

An agent's final message can be perfect while the path to it was a disaster: it emailed the wrong customer, queried a table it shouldn't have, or burned forty tool calls in a loop. Agent evaluation therefore scores the trajectory.

• Action assertions: deterministic checks on tool calls and arguments, such as "no refund over $500 without an approval step" or "only email the address on the ticket".
• State checks: inspect the sandboxed environment after the run. Was the record updated correctly? Was anything deleted?
• Process quality: number of steps, redundant calls, recovery from tool errors.
• Simulated users: a model plays the customer, including impatient, confused and adversarial personas, so multi-turn behaviour is exercised.

Because agents are non-deterministic, run each scenario with several seeds and report pass rates with confidence intervals rather than a single pass/fail.

Statistics: separating regressions from noise

If your suite has 200 cases and the pass rate moves from 92% to 90%, nothing may have changed at all. With a binomial pass rate, the 95% confidence interval around 92% on 200 cases is roughly ±3.8 points. Teams that ignore this either chase phantom regressions or wave real ones through.

Three habits fix most of it:

• Size categories for the effect you care about. To detect a 2-point drop from a 95% pass rate with 95% confidence and 80% power, you need roughly 1,700–2,200 cases per category for an unpaired comparison; paired designs need fewer.
• Use paired comparisons. Run old and new versions on the same cases and test the per-case differences (McNemar's test for pass/fail). Paired tests need far fewer cases than comparing two independent averages.
• Run multiple seeds at the temperature you use in production, and report variance alongside the mean.

Wiring evals into CI and production

An eval suite that runs when someone remembers is a report. One that runs on every change is a control. The mature setup looks like this:

1. Pre-merge: a fast smoke suite (a few hundred cases, a couple of minutes) on every pull request that touches prompts, tools, retrieval or model configuration.
2. Pre-release: the full release suite with multiple seeds and statistical gates. A significant regression in any high-severity category blocks the release; lower-severity regressions require a named approver.
3. In production: online evaluation samples live traffic, scores it with the same graders and alerts on drift. Anything that fails is promoted into the offline suite.

The loop matters more than any single score. Every failure found in production, in red teaming or by a customer should become a test that runs forever. That is how a suite gets better each month instead of going stale.

A 30-day plan

1. Week 1: threat-model one use case; agree on 6–10 categories with owners and severity.
2. Week 2: assemble cases from production samples, incidents and expert-written gold; build graders; label a calibration sample.
3. Week 3: baseline the current system; check judge agreement; size categories; set thresholds.
4. Week 4: wire the smoke suite into CI and the release suite into your deployment pipeline; enable online sampling.

That is exactly the scope of our 30-day pilot, if you'd rather not do it alone.

Frequently asked questions

Related: Reinforced Evals · Evals vs. guardrails · AI red teaming

Keep reading