What are AI evals? A complete guide to LLM evals: types, components and how to build one

AI evals (short for evaluations) are repeatable measurements of how an AI model or AI-powered system behaves: a fixed set of inputs, a harness that runs the system on them, a grader that scores each output, and a metric that aggregates the scores into a number with an error bar. LLM evals are the same idea applied to language models and the applications built on them. Teams use evals to compare models, catch regressions before release, monitor production and show that a system is safe enough to ship.

This pillar page covers the whole topic: what an eval is and is not, the main types (capability, safety and product evals), the four components every eval has, where measurement error comes from, how offline and online evals differ, and a step-by-step recipe for building one. Each section links to a deeper article. Two simulations, one of an eval's error budget and one of drift in production, supply the numbers.

What is an AI eval? A concrete example#

A team ships an assistant that handles refund requests. Its policy: refunds up to a set limit are approved automatically, larger ones need a human, and refunds outside the return window are declined with an explanation. The team writes 400 test conversations covering the policy's branches, including tricky ones (a request just over the limit, a customer who insists, a request with a forged order number in an attachment). For each conversation the harness runs the assistant in a sandbox, records what it said and which tools it called, and a grader checks two things: did it take the right action, and did its explanation match the policy?

The first run passes 348 of 400 cases: 87.0%. With 400 cases, the standard error is $\sqrt{0.87\times 0.13/400}\approx 1.7$ points, so the 95% interval is roughly 83.7% to 90.3%. Broken down by category, almost all failures are in "requests just over the limit", where the assistant approves instead of escalating. The team fixes the prompt and reruns: that category improves, and a check on the other categories confirms nothing else got worse. That loop, measure, diagnose, change, re-measure, is what evals are for.

Three things make this an eval rather than a demo. The inputs are fixed and versioned, so the next run measures the same thing. The grading criteria are explicit, so two people would score an output the same way. And the result is a number with uncertainty, so the team can tell a real change from noise.

Evals vs benchmarks vs tests#

The words overlap, so it helps to separate them. A benchmark is a public, shared eval used to compare models on a general capability, such as MMLU for knowledge or SWE-bench for software engineering; our guide to LLM benchmarks covers what they measure and how to read their leaderboards. A software test checks deterministic behaviour and passes or fails; an eval scores stochastic behaviour and reports a rate. An eval, in the sense used on this page, is broader than both: it can be public or private, about a model or a whole system, run once or continuously. Benchmarks tell you which model to shortlist. Your own evals tell you whether the system you built does the right thing.

Why do AI systems need evals?#

Traditional software is tested by asserting exact outputs. That does not work for language models, for four reasons. Outputs are open-ended, so many different answers can be correct. Behaviour is stochastic, so the same input can produce different outputs. Small changes have non-local effects: a prompt edit that fixes one category can break another. And many failures are rare, so a handful of manual checks will not find them. Evals answer these with volume (many cases), explicit criteria (graders), statistics (rates with intervals) and repetition (every change, every release).

Evals also do work beyond engineering. They are how safety claims are substantiated: a statement that a system "refuses harmful requests" means little without a harmful-compliance rate on a defined set. Regulatory frameworks increasingly expect documented testing; see our pages on EU AI Act testing requirements and the NIST AI RMF Measure function. And evals sit inside training itself: a reward model is an eval used as an optimisation target, which is why reward hacking is, at heart, an eval failure.

Types of AI evals#

Evals can be grouped by what they measure, what they are run on, and how they are graded. The first grouping is the one most teams start from.

Capability evals#

Capability evals ask what a model can do: answer graduate-level science questions, solve competition maths, fix real bugs, use tools. Public benchmarks are the familiar form, and private capability evals built for a specific domain (clinical reasoning, legal drafting, a company's codebase) are often more informative. Capability evals need effort in elicitation: the score depends on prompting, examples and scaffolding, and a low score under a weak prompt does not prove absence. That matters most for dangerous capability evaluations, where the question is whether a model could help with serious harm if someone tried hard, and where Shevlane et al. argue developers must look for both dangerous capabilities and the propensity to use them.

Safety evals#

Safety evals measure behaviour that should or should not happen regardless of capability: complying with harmful requests, refusing benign ones (over-refusal), leaking private data, following instructions injected through documents, taking unauthorised actions. Two features distinguish them. Their failure rates are usually small, so they need large samples and careful statistics. And they face adaptive adversaries, so a fixed set of attack prompts measures resistance to yesterday's attacks; that is why safety evals are paired with red teaming, compared in detail in AI red teaming vs evals. Public suites are surveyed in AI safety benchmarks, and the wider field in what AI safety is.

Product and system evals#

Product evals ask whether a specific application does its job for its users under its policies: task success, correctness against a knowledge base, faithfulness to retrieved sources, tone, format, latency and cost. They run on the whole system (prompts, retrieval, tools, guardrails and post-processing), not the bare model, because that is what users meet. They are usually the most valuable evals a team owns and the least portable. Our LLM evaluation guide covers building them for production.

By subject and by grading method#

The same eval logic applies to different subjects: a bare model, a full application, a multi-step agent whose whole trajectory is graded, or a policy trained by reinforcement learning, where the randomness of training itself has to be measured (see how to evaluate RL agents). Graders come in four kinds, often combined in one suite:

The guide to LLM-as-a-judge and the comparison of model judges and human evaluation go deeper on the last two rows.

The four components of an eval#

Formally, with a system $f$, cases $x_1,\dots ,x_n$, a grader $g$ that maps an input and output to a score, and an aggregation, the simplest eval estimates

where $\hat{\theta }$ is the reported score, for example a pass rate. Everything that can go wrong with an eval is a problem with one of the four pieces in that formula.

Dataset#

The dataset defines what the eval is about. Each case holds an input plus whatever context the system needs (conversation history, documents, user attributes, tool state), a reference answer or grading notes where possible, and metadata: category, severity, source and the policy version it tests. Good datasets mix production samples (so the suite reflects real use), incidents and near misses, targeted synthetic or adversarial cases for rare situations, and expert-written gold cases for high-stakes domains. Ribeiro et al.'s CheckList offers a useful design vocabulary: minimum-functionality tests for basic capabilities, invariance tests (the answer should not change when an irrelevant detail does) and directional tests (it should change in a known way when a relevant detail does). Raji et al. warn against the opposite failure: a dataset presented as measuring a general ability it cannot represent.

Two rules prevent most dataset problems. Keep a held-out slice that the people tuning the system never see, or the suite becomes a training set. And never let eval items reach training data, or the score measures memory; our research on contamination detection shows how to check.

Harness#

The harness runs the system on each case exactly as it would run in production: same prompt templates, retrieval, tools (sandboxed), guardrails, decoding settings and post-processing. For agents it includes the environment and often a simulated user. The harness should record everything (inputs, outputs, tool calls, latencies, seeds) so failures can be diagnosed and runs reproduced. Small harness choices move scores: answer extraction, prompt formatting and few-shot examples can shift a benchmark by several points, which is why a score is only meaningful together with its harness configuration.

Grader#

The grader turns an output into a score. Prefer the cheapest grader that is accurate for the criterion: rules and execution where answers are checkable, a calibrated model judge for open-ended criteria, humans for the subtle and high-stakes. A model judge is a measuring instrument with its own error rates and biases (Zheng et al. documented position and verbosity preferences when introducing MT-Bench), and at low failure rates its false positives can swamp the real failures; our note on judge error shows how to measure and correct for that, and the one on rubric evals shows how rubric design changes reliability.

Metric#

The metric aggregates graded outputs: pass rate, accuracy, F1, pass@k, calibration error, win rate, and so on. Choosing it is a decision about what you value, and each metric has blind spots. LLM evaluation metrics defines the common ones and shows where each misleads. Two rules apply everywhere: report every metric with an error bar, and pair each safety metric with a helpfulness metric, or the easiest way to improve it will be to refuse more.

Where does eval error come from?#

An eval score is an estimate, and it has several independent sources of error. Thinking about them as a budget tells you where to spend effort.

• Question sampling. Your cases are a sample of possible cases. Another equally valid set would give a different score.
• Generation randomness. At nonzero temperature, a rerun on the same cases gives different outputs.
• Grader error. The grader mislabels some outputs. Random mislabelling adds noise; systematic mislabelling adds bias that error bars do not show.
• Harness and prompt choices. Formatting and extraction choices shift scores, often by more than the effect being measured.
• Contamination and overfitting. If the system has seen the cases, the score overstates performance on new ones.

For a pass/fail eval, the first two combine by the law of total variance. If case $i$ has true pass probability $p_i$ and you sample $k$ outputs per case,

where the first term is how much cases differ in difficulty and the second is generation noise. More cases shrink both terms; more samples per case shrink only the second.

Simulation. We modelled a system whose per-case pass probabilities follow a Beta(2.4, 0.6) distribution (mean 80%, with a long tail of hard cases), graded by a judge that catches 92% of real failures and wrongly fails 5% of real passes. We reran the whole eval 4,000 times at each size.

The bias deserves emphasis. With the judge's error rates fixed, the reported rate is $0.8\times 0.95+0.2\times 0.08=77.6\%$ whatever the sample size. A constant bias cancels when comparing two systems graded by the same judge, but only if the judge's errors do not depend on the system, which is exactly what fails when one system writes longer answers or a different style. The practical recipe, covered in statistical significance for evals, is to compare systems on the same cases with paired tests, sample several outputs per case when generation noise dominates, and calibrate the grader against human labels. Our notes on eval sample size and clustered standard errors give the sizing formulas.

Offline vs online evals#

An offline eval runs a fixed, versioned suite against a candidate system before release. It answers "did this change make things better or worse?" and is the basis of release gates and CI checks. An online eval samples live production traffic, grades it with the same graders, and tracks the rate over time. It answers "is the system still behaving well on what users actually send?"

The two fail differently, and the difference is easy to underestimate. If an upstream change (a provider model update, a retrieval index rebuild, a new customer segment) lowers quality without any change to your code, every offline run still passes, because the offline suite and the system it tests are unchanged. Only the online sample sees it.

Simulation. Ninety days of production with a true pass rate of 92%. On day 46 an upstream change drops it by 3 points. Each day we grade $k$ randomly sampled production outputs. The monitor takes days 1 to 14 as a baseline and, from then on, alarms when the 7-day rolling pass rate falls below the baseline by more than 2.326 standard errors (a one-sided 1% test per check). We ran 2,000 replications per setting.

At 100 graded samples a day, a 3-point drop was caught in a median of 7 days; a 1-point drop was caught in only 52% of runs within 45 days, a 5-point drop in a median of 4 days. Note the last column: a daily check at a 1% false-alarm rate, repeated for a month, raised at least one false alarm in roughly one run in eight. Repeated checks are multiple comparisons in time, and an online monitor needs either a stricter per-check threshold or a sequential method designed for continuous monitoring. In practice, run both kinds of eval: offline to gate what you change, online to catch what changes around you, and promote every online failure into the offline suite.

How to build an AI eval, step by step#

1. Start from decisions and risks. Write down what the eval will decide (ship or not, which model, whether a fix worked) and what failures would hurt: wrong actions, harmful content, data leaks, policy violations, bad answers. A short threat-modelling session with product, security and domain experts produces a better list than any template.
2. Define categories with owners. Organise by failure mode ("refunds above the limit"), not by data source. Give each category a severity and an owner.
3. Write grading criteria before collecting data. For each category, state what a pass is in terms two reviewers would apply identically. If you cannot, the category is not ready.
4. Assemble cases. Mix production samples, incidents, targeted synthetic cases and expert-written gold cases. Start small (a few dozen per category to find obvious failures) and grow towards the size your decisions need.
5. Build the harness on the real system. Same prompts, tools, retrieval and settings as production, sandboxed, with full logging and fixed seeds where possible.
6. Choose and calibrate graders. Use rules where you can. For model judges, label a calibration sample with experts and measure agreement per category before trusting the judge.
7. Size and baseline. Run the current system, compute per-category rates with intervals, and work out how many cases each category needs to detect the change you care about.
8. Wire it in and close the loop. Fast subset on every change, full suite before release with statistical gates, online sampling in production, and every failure from production or red teaming added as a permanent case.

The LLM eval plan template turns these steps into a document you can fill in.

A minimal eval harness#

import json, random, statistics
from dataclasses import dataclass

@dataclass
class Case:
    id: str
    category: str
    input: str
    expected_action: str          # e.g. "approve", "escalate", "decline"

def run_system(case, seed):
    """Stand-in for the real system: call your app with the production config here."""
    rng = random.Random(f"{case.id}-{seed}")
    return {"action": rng.choice(["approve", "escalate", "decline"]), "text": "..."}

def grade(case, output):
    return 1.0 if output["action"] == case.expected_action else 0.0

def evaluate(cases, samples_per_case=3):
    by_cat = {}
    for c in cases:
        score = statistics.mean(grade(c, run_system(c, s)) for s in range(samples_per_case))
        by_cat.setdefault(c.category, []).append(score)
    report = {}
    for cat, s in by_cat.items():
        m = statistics.mean(s)
        se = statistics.stdev(s) / len(s) ** 0.5 if len(s) > 1 else float("nan")
        report[cat] = {"n": len(s), "pass_rate": round(m, 3), "ci95": (round(m - 1.96 * se, 3), round(m + 1.96 * se, 3))}
    return report

random.seed(0)
cases = [Case(f"c{i}", random.choice(["under_limit", "over_limit", "late"]), "...",
              random.choice(["approve", "escalate", "decline"])) for i in range(300)]
print(json.dumps(evaluate(cases), indent=1))

The structure is what matters: cases with categories and expectations, a harness that calls the real system, a grader per criterion, and a report per category with intervals. Everything else (model judges, parallelism, dashboards) is an extension of these four functions.

Evals for agents and RL-trained systems#

Agents add three difficulties. Success depends on a whole trajectory, so graders should check the final environment state and the actions taken on the way, not just the last message; our research on agent trajectory evals covers trajectory scoring. Outcomes are more variable, so the same task must be attempted several times; whether you report pass@k (succeeds at least once) or pass^k (succeeds every time, the metric τ-bench introduced for agent reliability) changes the story, as the note on pass@k vs pass^k shows. And agents act, so a correct final answer reached through a forbidden tool call is a failure. Building good environments for this is its own subject, covered in RL environments for agents.

When the system was trained with reinforcement learning, evaluation also has to account for training randomness: two runs of the same recipe can differ by more than the effect being claimed. How to evaluate RL agents covers seeds, outlier-resistant aggregates such as the interquartile mean, and bootstrap intervals. Evals also guard training: any automated reward is itself an eval, and an optimiser will find its blind spots, the dynamic described in reward hacking and specification gaming. Held-out evals that the training process never sees are how those failures are caught.

Evals and red teaming#

Evals measure rates on inputs you chose in advance. Red teaming searches for inputs you did not think of, adaptively, and reports what it found. Each covers the other's blind spot: a fixed eval never contains the attack an adversary invents next week, and a red team's findings are not a rate of anything users will see. The working arrangement is a loop in which evals establish baselines, red teams target the gaps, and every finding becomes a regression case. AI red teaming vs evals lays out the comparison with a simulation, and our red teaming methodology covers running an engagement.

Common ways evals go wrong#

Where to go next#

The articles in this topic each take one part of the picture further:

• LLM benchmarks: what MMLU, GPQA, SWE-bench, Chatbot Arena and others measure, how they saturate and leak, and how to read a leaderboard.
• LLM evaluation metrics: accuracy, F1, exact match, pass@k, calibration error, win rate, BLEU and ROUGE, and when to use which.
• Statistical significance for evals: paired tests, bootstrap, multiple comparisons and power.
• How to evaluate RL agents: seeds, interquartile mean, stratified bootstrap and performance profiles.
• AI red teaming vs evals: goals, methods and metrics compared, and how to combine them.

For broader surveys, Chang et al. review LLM evaluation across tasks and methods, HELM shows what a multi-metric evaluation of many models looks like in practice, and Dynabench describes evaluation datasets that keep growing as models improve. Miller's Adding Error Bars to Evals is a short, practical treatment of the statistics.

Frequently asked questions#

Sources#

1. P. Liang et al.. Holistic Evaluation of Language Models, 2022. arxiv.org
2. Y. Chang et al.. A Survey on Evaluation of Large Language Models, 2023. arxiv.org
3. M. T. Ribeiro, T. Wu, C. Guestrin, S. Singh. Beyond Accuracy: Behavioral Testing of NLP models with CheckList, 2020. arxiv.org
4. I. D. Raji, E. M. Bender, A. Paullada, E. Denton, A. Hanna. AI and the Everything in the Whole Wide World Benchmark, 2021. arxiv.org
5. D. Kiela et al.. Dynabench: Rethinking Benchmarking in NLP, 2021. arxiv.org
6. T. Shevlane et al.. Model evaluation for extreme risks, 2023. arxiv.org
7. L. Zheng et al.. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. arxiv.org
8. E. Miller. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, 2024. arxiv.org
9. S. Yao, N. Shinn, P. Razavi, K. Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, 2024. arxiv.org