Online evaluation is the continuous scoring of a sample of real production traffic, using the same graders and rubrics as the offline eval suite. It answers a question offline tests cannot: how the deployed system behaves on what users actually send today.
How it works
- Sample: take a random fraction of conversations, plus targeted samples (for example, every response a guardrail blocked or nearly blocked, or every high-impact tool call).
- Grade: run calibrated LLM judges and rule-based checks asynchronously, so scoring adds no user-facing latency.
- Track: plot rates per category over time with confidence intervals and alert on significant changes.
- Feed back: route novel failures to human review and add confirmed ones to the offline suite.
Worked example: can you see the drift?
Illustrative numbers: a system handles 200,000 conversations a day and grades a 1% sample, 2,000 conversations. The baseline failure rate is 2%, so the daily standard error is points. A jump to 3% is about 3.2 standard errors above baseline and would be caught within a day. A jump to 2.3% would not; detecting it needs a week of pooled samples or a larger sampling rate for that category.
How it differs from offline evaluation
- There is no fixed answer key, so graders must work reference-free or against retrieved sources.
- The input distribution shifts with product changes, seasons and attackers, so a stable score can hide a changing mix.
- Privacy rules apply: sampling, retention and human review must follow the data policy, and some deployments can only log decisions and metadata.
Why it matters
Every offline suite is a guess about future traffic. Online evaluation is how you find out where the guess was wrong: new jailbreak styles, topics the suite never covered, rising hallucination on a new document set, or a guardrail whose false-positive rate crept up. It closes the loop between deployment and the regression gates that protect the next release.