Paired gates by default
Release gates now run McNemar's test against production by default, with Bonferroni correction across strict categories. Unpaired comparisons remain available for suites that can't share cases.
Changelog
What changed, and why.
Product and research updates, newest first.
Research note: eval sample sizes
Published our analysis of Wilson intervals, power and false-alarm rates for LLM release gates, with reproducible code.
Judge-corrected failure rates
Evals reports now show Rogan–Gladen-corrected failure rates alongside raw judge rates, with intervals that include calibration uncertainty.
pass^k in agent reports
Agent suites report pass^k with unbiased estimates and intervals; k defaults to 4 and is configurable per suite.
MCP definition pinning in Guard
Guard hashes approved MCP tool definitions and blocks calls when a definition changes until it is re-approved.
Per-channel thresholds
Guard policies accept thresholds per input channel and action class, with a cost-ratio helper for choosing them.
Hedged checks
Guard re-issues slow classifier checks at their p95 latency, reducing tail latency for multi-check policies.
Know the number before you ship.
A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.