Guardrail classifiers under distribution shift

A guardrail classifier is validated once and then runs for months against traffic that keeps changing: new product features bring new benign topics, attackers change style, and the share of harmful requests moves with publicity and seasons. This note separates the kinds of shift that matter, shows with a seeded simulation how each one moves precision and recall at a fixed threshold, and works through the two tools teams actually have: small labelled audits and label-free monitoring signals. The headline results are that base-rate drift alone is fixable without labels, that a 500-item audit cannot reliably reset a recall target, and that some drift cannot be fixed by moving the threshold at all.

Six months in the life of a guardrail#

Picture an input classifier in front of a customer assistant for a health insurer. At launch it was tuned on a labelled sample to catch 90% of harmful requests (jailbreaks, requests for other members' data, self-harm content that needs routing), and it did. Over the next six months three things happen at once:

• A new feature lets members ask about prescription coverage. Questions about controlled medications and dosages are benign, but the classifier was never shown them and scores them as risky.
• Attackers move from blunt instruction overrides to multi-paragraph role-play framings. Many of these score well below the threshold.
• A news story about chatbot jailbreaks lifts the share of harmful traffic from 2% to 5%.

Nobody retrains the model. The dashboard shows the flag rate creeping up, which looks like the guardrail doing more work. What actually happened is the subject of this note.

Three kinds of shift, in score space#

With $x$ a request and $y\in \{0,1\}$ its true label (1 for harmful), the textbook categories are:

• Label shift: $p(y)$ changes while $p(x\mid y)$ stays fixed. Harmful requests look the same; there are just more of them.
• Covariate shift: $p(x)$ changes while $p(y\mid x)$ stays fixed. New kinds of requests arrive, and the true relationship between content and harm is unchanged, but the classifier may not have learned it for the new region.
• Concept or subpopulation shift: the class-conditional distributions change in ways the classifier did not anticipate, for example a new attack style.

The WILDS benchmark of Koh and colleagues documented how large the performance drops from such in-the-wild shifts can be across domains, even for models that look strong in distribution. For a deployed classifier with score $s(x)$ and a fixed threshold $\tau$, all that matters operationally is the two class-conditional score distributions and the base rate. We model each class as a two-component Gaussian mixture with unit variances, where a growing share of traffic comes from a new component:

Here $w_b$ is the share of benign traffic from the new topic (which scores higher than ordinary benign traffic), $w_a$ is the share of attacks in the new style (which score lower than ordinary attacks), and the means are in logit units. Over 26 weeks $w_b$ grows linearly from 0 to 0.30, $w_a$ from 0 to 0.50, and the base rate $\pi$ from 2% to 5%. Recall, false positive rate and precision at threshold $\tau$ follow directly:

In these, $\bar{\Phi }$ is the standard normal upper tail, ${\mu }_h$ and ${\mu }_h^{\prime }$ are the means of the old and new attack components, and ${\mu }_b$ and ${\mu }_b^{\prime }$ those of the old and new benign components. Recall depends only on the harmful distribution, the false positive rate only on the benign one, and precision mixes both with the base rate. The launch threshold is $\tau =0.22$, which gives 90% recall and a false positive rate of 1.33% at week 0, for a precision of 58.1%.

What each shift does at a fixed threshold#

We ran the four scenarios separately, each for 26 weeks. The figures below are exact under the model; the label-free monitoring results further down use weekly samples of 20,000 requests.

Three observations matter for operations.

Base-rate drift is benign for precision and invisible to recall. More attacks with the same appearance means a larger share of flags are real. Teams sometimes "fix" a rising flag rate by raising the threshold, which in this case only throws away recall.

A new benign topic is the precision killer. Thirty percent of benign traffic moving to a region the classifier finds suspicious multiplies the false positive rate by 5.4, and precision falls from 58.1% to 20.5%. In a health or finance deployment this is usually the first complaint: customers asking legitimate questions get refused, the over-refusal side of the trade-off.

A new attack style is the recall killer, and it hides. Recall drops from 90% to 60.1%, but missed attacks generate no flags and no complaints. Nothing on a flag-rate dashboard moves in the attack-only scenario except a slight fall.

Explore the full scenario in three dimensions below. Each ribbon is one week's score distribution for one class; the highlighted slice follows the slider. Turn on re-thresholding to see where the threshold would have to move to hold 90% recall.

Base-rate drift can be corrected without labels#

If the shift really is label shift, the classifier's calibrated posterior at launch, $p_0(y=1\mid x)$, can be adjusted to a new prior $\pi$ by reweighting the odds:

where ${\pi }_0$ is the launch base rate and $p_0(x)$ is shorthand for the launch posterior. The new prior can be estimated from unlabelled traffic with the expectation-maximisation procedure of Saerens, Latinne and Decaestecker: start at ${\pi }_0$, compute adjusted posteriors for a sample, set $\pi$ to their mean, and repeat. Lipton, Wang and Smola's black-box shift estimation solves the same problem with a confusion-matrix inversion, and Alexandari and colleagues found that EM with a well-calibrated model is hard to beat. All of these rest on the label-shift assumption, so we tested what EM reports when the assumption is false. Each cell is 50 weekly samples of 20,000 unlabelled requests at week 26:

Under pure label shift, EM recovers the base rate almost exactly. Under covariate shift it reports a base rate nearly four times too high, because the new benign topic looks like extra attacks. Under attack drift it reports too low a base rate, because the new attacks look benign. The estimator does not know which world it is in, and its output is a confident number in all four. Use it to adjust thresholds only when you have separate evidence (an audit, below) that the class-conditional distributions have not moved.

Recalibrating with a small labelled set#

The standard remedy is a periodic labelled audit: sample production traffic, have experts label it, and re-pick the threshold to restore the recall target. The arithmetic is unforgiving. At a 5% base rate, a uniform sample of 500 requests contains about 25 harmful ones, and the threshold for 90% recall is the 10th percentile of their scores, which is estimated from two or three items in the tail.

We simulated audits at week 26: draw a labelled sample, set the threshold at the (weighted) 10th percentile of harmful scores, and record the true recall and precision that threshold achieves. We compared uniform sampling with two score-stratified designs: one that spends 75% of the budget on the top 20% of scores (the usual "enrich for positives" design), and one that spends 65% of the budget between the 50th and 95th score percentiles. Stratified samples are reweighted by inverse inclusion probability, the Horvitz-Thompson estimator:

where $S$ is the labelled sample, $h(i)$ is the stratum of item $i$, $N_h$ is the number of production items in stratum $h$, and $n_h$ the number sampled from it. Each weight says how many production items one labelled item stands for. Results over 1,000 simulated audits per design:

Two results are worth remembering. A 500-item uniform audit missed the recall target by more than five points in 24% of audits; quadrupling the budget cut that to 7%. And neither stratified design beat uniform sampling for the recall threshold, even though both found far more harmful items. Enrichment toward high scores was the worst, with 48% of 500-item audits ending below 85% recall, and the mid-range design came second at 37%. After drift, the harmful items that determine the 10th percentile are the ones the classifier now scores low, mixed into the bulk of benign traffic. Any design that samples low scores thinly and then multiplies each of those items by a large weight estimates that tail badly. Stratification does help the precision estimate, which depends only on the flagged region. Choose the design for the quantity you need: enrich above the threshold for precision, and for recall sample uniformly or stratify on something other than the guardrail's own score, such as disagreement with a second classifier.

Monitoring without labels: confidence and disagreement#

Labels are slow and expensive, so production monitoring leans on signals that need none. We tracked four in the combined scenario, each computed from a weekly sample of 20,000 requests:

• Flag rate: the share of traffic above threshold.
• EM base-rate estimate: the Saerens estimate above.
• Mean confidence: the average of $\max (p,1-p)$ under the launch-calibrated posterior. Confidence-based accuracy predictors such as the average-thresholded-confidence method of Garg and colleagues build on this signal.
• Disagreement: the share of requests where a second, differently trained classifier (for example an open safeguard classifier with its own threshold) makes the opposite call. Jiang and colleagues showed that disagreement between independently trained models tracks test error surprisingly well in distribution; out of distribution, it rises when the models' blind spots differ.

As alarms, the count-based signals are fast. Setting an alarm at three standard deviations above week-0 sampling noise, disagreement fired at week 1 and flag rate at week 2, while precision did not fall 20% below launch until week 6 and recall did not fall below 85% until week 5. That is the good news. The bad news is that none of these signals says which shift occurred: a rising flag rate is consistent with more attacks (precision up), a new benign topic (precision down) or both. A rising EM estimate is consistent with real base-rate drift and with covariate shift that EM misreads. Label-free signals tell you when to spend labels, not what the labels would say. Rabanser, Günnemann and Lipton found that two-sample tests on a black-box model's outputs were among the most effective ways to detect dataset shift; detection, though, is not diagnosis.

Disagreement has one more useful property: it localises. Requests where the two classifiers disagree are concentrated in the new regions of traffic, so they are the natural place to spend the next audit's labels, and to look for the new topic or attack style by hand.

What to do when the guardrail drifts#

Monitoring checklist#

1. Log raw scores, not only decisions, so thresholds can be re-evaluated retrospectively.
2. Track flag rate and disagreement with a second classifier weekly, with alarm limits set from week-0 sampling noise.
3. Run the EM base-rate estimate, but treat it as a symptom, not a diagnosis.
4. Budget labelled audits in harmful items, not total items: decide how many harmful examples you need in the low-score tail and size the sample from the expected base rate.
5. Use separate sampling designs for precision (enrich above threshold) and recall (uniform, or stratified on disagreement rather than on the guardrail score).
6. Keep a fixed adversarial test set from launch and a rolling one from recent red-teaming; recall on the rolling set is the only direct view of attack drift.

Code: EM base-rate estimation#

import numpy as np

def em_base_rate(p0, pi0, iters=200, tol=1e-8):
    """Saerens-Latinne-Decaestecker EM for a new class prior.
    p0: calibrated launch posteriors P(harmful | x) on an unlabelled production sample
    pi0: base rate in the data the posteriors were calibrated on
    Valid only under label shift: p(x | y) must be unchanged."""
    pi = pi0
    for _ in range(iters):
        a = (pi / pi0) * p0
        b = ((1 - pi) / (1 - pi0)) * (1 - p0)
        new = np.mean(a / (a + b))
        if abs(new - pi) < tol:
            break
        pi = new
    return pi

Failure modes#

• Raising the threshold when the flag rate rises. Under base-rate drift this throws away recall for no precision gain.
• Treating a flat dashboard as a healthy guardrail. Attack drift lowers recall without producing flags.
• Auditing with too few positives. A 500-item uniform audit at a 5% base rate sets a recall threshold from a handful of tail items.
• Enriching for the wrong quantity. Score-enriched samples estimate precision well and recall thresholds badly.
• Trusting calibration after shift. Calibration is a property of a model on a distribution. Ovadia and colleagues found that predictive uncertainty degrades under dataset shift across methods; Guo and colleagues' temperature scaling fixes calibration on the distribution it was fitted on, not on next quarter's traffic.
• Using the same model as monitor and guardrail. Disagreement only carries information if the second classifier's blind spots differ from the first's.

The model is deliberately simple: Gaussian scores, linear drift, one new component per class, and a second classifier whose noise correlates with the first at 0.5. Real drift is lumpy and seasonal, and new components are rarely Gaussian. The qualitative results, that each shift moves a different metric, that EM misreads non-label shift, and that small audits cannot pin down a recall threshold, follow from the formulas, not from the specific numbers.

Frequently asked questions#

Sources#

1. Saerens, Latinne, Decaestecker. Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure, 2002. doi.org
2. Guo, Pleiss, Sun, Weinberger. On Calibration of Modern Neural Networks, 2017. arxiv.org
3. Lipton, Wang, Smola. Detecting and Correcting for Label Shift with Black Box Predictors, 2018. arxiv.org
4. Rabanser, Günnemann, Lipton. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift, 2018. arxiv.org
5. Alexandari, Kundaje, Shrikumar. Maximum Likelihood with Bias-Corrected Calibration is Hard-To-Beat at Label Shift Adaptation, 2019. arxiv.org
6. Ovadia, Fertig, Ren, Nado et al.. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift, 2019. arxiv.org
7. Koh, Sagawa, Marklund, Xie et al.. WILDS: A Benchmark of in-the-Wild Distribution Shifts, 2020. arxiv.org
8. Jiang, Nagarajan, Baek, Kolter. Assessing Generalization of SGD via Disagreement, 2021. arxiv.org
9. Garg, Balakrishnan, Lipton, Neyshabur et al.. Leveraging Unlabeled Data to Predict Out-of-Distribution Performance, 2022. arxiv.org
10. Inan, Upasani, Chi, Rungta et al.. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations, 2023. arxiv.org