Guardrail thresholds under low base rates

A minimal model of a guardrail classifier#

Let a detector assign each request a score $s$. Benign requests have $s\sim \mathcal{N}(0,1)$ and attacks have $s\sim \mathcal{N}(d,1)$. This equal-variance binormal model is the standard textbook model of a detector, and its single parameter maps to a familiar summary:

We use $d=2.5$, so $\mathrm{AUC}=0.961$, a strong classifier by any leaderboard standard. At threshold $t$, $\mathrm{TPR}(t)=1-\Phi (t-d)$ and $\mathrm{FPR}(t)=1-\Phi (t)$. Setting $\mathrm{TPR}=0.95$ requires $t=d-1.645$, which gives $\mathrm{FPR}=19.6\%$.

Precision collapses at low base rates#

The quantity operators experience is precision, the probability that a blocked request was really an attack. Bayes' rule gives it in terms of the attack base rate $\pi$:

Attacks are rare in most enterprise traffic. Figure 2 shows precision at the 95%-recall operating point across base rates.

This is the base-rate fallacy familiar from medical screening and intrusion detection. It explains a pattern we see constantly: a guardrail that looked excellent on a balanced red-team set is switched to "log only" within a week of launch, because support tickets about wrongly blocked customers outnumber real attacks a hundred to one.

Choosing a threshold from costs#

The right operating point minimises expected cost per request. With $C_{\text{fn}}$ the cost of a missed attack and $C_{\text{fp}}$ the cost of wrongly blocking a legitimate request:

Differentiating, the optimum sits where the likelihood ratio of the score equals the cost-weighted odds:

for the binormal model. The threshold depends on base rate and cost ratio together, so a single threshold shared across deployments with different exposure is almost always wrong for most of them. We solved for $t^{\ast }$ numerically on a fine grid and tabulated the consequences per million requests:

Reading the table. At $\pi =0.1\%$ with a 100:1 cost ratio, a fairly severe miss cost for a support bot, the optimal threshold gives only 63% recall and still wrongly blocks 14,970 legitimate requests per million. To get recall near 99% you must accept blocking more than a third of all traffic. No threshold on this classifier makes a high-impact action safe at an acceptable false-positive cost.

What this means for architecture#

The table argues for splitting the problem by consequence rather than tuning one classifier harder:

• Low-impact paths (answering a question, summarising a document): use the classifier at a cost-optimal threshold, and act on flags softly by stripping suspicious spans, adding a warning or logging.
• High-impact actions (payments, sending email externally, changing permissions): don't rely on a probabilistic detector. Use deterministic tool-call policy and provenance rules, whose false-positive rate on legitimate actions you control directly, and human approval above a limit. These controls don't suffer the base-rate problem because they don't classify text; they constrain actions.
• Raise the effective base rate before classifying. Running the expensive detector only on content from untrusted channels, such as external email, web pages and uploaded files, increases $\pi$ within the screened stream by orders of magnitude, and precision with it.

Our note on correlated layers makes the complementary point: these deterministic controls also fail independently of the classifier, which is what makes layering worthwhile.

Measuring your own operating point#

1. Estimate $\pi$ per channel from production samples labelled by humans. Don't use your red-team set, whose base rate you chose.
2. Measure TPR on an adversarial corpus and FPR on a large benign sample from the same channel. FPR needs tens of thousands of benign cases for a tight interval near 0.1%; see sample sizes.
3. Agree $C_{\text{fn}}:C_{\text{fp}}$ with the business owner, per action class.
4. Compute $t^{\ast }$ per channel and action class, deploy in dry-run, and compare the observed block rate with the prediction before enforcing.

Do two-stage cascades rescue precision?#

A common response is to stack a second, more expensive detector (often an LLM judge) on whatever the first one flags. If the two stages were independent given the class, the cascade's rates would multiply:

Take two copies of our detector, each at 95% recall ($\mathrm{FPR}=19.6\%$). The cascade has 90.3% recall and an FPR of about 3.85%, and at $\pi =0.1\%$ its precision is

That is better than 0.5%, but it is still a stream of blocks of which 98 in 100 are mistakes, and that is the optimistic case. Two ML detectors looking at the same text are positively correlated, so the real FPR product is larger, by the same mechanism our correlated-layers simulation quantifies. Cascades are good for cost, since they run the expensive check only on flagged traffic. They don't escape the base rate.

Translating rates into an operations budget#

A threshold choice is also a staffing decision whenever flagged items go to human review. With daily volume $V$, the review load is $V[\pi \mathrm{TPR}+(1-\pi )\mathrm{FPR}]$. From the table, at $\pi =0.1\%$ and a 100:1 cost ratio, roughly 15,599 alerts per million requests. A product doing 5 million requests a day would generate about 77,993 review items a day, of which about 3,146 are real. Few teams can staff that, which is the practical reason to keep classifiers on soft actions and put hard actions behind deterministic policy.

Limitations#

Frequently asked questions#

Sources#

1. J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1):29–36, 1982. doi.org
2. S. Axelsson. The base-rate fallacy and the difficulty of intrusion detection. ACM Transactions on Information and System Security 3(3):186–205, 2000. doi.org