The latency budget of AI guardrails: sequential, parallel and hedged

Model#

Each guardrail check (injection classifier, PII detector, policy evaluation, grounding check) takes a random time $X_j$. Service latencies are right-skewed, so we model each as log-normal with median 8 ms and shape $\sigma =0.6$:

With $k$ checks there are three ways to run them:

• Sequential: $T={\sum }_j{X}_j$. Simple, and common when checks are chained middleware.
• Parallel: $T={\max }_j{X}_j$. All checks fan out and the request waits for the slowest.
• Parallel + hedged: if a check hasn't returned by its own p95, send a duplicate to another replica and take whichever answers first, $\min (X_j,p95+X_j^{\prime })$. This is the "hedged request" technique from Dean and Barroso's The Tail at Scale [1].

Why parallel tails grow with k#

For independent checks, the CDF of the maximum is the product of the individual CDFs:

With $k=8$, the parallel p99 is the single-check quantile at $0.99^{1/8}\approx 0.9987$. Fan-out turns a far-tail event of one check into a routine event of the request. Hedging works by cutting off exactly that tail: a check only waits past its p95 if both the original and the duplicate are slow.

Results#

One million simulated requests per configuration.

Findings. (1) Going from sequential to parallel at $k=4$ cuts the median from 36.4 to 14.6 ms and p99 from 77.4 to 43.0 ms. (2) Hedging at the single-check p95 cuts eight-check p99 by 28% (49.0 → 35.2 ms) and leaves the median unchanged, for about 5% extra check traffic, since only slow checks are duplicated. (3) With hedging, adding checks is nearly free at the tail: p99 rises only 4.5 ms from two checks to eight.

Budget rules we use#

1. Never chain ML checks sequentially on the request path. If one check depends on another's output, restructure so the dependent check runs on the stream, after the first token.
2. Deterministic rules first and inline. Spend limits, allow-lists and schema validation cost microseconds; they shouldn't share a latency budget with classifiers.
3. Hedge at p95 for any check with a heavy tail, and cap duplicates per second so hedging can't amplify an overload.
4. Race the model call. Input checks can run concurrently with the first model call; cancel the call if a check fires. That moves input-check latency off the critical path entirely.
5. Budget at p99, report at p99. A median of 15 ms says nothing about the 1-in-100 users who wait 50 ms on every turn of a multi-turn session.

What hedging costs, exactly#

Hedging at threshold $t_h$ duplicates exactly the checks that exceed it, so the extra load is the tail probability of a single check:

Hedging at the p95 therefore costs 5% more check traffic by construction, which matches the simulated 5.0%. Hedging earlier buys more tail reduction at a linearly rising cost: at the p90 the load is 10%, at the median it is 50%. The benefit has a ceiling, because a hedged check still can't finish before $t_h$ plus the duplicate's own latency. For our parameters, the hedged single-check p99 is bounded below by roughly $t_h+\text{median}\approx 29$ ms whenever the original is slow. Going much below p95 rarely pays.

The duplicate must go to a different replica. If both copies queue behind the same overloaded instance, their latencies are correlated and the minimum barely improves. This is the main way hedging fails in practice.

Output checks on a stream: chunk size is a security parameter#

Output guardrails (PII, secrets, policy) shouldn't wait for the whole completion, or they add the full generation time to the user's wait. Checking the stream in chunks of $c$ tokens adds a latency of roughly one check per chunk on the critical path, but it also defines how much text can reach the user before a violation is caught. With a generation speed of $v$ tokens per second and check latency $L$, the worst-case exposure is

At $v=80$ tokens/s, $c=32$ and a p99 check latency of 40 ms, at most about 35 tokens can be shown before a block, enough for a partial card number. Two mitigations: hold back the last $c$ tokens from display until their chunk has been checked (a small, constant delay of $c/v$ seconds), and use a deterministic pattern check, which costs microseconds, for the highest-risk secrets so they are never displayed.

A worked budget for a real policy#

A support agent with five checks, a 50 ms p99 guardrail budget per turn, and the latency model above:

Only one ML check sits on the request's critical path, plus per-chunk output checks. The simulated single-check hedged p99 of 28.5 ms fits the 50 ms budget with room for network overhead. The same five checks chained sequentially would have a p99 near 77.4 ms for the four non-trivial ones, which is the configuration that gets guardrails switched off.

Limitations#

Frequently asked questions#

Sources#

1. J. Dean and L. A. Barroso. The Tail at Scale. Communications of the ACM 56(2):74–80, 2013. doi.org