AI safety, with error bars.
Reinforced Labs is an evaluation laboratory for models and agents. We measure what your system does with stated uncertainty, stop what it must never do at runtime, and supply the expert data that fixes what we find.
A point estimate is an opinion.
"94% pass rate" means nothing without the sample size, the grader's error rate and the variance across runs. Most AI launch decisions are made on numbers that couldn't survive a methods review. Ours are built to.
adversarial test cases executed for customers last quarter
graders, each published with its measured agreement against expert labels
median Guard decision latency, tool-call inspection included
credentialed specialists writing and judging the hardest cases
Four instruments, one calibration standard.
Each product measures or controls a different part of the system, and all four share the same graders, policies and audit trail.
MeasureReinforced Evals
Release-gating eval suites for models, RAG pipelines and agents. Calibrated graders, multi-seed runs and statistical gates in CI.
Evals →EnforceReinforced Guard
Runtime policy for agents. Inspects inputs, retrieved content, outputs and every tool call; blocks injection, exfiltration and unsafe actions.
Guard →AttackReinforced Red
Attacker agents probe staging every night; specialists run deep engagements before launches. Every finding becomes a regression test.
Red Teaming →TeachReinforced Data
Red-team corpora, preference pairs and refusal-calibration sets written by credentialed experts and delivered with agreement statistics.
Safety Data →Every failure becomes a permanent test.
Red teaming finds it, Guard contains it, Evals make sure no future change brings it back.
Safety checks that run like unit tests.
Define a suite once, run it on every pull request, and block the merge when a category regresses beyond sampling noise, not on every wobble.
- Python and TypeScript SDKs, plus a CLI for any CI system
- Paired tests and multi-seed runs, so a 1-point dip on 200 cases doesn't block a release
- Grader error corrected using a human-labelled calibration set
- Per-case diffs that show exactly which behaviour changed
from reinforced import Suite, graders
suite = Suite.load("support-agent/release")
suite.add(graders.ToolCallPolicy("policies/refunds.yaml"))
suite.add(graders.InjectionResistance(corpus="indirect/email-v3"))
run = suite.run(target="https://staging.internal/agent",
concurrency=64, seeds=3)
# paired test vs. main; block only on significant regressions
run.gate(min_score=0.95, max_regression=0.02, confidence=0.95)Recent research notes.
Each note comes with the code that produced its figures. All research →
How many test cases does an LLM eval need?
Wilson intervals, power analysis and a Monte Carlo study of paired vs. unpaired tests. Detecting a 2-point regression takes thousands of cases, not hundreds.
Your LLM judge is a diagnostic test: correcting measured failure rates
Sensitivity, specificity and the Rogan–Gladen correction. At a 0.5% true failure rate, a 97%-accurate judge reports 3.4%.
Evaluating agents against adaptive, multi-turn attacks
Why static attack suites overstate defences, how success grows over turns, why attack budgets must be reported, and a protocol that avoids overfitting.
pass@k vs. pass^k: measuring agent reliability
Three task populations with identical pass@1 = 0.60 have pass^8 between 1.7% and 28.8%. How to estimate pass^k and how many trials you need.
Built for the people who sign the launch memo.
ML and platform leads
Replace notebook evals with a versioned suite that runs in CI and says, with an interval, whether the new model is better.
CISOs and security teams
Treat agents like any privileged workload: threat model, continuous adversarial testing, runtime enforcement and an audit trail for every blocked action.
Risk, compliance and legal
Evidence mapped to the EU AI Act, NIST AI RMF and ISO/IEC 42001, generated from real test runs rather than questionnaires. How the mapping works.
Model developers
Pre-release capability and misuse evaluations, expert red teaming, and post-training data from specialists in biology, cyber, finance, law and medicine.
We had forty evals in notebooks and no idea which ones mattered. Six weeks later every merge to main runs eleven thousand cases and blocks only on regressions that are real. Launch reviews went from a week of arguing to one report.Head of ML Platform, Fortune 100 financial services firm
Questions we get asked.
What does Reinforced Labs do?
We build the measurement and control layer for production AI. Evals tells you what your model or agent actually does, Guard stops what it must never do at runtime, Red Teaming keeps finding new ways it can fail, and Safety Data supplies the expert-labelled examples that teach it the difference.
How is this different from public benchmarks?
Public benchmarks measure a model in general. We measure your system: your prompts, your tools, your retrieval, your policies and your users. A model that scores well on a leaderboard can still issue refunds it shouldn't, leak a customer record through a tool call, or follow instructions hidden in a PDF. Those failures only show up when you test the deployed system.
Which models and frameworks do you support?
Any model you can call: closed API models, open-weight models served on vLLM, TGI or SGLang, and fine-tuned variants. Agent frameworks are supported through OpenTelemetry traces and native SDKs for Python and TypeScript. You can point Evals at a raw model, a RAG pipeline or a full multi-step agent.
Can we run it inside our own cloud?
Yes. Every product can run as managed SaaS, as a single-tenant deployment, or fully inside your VPC on AWS, Google Cloud or Azure. Air-gapped on-prem installs are available for regulated and public-sector customers.
How long until we have a working eval suite?
Most teams have a first suite running against staging in the first week. Our applied team typically delivers a release-gating suite, with a threat model and baseline report, within 30 days.
Do you train on our data?
No. Customer prompts, outputs, traces and datasets are never used to train our models or shared with other customers. Data retention is configurable down to zero for Guard traffic.
Know the number before you ship.
A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.