Research · Methods
How our figures are made.
Research notes on this site are built from small, self-contained Python programs. They produce JSON; our site build turns the JSON into the static charts and tables you see. Nothing is drawn by hand.
Principles
- Fixed random seeds; reruns reproduce every number exactly.
- Synthetic data with stated distributions, never presented as measurements of a real system.
- Each note ends with its limitations.
- Equations are typeset at build time; no client-side rendering.
- External sources are cited to primary papers and standards.
| Program | Note | Replications |
|---|---|---|
| eval_sample_size.py | Eval sample size | 2,000 / cell |
| judge_bias.py | Judge error | 3,000 / cell |
| pass_k.py | pass^k | 200k tasks; 4,000 |
| injection_retry.py | Retry risk | 2M draws / ρ |
| guardrail_thresholds.py | Thresholds | analytic grid |
| guardrail_latency.py | Latency | 1M requests |
reproduceshell
python3 -m pip install numpy scipy # tested: numpy 2.2, scipy 1.15
for s in eval_sample_size judge_bias pass_k \
injection_retry guardrail_thresholds guardrail_latency; do
python3 research/$s.py # writes research/out/$s.json
doneThe scripts are available to customers and research collaborators on request through the contact form.
Know the number before you ship.
A 45-minute working session with an evaluation engineer. Bring one use case; leave with a threat model, a draft eval plan with sample sizes, and an honest read on where you stand.