Safety training data: a buyer's guide

What safety training data is

Safety training data is labelled material used to shape how a model behaves in situations involving harm, misuse, privacy, regulated advice or adversarial pressure. It feeds supervised fine-tuning (SFT), reward model training, preference optimisation such as RLHF and DPO, and the construction of graders and evaluation sets.

It differs from general instruction data in two ways. The inputs are deliberately difficult: adversarial, borderline, ambiguous or high-stakes. And the labels need judgement that generalist annotators often can't provide. Deciding whether an answer about drug interactions is responsible, or whether a security explanation gives meaningful uplift, needs a pharmacist or a security researcher.

The four main types

1. Red-team corpora

Adversarial prompts and multi-turn conversations designed to elicit unsafe behaviour, labelled by technique (role-play, encoding, crescendo, injection), harm category and severity. Paired with reference safe completions for SFT, or used directly as evaluation sets.

2. Preference data

Pairs or rankings of responses to the same prompt, judged on criteria such as harmlessness, helpfulness and honesty, ideally with written rationales. Used to train reward models or directly with DPO-style methods. The quality of the rejected response matters as much as the chosen one: pairs where both answers are bad teach little.

3. Refusal calibration sets

Matched sets of borderline-but-legitimate requests and look-alikes that should be declined. They measure and correct both failure directions at once: harmful compliance and needless refusal. Without them, safety tuning tends to drift towards refusing anything that sounds sensitive.

4. Gold sets for graders

Expert-labelled samples used to calibrate model-based graders and measure their agreement with humans. They're small, typically a few hundred to a few thousand items per category, but every automated safety score you produce depends on them.

How to specify a dataset

The most common cause of disappointing safety data is a vague specification. A good spec includes:

• The policy, written down and versioned: what the model should do, refuse, or handle with care, with worked examples at the boundaries.
• A taxonomy of harm categories and attack techniques with target counts per cell, so coverage is deliberate rather than whatever annotators found easy.
• Difficulty targets: how many items should defeat your current model. A dataset your model already handles perfectly is wasted money.
• Expertise requirements per category, with credentials or tests annotators must pass.
• Format: schema, multi-turn structure, rationale fields, metadata, and how it maps onto your training stack.
• Quality acceptance criteria: agreement thresholds, gold accuracy, and what happens to batches that miss them.

Measuring annotation quality

You should never train on safety data whose quality you haven't measured. The core instruments are:

Ask every vendor for these numbers per delivery, with the raw double-labelled sample so you can recompute them yourself.

Common pitfalls

• Over-refusal by construction. If every risky-sounding prompt has a refusal as its reference answer, the model learns to refuse topics instead of harms. Include legitimate look-alikes.
• Template collapse. Adversarial prompts written to quota converge on a few patterns. Measure diversity, not just count.
• Stale attacks. Jailbreak techniques move fast. A corpus from last year over-represents attacks that current models already resist.
• Evaluation contamination. Keep training and evaluation data strictly separated, ideally produced by different annotator pools.
• Unsafe handling of hazardous content. Data covering dangerous capabilities needs access controls, vetted annotators and careful decisions about what detail is written down at all.

Build, buy or both?

Build in-house when the policy is unique to your product and changes often, or when the data involves confidential material that can't leave your organisation. Buy when you need specialist expertise you don't employ, scale beyond your team's capacity, or adversarial creativity from people who haven't seen your system. Most mature teams do both: an internal team owns policy and gold sets, and external specialists produce volume and adversarial coverage against that policy.

Whichever you choose, insist on provenance: for every record, who wrote it, who reviewed it, what they disagreed on, and which policy version it reflects. That trail is what lets you debug a model behaviour back to the data that caused it.

How to review a vendor sample in an afternoon

Before signing a data contract, ask for a sample of 200–500 records in your target categories, plus the double-labelled subset. Then check:

1. Run it against your model. What share of adversarial prompts actually break your current model? If it's under 10%, the data is too easy for you.
2. Read 50 records end to end. Look for repetitive templates, prompts that read like they were written to a quota, and reference answers you'd be embarrassed to ship.
3. Recompute agreement from the double-labelled subset yourself rather than trusting the summary number.
4. Check the rejected responses in preference pairs. They should be plausible mistakes, not strawmen.
5. Look for over-refusal in references. Count how many "safe" reference answers are refusals, and whether a careful expert would actually have refused.
6. Check near-duplicates with embedding similarity. Paraphrase padding inflates counts without adding coverage.
7. Verify expertise. Ask which credentials the annotators of each category hold and how they were tested.
8. Confirm provenance fields are present on every record: author, reviewer, policy version, adjudication notes.
9. Review the datasheet for known limitations, language coverage and handling of hazardous content.
10. Ask about exclusivity. Custom data should be yours alone; licensed corpora may also be used by your competitors.

Multilingual safety data

Safety behaviour learned in English transfers imperfectly to other languages, and low-resource languages remain a reliable jailbreak route. If your product serves users in multiple languages, budget for native-speaker red teaming and preference data in each major language, not machine translation of an English set. Translated adversarial prompts lose the idioms, cultural context and code-switching that real attackers use, and translated reference answers often read as stilted enough to teach the model the wrong register.

Frequently asked questions

Related: Reinforced Data · LLM evaluation · Refusal calibration research

Keep reading