Safety training data is the labelled data used to teach a model to behave safely: when to refuse, how to refuse, when to help despite a sensitive topic, and how to resist manipulation. It is distinct from the general instruction data a model learns helpfulness from, though the two must be balanced.
Main types
- Red-team corpora: adversarial prompts collected by people or automated attackers, with the model's responses and harm labels. See AI red teaming.
- Preference pairs: two responses to the same prompt with a judgement of which is safer or better, used to train reward models or directly in DPO. See preference data.
- Demonstrations: ideal responses written to policy, for supervised fine-tuning, including good refusals and safe completions that help without giving uplift.
- Refusal calibration sets: benign prompts that look harmful, labelled "should answer", to counter over-refusal.
- Policy-grounded labels: data labelled against a written policy or set of principles, as in constitutional AI.
What makes it good
- Clear policy: labels encode a written policy with worked edge cases; vague policies produce low agreement.
- Coverage: harm categories, languages, multi-turn and agentic settings, not just single English prompts.
- Balance: enough benign look-alikes that the model learns intent, not keywords.
- Expertise: categories such as medicine, chemistry or finance need domain experts to judge both harm and helpfulness.
- Provenance and versioning: every item traceable to its source, labeller and policy version.
Worked example
Illustrative numbers: a team collects 10,000 safety preference pairs. Each is labelled by two annotators; they agree on 8,600, and a third adjudicates the rest. A 300-item gold set audits each labeller, and two whose accuracy against gold falls below 85% are retrained. Dropping the 400 pairs that remain ambiguous after adjudication leaves 9,600 pairs. Small numbers, but each step removes noise the reward model would otherwise learn.
Why it matters
A model's safety behaviour is largely a reflection of its safety data. Label noise becomes reward noise, keyword-driven data becomes over-refusal, and gaps in coverage become jailbreaks. Investing in the data usually moves safety metrics more than changing the training algorithm.