Building training and evaluation data in regulated domains

Training and evaluation data for clinical, financial and legal models usually starts as records that identify real people. Turning them into something a model team can use is an engineering pipeline with measurable failure rates, not a checkbox. This note walks through that pipeline for a concrete case: what the de-identification rules actually require, how a PII detector's recall becomes a count of identifiers left in your corpus, why removing direct identifiers is not the same as low re-identification risk, how to measure detector recall honestly, where synthetic data helps, and what an audit trail needs to contain. All quantities come from a seeded simulation, labelled as such, and nothing here is legal advice: the regulatory references describe what the rules say so that engineers and their privacy counsel can talk about the same thing.

The case: an evaluation set from discharge summaries#

A team wants an evaluation set of 10,000 hospital discharge summaries to test a clinical summarisation model (the evaluation design itself is covered in our note on clinical LLM evaluation), plus structured fields (age, sex, ZIP code, admission date) to stratify results. The notes are free text written by clinicians. Each contains names (the patient, relatives, clinicians), record numbers, phone numbers, dates, ages and places, scattered through prose and occasionally inside tables pasted from other systems.

The team has two decisions to make. Which de-identification standard will the data meet, and how will they show, with numbers, that the pipeline meets it? The rest of this note answers the second question for whichever answer counsel gives to the first.

What the rules say#

HIPAA: Safe Harbor and Expert Determination#

Under the HIPAA Privacy Rule, health information is de-identified, and no longer protected health information, if it meets one of two methods in 45 CFR 164.514(b):

• Safe Harbor (164.514(b)(2)) lists 18 types of identifiers of the individual or of relatives, employers or household members that must be removed: names; geographic subdivisions smaller than a state, except that the first three digits of a ZIP code may be kept if the area they cover contains more than 20,000 people (otherwise they become 000); all elements of dates except the year for dates directly related to the individual, and ages over 89 aggregated into a single "90 or older" category; telephone and fax numbers; email addresses; Social Security, medical record, health plan, account and licence numbers; vehicle and device identifiers; URLs and IP addresses; biometric identifiers; full-face photographs; and any other unique identifying number, characteristic or code. The covered entity must also have no actual knowledge that the remaining information could identify someone.
• Expert Determination (164.514(b)(1)) requires a person with appropriate statistical and scientific expertise to determine that the risk is "very small" that the information could be used, alone or with other reasonably available information, by an anticipated recipient to identify an individual, and to document the methods and results.

The HHS Office for Civil Rights published guidance on both methods in 2012, including how experts reason about replicability, data availability and distinguishability of quasi-identifiers. Section 164.514(c) separately allows a re-identification code, provided it is not derived from information about the individual and the mechanism is not disclosed.

The engineering consequence is that Safe Harbor is a rule about what must be absent, while Expert Determination is a claim about risk. For free text, Safe Harbor still becomes a measurement problem, because no automated detector removes every listed identifier, and "removed" in practice means "removed with recall below 100%".

Finance, legal and the EU#

Financial data carries its own regimes (nonpublic personal information under the Gramm-Leach-Bliley Act in the US, cardholder data under the PCI DSS industry standard), and legal documents carry duties of client confidentiality and privilege. In the EU, the GDPR distinguishes anonymous information, which falls outside the regulation (Recital 26), from pseudonymised data (Article 4(5)), which remains personal data. For high-risk AI systems, Article 10 of the EU AI Act requires data governance practices for training, validation and testing data, covering among other things data collection and origin, preparation steps such as annotation and cleaning, examination for possible biases, and identification of data gaps. NIST SP 800-188 (2023) is a useful neutral reference for de-identification governance across all of these. What they share is the need to show what was done to the data and how well it worked, which is what the rest of this note is about.

From detector recall to identifiers left behind#

Suppose each note contains on average ${\lambda }_d=4$ spans of direct identifiers (names, record numbers, contact details) and ${\lambda }_q=8$ spans of quasi-identifiers in the text (dates, ages, places). These are assumptions for the simulation; measure your own corpus. A detector with span-level recall $r$ misses a fraction $1-r$ of spans, so the expected number left in a corpus of $N$ notes is

which is just volume times miss rate. If spans in a note are roughly Poisson and misses independent, the probability that a note keeps at least one direct identifier is

where the exponent is the expected number of missed direct identifiers per note. For small miss rates, $P(\text{leak})\approx {\lambda }_d(1-r)$: the leak rate per note is roughly the miss rate times the number of identifiers per note.

Worked through for our corpus at 99% recall: $10,000\times 4\times 0.01=400$ residual direct identifiers and $1,200$ residual spans in total. The chance a given note keeps at least one direct identifier is $1-e^{-0.04}=3.9\%$, about 392 notes out of 10,000 (a simulated corpus agreed: 3.9%). A detector that sounds excellent at 99% still leaves an identifier in roughly one note in 25. At 95% recall it is one in 6.

Quasi-identifiers: why removal is not the same as low risk#

Direct identifiers are only half of the problem. Sweeney estimated in 2000 that 87% of the US population was likely unique on five-digit ZIP code, sex and full date of birth; Golle's 2006 re-analysis with later census data put the figure at 63%. Rocher, Hendrickx and de Montjoye showed in 2019 that with 15 demographic attributes, 99.98% of Americans would be correctly re-identified in any dataset. Combinations of ordinary fields single people out.

The standard measure is $k$-anonymity, due to Sweeney: group records by their quasi-identifier values, and let $k_i$ be the size of the group containing record $i$. A dataset is $k$-anonymous if every group has at least $k$ members. Under the prosecutor model, where an attacker knows the target is in the data and knows the target's quasi-identifiers, the chance of picking the right record is $1/k_i$:

The second identity holds because a group of size $k$ contributes $k\times 1/k=1$. In plain terms: the attacker can expect to re-identify about one person per distinct combination of quasi-identifiers.

We generated a synthetic population of 10,000 patients of one regional health system: ages from a skewed distribution, sex, 12 three-digit ZIP areas with about 25 five-digit ZIP codes each, and admission dates over three years. We then measured $k$ under four levels of generalisation:

The Safe Harbor-style row is the important one. Keeping age in years (with 90 and over pooled), sex, the three-digit ZIP and the admission year satisfies the Safe Harbor rules for those fields, yet 13% of our records are unique on them and the mean prosecutor risk is 37%. Safe Harbor does not promise a risk level; it promises that specific fields are absent. Whether the remaining combination is risky depends on the size and shape of the dataset, which is exactly what Expert Determination analyses. Note also that the prosecutor model is the worst case: an attacker who does not know whether the target is in the data faces lower risk, and an expert would also weigh how available the quasi-identifiers are to plausible recipients.

Putting the two together#

Combine the leaked direct identifiers with the structured quasi-identifiers. Suppose a single leaked direct identifier lets an attacker link the record with probability $q=0.5$ (a common surname links less often than a record number does). A record is re-identified if either route works:

Here the first factor is the chance the text route fails and the second the chance the quasi-identifier route fails, treated as independent. Summing over records gives the expected number of re-identifiable records.

The table shows the two failure modes trading places. With coarse quasi-identifiers, detector recall dominates: moving from 95% to 99.9% recall cuts expected re-identifications from 934 to 50. With Safe Harbor-style fields, the quasi-identifiers dominate, and the same recall improvement moves the total only from 4,252 to 3,692. The practical rule: work out which term is larger before spending money on the other one.

Explore the trade-off below. The precision slider shows the cost on the other side: false positives that redact clinically meaningful text.

Precision, utility and surrogates#

Recall protects patients; precision protects the data's usefulness. Each false positive removes a span that was not an identifier, and in clinical text those spans are often the content a model needs: eponymous diseases (Parkinson, Hodgkin, Graves), drug names that look like surnames, hospital units named after donors. At 99% recall the number of over-redacted spans per 10,000 notes is $N({\lambda }_d+{\lambda }_q)r(1-p)/p$: about 13,200 at 90% precision and 1,200 at 99%. For an evaluation set, over-redaction biases results, because the model is tested on text that no longer contains what a clinician wrote.

Replacing detected identifiers with realistic surrogates, rather than with tags like [NAME], helps on both sides. Carrell and colleagues showed that surrogate substitution "hides in plain sight" the identifiers a detector misses, because a reader cannot tell a real leaked name from a surrogate. Surrogates also keep the text natural for models. They complicate audits, though, so keep a secured mapping and mark surrogate spans in an internal annotation layer.

Measuring detector recall honestly#

Every number above depends on recall, so recall has to be measured on your notes, by annotators, not quoted from a benchmark. Shared tasks such as the 2014 i2b2/UTHealth de-identification track, described by Stubbs, Kotfila and Uzuner, and neural systems such as that of Dernoncourt and colleagues report high recall on their corpora; your note types, templates and abbreviations will differ.

The statistical trap is that spans cluster in documents, the same problem that makes benchmark error bars too narrow. One note with an unusual template can hide a dozen identifiers the detector has never seen. We simulated detectors with an average span recall of 99% but with recall that varies across documents (a beta distribution under which 4% of documents have recall below 95%), then estimated recall from annotated samples of documents with two 95% intervals: a Wilson interval that treats spans as independent, and a clustered interval that treats documents as the unit. Each row is 4,000 repetitions:

Both intervals fall short of their nominal 95% coverage at small samples. The span-level Wilson interval stays at about 90% however many documents you add, because it ignores the clustering; the document-level interval converges toward 95% as the number of documents grows, but needs a few hundred documents to get there. For a claim like "recall is at least 98%", plan on annotating 200 or more documents, report the interval with documents as the unit, and report per-identifier-type recall separately, since rare types (device serials, unusual ID formats) are where detectors fail.

Where synthetic data helps, and where it does not#

Synthetic records are attractive because they seem to dissolve the problem. The evidence is more mixed. Stadler, Oprisanu and Troncoso evaluated a range of generative models and found that synthetic data either did not prevent inference attacks or did not retain utility, with no better privacy-utility trade-off than traditional anonymisation and a less predictable one. Generators trained on real text can also memorise and emit it: Carlini and colleagues extracted verbatim training sequences, including personal information, from a large language model, and Lukas and colleagues found that scrubbing reduces but does not prevent PII leakage from fine-tuned models, and that sentence-level differential privacy still leaked about 3% of PII sequences in their experiments.

Formal guarantees come from differential privacy, introduced by Dwork, McSherry, Nissim and Smith, which bounds how much any one record can change a generator's output distribution. The cost is utility, and for rare conditions and small subgroups, which are often what a clinical evaluation is meant to probe, the noise needed can erase the signal.

A workable split:

• Use synthetic data for pipeline development, unit tests, prompt iteration and red-team scaffolding, where realism matters less than safety.
• Use de-identified real data, with measured recall and documented risk, for the evaluation results you will report and act on.
• Before releasing any synthetic set derived from real records, run membership-inference and nearest-neighbour checks against the source, and treat a generator trained on identifiable text as identifiable itself.

The audit trail#

Regulators, internal review boards and customers' auditors will ask the same question: show what was done to these records. The answer is a lineage record per dataset version.

Checklist#

1. Count identifiers per note type in a pilot sample, by identifier type, before choosing a detector.
2. Annotate at least a few hundred documents for evaluation, stratified by note type and source system; report recall per type with documents as the unit.
3. Compute the $k$ distribution of the structured fields you plan to keep, and the expected re-identification count under stated assumptions.
4. Decide whether the text route or the quasi-identifier route dominates, and spend effort there.
5. Measure over-redaction on clinically meaningful terms, and check that model evaluation results are not shifted by redaction.
6. Prefer surrogates to tags for released text; keep the mapping secured and logged.
7. Version every dataset and keep the lineage record with it.

Code: k-distribution and expected risk#

import numpy as np

def expected_reidentifications(qi_columns, recall, lam_direct=4.0, q_link=0.5):
    """qi_columns: list of equal-length integer arrays (already generalised quasi-identifiers).
    Returns the expected number of re-identifiable records under the prosecutor model,
    combining class size k with the chance a direct identifier survives de-identification."""
    keys = np.stack(qi_columns, axis=1)
    _, inverse, counts = np.unique(keys, axis=0, return_inverse=True, return_counts=True)
    k = counts[inverse.ravel()]
    p_leak = 1 - np.exp(-lam_direct * (1 - recall))
    risk = 1 - (1 - q_link * p_leak) * (1 - 1 / k)
    return risk.sum(), (k == 1).mean()

Failure modes#

• Quoting benchmark recall. Recall on a public shared-task corpus says little about your templates. Measure on your notes.
• Span-level intervals. They look tight and undercover. Use documents as the unit.
• Treating Safe Harbor as a risk number. Small or narrow datasets can satisfy the field rules and still contain many unique records.
• Forgetting free-text quasi-identifiers. Occupations, rare events and family details in prose identify people and are invisible to table-based $k$ analysis.
• Over-redacting the evaluation set. A model evaluated on text stripped of eponyms and drug names is not being evaluated on clinical text.
• Synthetic data as a release shortcut. A generator trained on identifiable notes can reproduce them.

The simulation rests on assumptions stated in the text: Poisson identifier counts, independent misses within a note, a fixed linkage probability for leaked direct identifiers, and the prosecutor attack model. Real misses cluster (a template the detector never saw), which concentrates leaks in fewer notes; real attackers rarely know every quasi-identifier. The structure of the result, that two different routes to re-identification each need their own measurement, does not depend on those choices.

Frequently asked questions#

Sources#

1. US Department of Health and Human Services. 45 CFR 164.514: Other requirements relating to uses and disclosures of protected health information, 2002. ecfr.gov
2. HHS Office for Civil Rights. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule, 2012. hhs.gov
3. Garfinkel, Guttman, Near et al.. De-Identifying Government Datasets: Techniques and Governance (NIST SP 800-188), 2023. csrc.nist.gov
4. European Parliament and Council. Regulation (EU) 2016/679 (General Data Protection Regulation), 2016. eur-lex.europa.eu
5. European Parliament and Council. Regulation (EU) 2024/1689 (Artificial Intelligence Act), 2024. eur-lex.europa.eu
6. Sweeney. Simple Demographics Often Identify People Uniquely, 2000. dataprivacylab.org
7. Sweeney. k-Anonymity: A Model for Protecting Privacy, 2002. doi.org
8. Dwork, McSherry, Nissim, Smith. Calibrating Noise to Sensitivity in Private Data Analysis, 2006. doi.org
9. Golle. Revisiting the Uniqueness of Simple Demographics in the US Population, 2006. doi.org
10. Machanavajjhala, Kifer, Gehrke, Venkitasubramaniam. l-Diversity: Privacy Beyond k-Anonymity, 2007. doi.org
11. Carrell, Malin, Aberdeen, Bayer et al.. Hiding in Plain Sight: Use of Realistic Surrogates to Reduce Exposure of Protected Health Information in Clinical Text, 2013. doi.org
12. Stubbs, Kotfila, Uzuner. Automated Systems for the De-identification of Longitudinal Clinical Narratives: Overview of 2014 i2b2/UTHealth Shared Task Track 1, 2015. doi.org
13. Dernoncourt, Lee, Uzuner, Szolovits. De-identification of Patient Notes with Recurrent Neural Networks, 2016. arxiv.org
14. Rocher, Hendrickx, de Montjoye. Estimating the Success of Re-identifications in Incomplete Datasets Using Generative Models, 2019. doi.org
15. Carlini, Tramer, Wallace, Jagielski et al.. Extracting Training Data from Large Language Models, 2020. arxiv.org
16. Stadler, Oprisanu, Troncoso. Synthetic Data: Anonymisation Groundhog Day, 2020. arxiv.org
17. Lukas, Salem, Sim, Tople et al.. Analyzing Leakage of Personally Identifiable Information in Language Models, 2023. arxiv.org