Memorization and membership inference in fine-tuned models: attacks, canaries and what mitigations buy

When a model is fine-tuned on sensitive records, two questions follow: can someone tell whether a particular record was in the training set, and can someone get the record back out? This note covers the attacks used to answer the first question (loss, reference-model, Min-K% and zlib scores), the canary-and-exposure method used to answer the second, and what fine-tuning does to both. We ran a small real experiment: a brief full fine-tune of a 0.5B open model on synthetic patient-style records with inserted secret canaries, measuring membership attacks and exposure after every epoch. We close with mitigations and their costs, including what a differential-privacy guarantee would and would not buy.

The setting: fine-tuning on records you must protect#

A hospital fine-tunes a small open model on clinic letters so it drafts them in house style (the evaluation side of such deployments is covered in our note on clinical LLM evaluation). A bank fine-tunes one on complaint transcripts. In both cases the training set holds personal data, each record appears a handful of times, and the model is trained for several epochs because the dataset is small. That is close to the worst case for memorization. Pretraining runs see each document roughly once in a corpus of trillions of tokens, and membership attacks against pretrained models barely beat chance [10]. Fine-tuning reverses both conditions: a small dataset, many passes.

Two kinds of leakage matter, and they need different tests:

• Membership. Given a record, can an attacker tell whether it was in the training data? For a dataset defined by a condition (patients of an oncology clinic), membership itself is sensitive.
• Extraction. Can an attacker make the model reproduce training content, such as an identifier or a phone number, without already knowing it? Extraction of verbatim training sequences, including personal contact details, was demonstrated against GPT-2 [2] and is the risk regulators point to most directly [13].

Membership inference attacks#

Membership inference against machine learning models was formalised by Shokri and colleagues, who trained shadow models to imitate the target [3]. All the attacks here assign each candidate record $x$ a score, and call it a member if the score exceeds a threshold. They differ in what the score measures.

Loss attack#

The simplest score is the model's average negative log-likelihood on the record, with lower loss meaning "more likely a member" [4]:

Here $p_{\theta }$ is the fine-tuned model, $x_t$ the $t$-th token and $T$ the record length (we negate it so that a higher score means member). Its weakness is that some records are simply easy to predict, member or not.

Reference-model attack#

Calibrating against a reference model removes much of that difficulty effect. With the pre-fine-tuning model $p_0$ as reference,

where ${\mathcal{L}}_0$ and ${\mathcal{L}}_{\theta }$ are the average losses under the reference and fine-tuned models. The score is the improvement fine-tuning made on this specific record. This is the natural attack against fine-tuned models, because the base model is usually public, and it was used in studies of fine-tuning memorization [11]. Stronger versions train many shadow models to model the score distribution per example [5].

Min-K% and zlib#

Min-K% Prob averages only the $k\%$ least likely tokens in the record, on the idea that unseen text contains a few surprising tokens that seen text does not [6]:

where $S_k$ is the set of the $k\%$ of positions with the lowest token probability. The zlib score divides the loss by the record's zlib-compressed length, a reference-free way to discount text that is simply repetitive [2].

How to read an attack's success#

Report the true positive rate at a low false positive rate, not just AUC. An attack that identifies 5% of members with 0.1% false positives is a real privacy failure even if its AUC is 0.55, because it is confident about specific people [5]. Plot ROC curves on log axes so the low-FPR region is visible.

Canaries and exposure#

Membership attacks ask about records you already hold. Canaries ask whether the model will give up a secret it should never reveal [1]. Insert a synthetic record containing a random secret, for example "The door code for the Aspen ward is 483920", train, then ask how highly the model ranks the true secret among all possible secrets. Exposure is defined as

where $\mathcal{R}$ is the space of possible secrets (here all $10^6$ six-digit codes) and ${\mathrm{rank}}_{\theta }(s)$ is the position of the true secret when every candidate is sorted by the model's likelihood given the prefix. A canary the model has not learned sits around the middle of the ranking, giving exposure near 1 bit. A canary ranked first has the maximum, ${\log }_{2}10^6\approx 19.9$ bits, and can be extracted by simply asking. Exposure is useful because it is continuous: it shows memorization building up well before extraction succeeds.

Ranking a million candidates per canary is expensive. The original paper estimates the rank by fitting a distribution to the likelihoods of a sample of random candidates and reading off the tail probability of the true secret [1]. We do the same with a normal fit to 1,000 random candidates per canary:

where $\ell (s)$ is the log-likelihood of the true secret, $\hat{\mu }$ and $\hat{\sigma }$ are the mean and standard deviation over random candidates, and $Z$ is standard normal. The estimate is capped at 19.9 bits.

An experiment: fine-tuning a 0.5B model on synthetic patient records#

Setup (real run, synthetic data). We fully fine-tuned Qwen2.5-0.5B-Instruct [16] for four epochs on a laptop GPU (AdamW, learning rate $2\times 10^{-5}$, batch 16, fixed seed). The training data were 1,000 synthetic patient-style records, for example "Record 482913: Amara Okafor, age 54, seen at Riverside clinic on 12/3/2019. Complaint: persistent cough. Prescribed amoxicillin 500 mg, follow-up in 2 weeks." Every field was randomly generated; no real person is described. To study duplication, 600 records appeared once, 200 twice and 200 four times. A further 1,000 records from the same generator were held out as non-members. We inserted 20 canaries of the form "The door code for the Aspen ward is 483920." with random six-digit codes, four canaries at each of 1, 2, 4, 8 and 16 insertions, and kept 20 never-inserted canaries of the same form as controls. In total the model saw 1,924 training sequences per epoch.

After every epoch we scored all 2,000 records with the four attacks, estimated each canary's exposure from 1,000 random candidates, and tried greedy extraction by giving the model the canary prefix and decoding six tokens. The epoch-0 row is the untouched base model, which serves as a sanity check: the 40 canaries average 1.40 bits of exposure, close to the 1.44 bits expected for a secret the model knows nothing about, and every attack sits at chance (AUC 0.47 to 0.53).

Canaries: exposure rises within two epochs#

Mean exposure in bits; no control canary was ever extracted. After one epoch only the canaries inserted 8 or 16 times stood out. By the end of the second epoch, every canary inserted four or more times was at or near the ceiling, and 11 of 20 could be extracted by simply asking. By the fourth epoch, all four canaries that appeared once per epoch were extracted verbatim. Repetition sets how fast memorization arrives; epochs decide whether it arrives at all. Exposure also saturates well before extraction becomes reliable: at epoch 3, most canaries were at the exposure ceiling yet only 11 were reproduced by greedy decoding, so the two measurements answer different questions and both are worth running.

Membership: from chance to near-certain#

All 1,000 members against 1,000 held-out non-members. The non-member loss was lowest after epoch 1 and rose afterwards, the classic overfitting signature, and membership leakage tracked the gap between the two losses, as the overfitting analysis of Yeom and colleagues predicts [4]. By epoch 4 the loss attack identified 73% of members at a false positive rate of one in a thousand.

Duplication mattered as much as epochs. At epoch 2, when the attack against all members had an AUC of 0.67, records seen four times per epoch were already at 0.84, with 25.5% detected at 1% FPR, against 0.60 and 2.5% for singletons. At epoch 3, the four-times records were at an AUC of 0.999. This matches the deduplication literature [8, 9]: repeated sequences carry most of the risk.

The four attacks did not rank the way the literature might suggest. Min-K% and loss were close throughout (at epoch 3, AUC 0.869 and 0.866 for all members); zlib was a little weaker; and the reference-model attack was clearly the weakest (0.723). The reason is our data. Every record comes from the same template, so they are about equally hard for the base model, and subtracting the base model's loss removed little real difficulty while adding its noise. On natural text, where records differ widely in how predictable they are, reference calibration usually helps [5, 11]. Test several attacks on your own data rather than assuming one ordering.

What early stopping would have bought#

Stopping at the held-out loss minimum, after epoch 1, gives a sharply different privacy picture from training to epoch 4: a loss-attack AUC of 0.57 instead of 0.99, 2.2% of members detected at 1% FPR instead of 91.6%, and 2 of 20 canaries extracted (both inserted 16 times) instead of 19. In this run, standard good practice for generalization was also the single most effective privacy control we measured. That will not hold for every dataset, since a model that needs many epochs to learn the task will memorise along the way, but it makes held-out loss tracking a minimum requirement for any sensitive fine-tune.

Mitigations and their costs#

What a differential-privacy guarantee buys#

Training with $(\epsilon ,\delta )$-differential privacy bounds every membership test, whatever its method. For any attack with false positive rate $\alpha$, the true positive rate satisfies [14]

Here $\epsilon$ is the privacy budget and $\delta$ a small failure probability. At $\epsilon =1$ and $\alpha =1\%$ no attack can exceed a TPR of 2.7%; at $\epsilon =8$, a budget commonly reported for fine-tuning language models with DP-SGD, the bound at 1% FPR is vacuous (it allows a TPR of 100%). Large budgets are not worthless, since real attacks fall far short of the worst case, but they should be reported as an empirical result backed by attacks like the ones above, not as a guarantee. Two further limits: the guarantee is per training example, so a patient with twenty letters in the data is protected at roughly twenty times the budget; and it says nothing about records that are already public.

Where this meets data protection rules#

Under the GDPR, whether a model trained on personal data is itself personal data was addressed by the European Data Protection Board in Opinion 28/2024, adopted on 17 December 2024 [13]. The Board considered that such models cannot in all cases be considered anonymous, and that for a model to be anonymous both the likelihood of extracting personal data about training subjects and the likelihood of obtaining such data from queries should be insignificant. It lists, among the evidence a controller may provide, measures to prevent extraction and assurance regarding resistance to attacks. Membership inference, extraction and canary tests are the measurements that kind of assessment needs.

In US health care, HIPAA's de-identification standard applies to the training data, through either the Safe Harbor list of identifiers or an expert determination [15]. De-identified text can still contain rare combinations of facts that a memorizing model can reproduce, so the evaluation in this note complements de-identification; it does not replace it.

Evaluation checklist for a sensitive fine-tune#

1. Hold out a random sample of eligible records from the same distribution as non-members. Distribution shift between members and non-members (for example, different time periods) inflates attack success and must be avoided [10].
2. Insert canaries in several repetition counts (1, 2, 4, 8, 16) with random secrets drawn from a known space; keep a matched set of never-inserted controls.
3. After each epoch, run loss, reference-model, Min-K% and zlib attacks; report AUC and TPR at 1% and 0.1% FPR, by duplication count.
4. Measure canary exposure and greedy extraction per repetition count.
5. Deduplicate, stop early, and rerun; record what each mitigation changed.
6. If DP-SGD is used, report $\epsilon$, $\delta$, the unit of privacy (record or person) and the empirical attack results together.
7. Keep the canary suite as a regression test for every retraining.

Failure modes in privacy evaluation#

• Shifted non-members. Non-members drawn from a different period or source make any attack look strong. The same trap affects benchmark contamination tests, which are membership inference by another name (see our note on contamination detection).
• AUC only. Averages hide confident identification of a few records.
• No duplication breakdown. Aggregate numbers are dominated by singletons and understate risk to repeated records.
• Canaries unlike real secrets. A canary format the model finds very unusual can be memorised faster or slower than real identifiers; mirror real formats.
• Testing once. Memorization changes with every epoch, learning-rate change and dataset refresh.

Code: canary exposure with a fitted tail#

import math, random, torch

@torch.no_grad()
def exposure(model, tok, prefix, secret, n_cand=1000, space_bits=math.log2(10**6), device="cpu"):
    """Estimated exposure of a 6-digit secret after `prefix` (Carlini et al. 2019, normal-tail fit)."""
    cands = [secret] + ["".join(random.choice("0123456789") for _ in range(6)) for _ in range(n_cand)]
    enc = tok([prefix + c for c in cands], return_tensors="pt", padding=True, padding_side="left").to(device)
    n = len(tok(secret, add_special_tokens=False).input_ids)        # digits are single tokens in many tokenizers
    logits = model(**enc, logits_to_keep=n + 1).logits[:, :-1].float()
    lp = torch.log_softmax(logits, -1).gather(-1, enc.input_ids[:, -n:, None])[..., 0].sum(-1).cpu()
    ll, rest = lp[0].item(), lp[1:]
    z = (ll - rest.mean().item()) / rest.std().item()
    tail = 0.5 * math.erfc(z / math.sqrt(2))                       # P(random candidate scores higher)
    return min(space_bits, -math.log2(max(tail, 1e-300)))

Check that every candidate tokenises to the same number of tokens as the secret; if the tokenizer merges digits, score whole candidate strings instead of the last $n$ positions.

Frequently asked questions#

Sources#

1. N. Carlini, C. Liu, Ú. Erlingsson et al.. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks, 2018. arxiv.org
2. N. Carlini, F. Tramèr, E. Wallace et al.. Extracting Training Data from Large Language Models, 2020. arxiv.org
3. R. Shokri, M. Stronati, C. Song, V. Shmatikov. Membership Inference Attacks against Machine Learning Models, 2016. arxiv.org
4. S. Yeom, I. Giacomelli, M. Fredrikson, S. Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting, 2017. arxiv.org
5. N. Carlini, S. Chien, M. Nasr et al.. Membership Inference Attacks From First Principles, 2021. arxiv.org
6. W. Shi, A. Ajith, M. Xia et al.. Detecting Pretraining Data from Large Language Models, 2023. arxiv.org
7. M. Abadi, A. Chu, I. Goodfellow et al.. Deep Learning with Differential Privacy, 2016. arxiv.org
8. K. Lee, D. Ippolito, A. Nystrom et al.. Deduplicating Training Data Makes Language Models Better, 2021. arxiv.org
9. N. Kandpal, E. Wallace, C. Raffel. Deduplicating Training Data Mitigates Privacy Risks in Language Models, 2022. arxiv.org
10. M. Duan, A. Suri, N. Mireshghallah et al.. Do Membership Inference Attacks Work on Large Language Models?, 2024. arxiv.org
11. F. Mireshghallah, A. Uniyal, T. Wang et al.. Memorization in NLP Fine-tuning Methods, 2022. arxiv.org
12. N. Lukas, A. Salem, R. Sim et al.. Analyzing Leakage of Personally Identifiable Information in Language Models, 2023. arxiv.org
13. European Data Protection Board. Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models, 2024. edpb.europa.eu
14. P. Kairouz, S. Oh, P. Viswanath. The Composition Theorem for Differential Privacy, 2013. arxiv.org
15. U.S. Department of Health and Human Services, Office for Civil Rights. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule, 2012. hhs.gov
16. Qwen team: A. Yang et al.. Qwen2.5 Technical Report, 2024. arxiv.org