LLMs, GenAI, RAG, agents, evaluation · insight-0575
Do not assume generated data is private; test it for memorised training records
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Applied AI, Data scientist
- Industries
- healthcare, retail
Summary: situation, action, reason, boundary
Situation
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
Action
Deduplicate training data, search generated outputs for near-duplicates of training records, apply privacy techniques where required, and get governance sign-off before sharing anything outside the training boundary.
Reason
Research extracted memorised training images from diffusion models and found them less private than earlier GANs in its tests; training-data extraction has also been shown for language models.
Boundary
Synthetic data release, generative products trained on sensitive or licensed data.
Symptom or risk
Some outputs are near-copies of training examples, exposing personal or licensed content.
Exceptions
Models trained only on public, licensed-for-reuse data carry lower privacy risk but still need licence checks.
Trade-offs
Privacy techniques and filtering reduce output quality and add cost.
Pitfalls
Checking only exact duplicates, which misses near-copies.
Lessons and concepts
Claims and their sources
Carlini et al. extracted memorised individual training images from diffusion models and found them much less private than earlier generative models such as GANs in their tests.
Carlini et al. showed that verbatim training data can be extracted from large language models.