LLMs, GenAI, RAG, agents, evaluation · insight-0545

Anchor synthetic training data to real data and track its provenance

Situation
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
Action
Label every training record with its origin (human, model, model version), keep a fixed share of real data in every round, filter synthetic data against quality checks, and evaluate on a real held-out tail slice.
Reason
Research showed that indiscriminately training on model-generated content causes irreversible defects in which the tails of the original distribution disappear.
Boundary
Synthetic instruction data, data augmentation, distillation loops.

Symptom or risk

Rare intents and unusual phrasings fade from the data, and the model gets worse exactly on the tail cases that matter.

One-off, filtered synthetic augmentation with a real-data anchor is far less exposed than recursive loops.

Real data is slower and costlier to collect.

Evaluating on synthetic data generated by the same process.

Shumailov et al. showed that indiscriminate training on recursively generated data causes irreversible defects in which tails of the original distribution disappear.
AI models collapse when trained on recursively generated data