LLMs, GenAI, RAG, agents, evaluation · insight-1613draft — in review
Trace a deletion from the source record to chunks, embeddings, caches and datasets
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- e-commerce, telecom
Summary: situation, action, reason, boundary
Situation
A customer asks for removal or a document is withdrawn, and the original record is deleted from the source system.
Action
Keep lineage ids from source record to chunk, embedding, cache key and dataset row, and run a deletion job that fans out along them and emits a per-store completion report. Record what cannot be removed, such as backups and trained weights, with its expiry, and ship the report as the evidence that the request was handled.
Reason
Pipelines create derived copies, so deletion that stops at the source leaves them behind. Some vendor features keep objects such as files and vector stores until you delete them.
Boundary
Any system that copies user or document data into indexes, caches or datasets.
Symptom or risk
Copies persist in the vector index, chunk store, answer cache, evaluation sets, fine-tuning data, traces and vendor-side stores, and the assistant can still quote them.
Exceptions
Legal holds and records you must retain, handled by policy and not by silent skipping.
Trade-offs
Lineage metadata costs storage and engineering effort.
Pitfalls
Deleting the vector but not the cached answer, and forgetting data already sent into provider-side stores.
Lessons and concepts
Claims and their sources
OpenAI documents that files, fine-tuning jobs, batches, vector stores and conversations are stored until you delete them (as read on 2026-10-12).