LLMs, GenAI, RAG, agents, evaluation · insight-1541
Prove deletion end to end with a canary document that you delete and then probe for
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- banking, healthcare
Summary: situation, action, reason, boundary
Situation
A customer record or a withdrawn policy must disappear from the assistant, and the team deleted the source file and the main vector row.
Action
Maintain an inventory of every derived store, insert a canary document containing a unique phrase, delete it through the normal path, and run scripted probes against search, generation, caches and exports until all return nothing.
Reason
Erasure obligations apply to the data, not to one database, and derived artefacts multiply the places it can live; a probe tests the real behaviour.
Boundary
Systems ingesting personal or contract data, or content that can be withdrawn.
Symptom or risk
Copies survive in the lexical index, generated summaries, semantic or answer caches, evaluation sets and logs, and the assistant still reveals the content.
Exceptions
Corpora of public, non-personal documents that are never withdrawn.
Trade-offs
Deletion propagation can require rebuilding derived summaries and invalidating caches.
Pitfalls
Treating a tombstoned row as erased when the engine only removes it at a later merge or compaction.
Lessons and concepts
Claims and their sources
Article 17(1) of the GDPR obliges a controller to erase personal data without undue delay when a listed ground applies, such as the data no longer being necessary or consent being withdrawn; this is not legal advice.