LLMs, GenAI, RAG, agents, evaluation · insight-1610
Scan model output for secrets and personal identifiers before it leaves the trust zone
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- banking, healthcare
Summary: situation, action, reason, boundary
Situation
Answers can quote retrieved documents, tool results and logs, and they go to end users, other systems or chat channels.
Action
Run a deterministic output filter: secret patterns and high-entropy tokens, checksum-validated numbers such as card numbers, and identifiers that are absent from the requesting user's own data. Block or mask, log the event with the source chunk id.
Reason
OWASP treats model output as untrusted input to downstream components. A filter independent of the model still works when the model has been talked into quoting something it should not.
Boundary
Assistants with retrieval, tool results or log access that answer to users or other systems.
Symptom or risk
An API key from a config file, a card number or another customer's email appears in a reply, and schema validation passes because the shape is fine.
Exceptions
Allow identifiers that the requesting user supplied in the same conversation.
Trade-offs
Masking false positives annoy users; tune patterns on real traffic.
Pitfalls
Relying on regex alone, and assuming a detector finds everything, which Presidio's own documentation says it cannot guarantee.
Lessons and concepts
Claims and their sources
OWASP recommends treating model output as untrusted and validating it before it reaches downstream components.
Presidio's documentation warns that its automated detection cannot guarantee that all sensitive information is found.