LLMs, GenAI, RAG, agents, evaluation · insight-1614
Keep prompt logs in a separate store with a short TTL, narrow access and access auditing
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- telecom, banking
Summary: situation, action, reason, boundary
Situation
Engineers need prompts and responses to debug, and the content flows into the general observability platform with long retention and broad read access.
Action
Send structured metadata (ids, model, token counts, latency, finish reason) to general telemetry and raw text only to a restricted store. Give that store role-based read access, an audit log of reads, a TTL by data class measured in days, sampling instead of capture-all and a break-glass flow for incidents.
Reason
OpenTelemetry's generative-AI conventions make instructions and message content opt-in and flag them as possibly sensitive; the storage-limitation principle asks for identifiable data to be kept no longer than necessary.
Boundary
Production assistants whose conversations contain customer or employee text.
Symptom or risk
Anyone with a dashboard login can read customers' messages, and retention follows the platform default rather than any decision.
Exceptions
Synthetic traffic and internal test environments without personal data.
Trade-offs
Debugging is slower when raw text expires or needs approval to read.
Pitfalls
A TTL on the main store while copies live on in exports, notebooks and backups.
Lessons and concepts
Claims and their sources
The OpenTelemetry generative-AI semantic conventions mark content attributes such as input and output messages and system instructions as opt-in and potentially sensitive.
GDPR Article 5(1)(e) states storage limitation, keeping identifiable data no longer than necessary.