LLMs, GenAI, RAG, agents, evaluation · insight-0658
Capture prompt and response content in traces only by explicit opt-in with redaction
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- healthcare, banking
Summary: situation, action, reason, boundary
Situation
You add tracing to an LLM application and the default plan logs full prompts, retrieved documents and outputs for every request.
Action
Log metadata by default (model, versions, token counts, latency, finish reason, document ids); capture content only for sampled or consented traffic, redact identifiers first, and set retention and access rules.
Reason
OpenTelemetry's GenAI conventions mark message, instruction and retrieval-query attributes as opt-in and note they may contain sensitive information, while model-name, token-count and finish-reason attributes are recommended.
Boundary
Any production tracing or LLM observability tool.
Symptom or risk
Traces become an unreviewed copy of customer messages, personal data and confidential documents, with longer retention and wider access than the source systems.
Exceptions
Short, access-controlled debugging windows on test data.
Trade-offs
Without content, some failures are harder to debug; keep a privacy-reviewed sampled store for that.
Pitfalls
Redacting the prompt but storing the retrieved documents verbatim.
Lessons and concepts
Claims and their sources
In the OpenTelemetry GenAI span conventions (status Development), content attributes such as input and output messages, system instructions and retrieval query text are opt-in and described as sensitive or possibly sensitive.