LLMs, GenAI, RAG, agents, evaluation · insight-0662

Trace each LLM request as spans per stage with model, tokens and finish reason

Situation
A user reports a slow or wrong answer, and logs show only the final response and total latency.
Action
Emit one trace per request with child spans for retrieval, rerank, each model call and each tool call; record requested and responded model, input and output tokens, finish reason, document ids and config versions.
Reason
Stage-level spans turn a vague complaint into a located failure. OpenTelemetry's GenAI conventions standardise attributes such as request and response model, input token usage and finish reasons, so traces stay comparable across vendors.
Boundary
Production RAG and agent systems (day 52 core skill).

Symptom or risk

You cannot tell whether time and errors came from retrieval, reranking, the model call or a tool, or which model version answered.

One-off notebooks.

Trace volume costs money; sample successful requests and keep all errors.

Logging only the requested model alias, not the model version that actually responded.

OpenTelemetry's GenAI span conventions (status Development) include gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens and gen_ai.response.finish_reasons.
OpenTelemetry semantic conventions for generative AI (spans and metrics)