LLMs, GenAI, RAG, agents, evaluation · insight-0641

Report steps, tool calls and tokens per task alongside agent success rate

Situation
Two agent designs reach similar task success, and the comparison stops there.
Action
Log per run the number of inference calls, tool calls, input and output tokens and wall-clock time; report their distribution (median and 90th percentile) for successful and failed runs.
Reason
Agent success is a ratio of outcomes; trajectory length is a separate cost and risk signal. OpenTelemetry's GenAI conventions define counts of inference and tool calls per agent invocation for this purpose.
Boundary
Agent design comparisons and release gates.

Symptom or risk

One design needs three times more model calls and tool calls per task, so it is slower, costlier and has more chances to hit a failing dependency.

Very early prototypes, where only feasibility matters.

More metrics invite cherry-picking; agree on the decision metric beforehand.

Looking at means only; agent trajectories are long-tailed.

OpenTelemetry's GenAI metric conventions (status Development) include the number of inference calls and tool calls made during an agent invocation.
OpenTelemetry semantic conventions for generative AI (spans and metrics)