LLMs, GenAI, RAG, agents, evaluation · insight-0639

Score agents on the end state they produce, not on how the transcript reads

Situation
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
Action
For each test task define the expected final state (records created or changed, messages sent, values) and compare the environment after the run with that goal state.
Reason
τ-bench evaluates agents by comparing the database state at the end of a conversation with an annotated goal state, which checks outcomes rather than wording.
Boundary
Tool-using agents that change records, tickets, bookings or files.

Symptom or risk

Polite, plausible transcripts hide wrong database changes: the wrong flight rebooked, a refund issued twice, a policy step skipped.

Purely conversational assistants, where response quality itself is the outcome.

Needs a resettable test environment with seeded data.

Checking only the target record and missing collateral changes elsewhere.

τ-bench judges agent success by comparing the final database state with the annotated goal state.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains