LLMs, GenAI, RAG, agents, evaluation · insight-0640
Run each agent test several times and report consistency, not one lucky pass
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- retail, cross-industry
Summary: situation, action, reason, boundary
Situation
An agent passes a test task once in a demo and the task is marked solved.
Action
Run each task k times (for example 4 to 8), report the share of tasks solved on every attempt (pass^k) next to the single-attempt rate, and investigate tasks that flip.
Reason
τ-bench introduced pass^k to measure consistency over repeated trials and found agents far less reliable across repeats than single-run success suggests.
Boundary
Customer-facing agents, any agent with sampling randomness or non-deterministic tools.
Symptom or risk
In production the same task succeeds on some attempts and fails on others, and users meet the failures.
Exceptions
Fully deterministic pipelines with fixed seeds and cached tool results; repeats add nothing there.
Trade-offs
k repeats multiply evaluation cost; use them on a core subset and on release candidates.
Pitfalls
Confusing pass^k (all k succeed) with pass@k (at least one succeeds); they answer opposite questions.
Lessons and concepts
Claims and their sources
τ-bench proposes pass^k as a reliability metric over multiple trials and reports pass^8 below 25% in its retail domain for a leading function-calling agent.