LLMs, GenAI, RAG, agents, evaluation · insight-0510
Repeat evaluation runs because temperature 0 does not guarantee identical outputs
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
Action
Run both variants several times (for example three runs each), report mean and spread, and treat differences inside the run-to-run range as no difference.
Reason
Provider documentation states that results are not fully deterministic even at temperature 0, so one run understates variance.
Boundary
Hosted-API evaluation, prompt comparisons, regression gates.
Symptom or risk
The gap is run-to-run noise; rerunning the same prompt gives a different score and the decision flips.
Exceptions
Some newer hosted models restrict sampling parameters altogether; local inference with fixed seeds and kernels can be reproducible, but verify rather than assume.
Trade-offs
Repeated runs multiply evaluation cost.
Pitfalls
Rerunning against a response cache and calling the result reproduced.
Lessons and concepts
Claims and their sources
At check time, Anthropic's Messages API reference stated that results are not fully deterministic even with temperature 0.0, and marked sampling parameters as restricted for newer models.