LLMs, GenAI, RAG, agents, evaluation · insight-0509
Set decoding parameters per task instead of one global temperature
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- aviation, e-commerce
Summary: situation, action, reason, boundary
Situation
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
Action
Use greedy or low-temperature decoding for extraction, classification and code; use moderate temperature or nucleus (top-p) sampling where variety is wanted; store decoding settings with each prompt version and evaluate them together.
Reason
Maximisation-based decoding tends to give bland, repetitive long text, sampling from the full distribution produces incoherent tails, and truncated sampling such as top-p sits between them.
Boundary
Every production prompt.
Symptom or risk
Extraction results change between runs and occasionally contain invented values, while any task switched to greedy decoding produces repetitive long text.
Exceptions
Self-consistency and candidate generation sample deliberately even for exact tasks.
Trade-offs
More settings to maintain, each needing regression tests.
Pitfalls
Treating low temperature as a factuality control; it makes outputs consistent, not correct.
Lessons and concepts
Claims and their sources
Holtzman et al. showed that maximisation-based decoding leads to bland, repetitive text and proposed nucleus (top-p) sampling, which truncates the unreliable tail of the distribution.