LLMs, GenAI, RAG, agents, evaluation · insight-0565
Calibrate LLM confidence on labelled data before using it as a threshold
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- insurance, banking
Summary: situation, action, reason, boundary
Situation
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
Action
On a labelled set, plot reliability (stated confidence versus observed accuracy), fit a calibration map or choose the threshold from observed precision at each cut-off, and re-check after every model or prompt change.
Reason
One technical report showed post-training reduced the calibration of a model's answer probabilities, and research found verbalised confidence tends to be overconfident.
Boundary
Auto-approval, routing to humans, abstention thresholds.
Symptom or risk
The confidence is systematically too high, so the auto-approval error rate far exceeds what the threshold implies.
Exceptions
Pretrained models on well-formatted multiple-choice questions can be well calibrated, but that does not transfer automatically to your task.
Trade-offs
Needs labelled data and re-calibration after changes.
Pitfalls
Reading 0.9 as 90 percent likely to be correct without measuring it.
Lessons and concepts
Claims and their sources
The GPT-4 technical report (Figure 8) shows answer probabilities on an MMLU subset were well calibrated before post-training and significantly less calibrated after it.
Kadavath et al. (an arXiv preprint) found larger pretrained models well calibrated on multiple-choice and true/false questions in the right format, with weaker calibration of their self-knowledge predictions on new tasks.
Xiong et al. found LLMs tend to be overconfident when verbalising confidence.