LLMs, GenAI, RAG, agents, evaluation · insight-0555
Put a confidence interval on every eval score and size the set for the difference you care about
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A prompt revision raises accuracy on a 200-item eval set from 80 to 83 percent, and the team announces an improvement.
Action
Report the standard error with every score, cluster it when items share a source document, and use a power calculation to pick the eval size needed to detect the smallest difference worth acting on.
Reason
Evaluation guidance recommends standard errors and power analysis, and a meta-analysis found many NLP experiments underpowered for typical improvements.
Boundary
Any LLM eval used for go or no-go decisions.
Symptom or risk
With 200 items the sampling uncertainty (about plus or minus 5.5 points at 80 percent) is typically larger than a 3-point gain, so the next run or a new sample may reverse it.
Exceptions
Large effects, such as a format failure rate dropping from 30 to 2 percent, are clear even on small sets.
Trade-offs
Larger eval sets cost labelling and run time.
Pitfalls
Treating overlapping intervals of two models as the formal test; use a paired comparison instead.
Example
At 80 percent accuracy on 200 items, the standard error is about 2.8 points, so a 95 percent interval is roughly plus or minus 5.5 points.
Lessons and concepts
Claims and their sources
Miller (an arXiv preprint, not peer reviewed) recommends reporting standard errors for eval scores, clustering them when questions are grouped, and using power analysis to choose the number of questions.
Card et al. found underpowered experiments common in NLP, including test sets too small to detect typical improvements reliably.