LLMs, GenAI, RAG, agents, evaluation · insight-0624
Compare RAG variants with paired per-question differences on small golden sets
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
Action
Score both variants on the same questions, count questions B fixed and broke, and put an interval on the paired difference (bootstrap over questions) before deciding.
Reason
Paired comparison removes question difficulty from the variance, and an interval shows whether the plausible effect includes zero.
Boundary
Golden sets under a few hundred items, which is most RAG and agent evaluations.
Symptom or risk
A three-point gap on 80 questions is a difference of two or three answers and can easily be noise, so the team ships churn.
Exceptions
Changes with a clear mechanism and no regressions on inspection can ship on qualitative evidence if risk is low.
Trade-offs
Narrow intervals need more labelled questions; prioritise labelling the slices where variants disagree.
Pitfalls
Running many variants and reporting the best one without accounting for multiple comparisons.
Example
Synthetic illustration: B fixes 6 questions and breaks 4, a net gain of 2 out of 80, which is weak evidence.