LLMs, GenAI, RAG, agents, evaluation · insight-1600
Publish an eval report with set version, judge version, runs, intervals and the number of variants tried
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Applied AI, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
An internal eval result is shared as "new prompt: 84% vs 81%" with no further context.
Action
Attach to every result: golden-set version and size, judge model and prompt version, runs per item, the interval on the difference, per-slice results with the worst slice, the number of variants tried, and known limitations; write "no detectable difference" with the effect size the set could have shown.
Reason
Test scores alone do not show which model is best, and conclusions can change with the compute or trials spent, so the search budget belongs in the result.
Boundary
Any eval result that informs a release, a vendor choice or a budget.
Symptom or risk
Readers cannot judge whether the gap is noise, whether the set or judge changed, or how many variants were tried to find it, so the number is cited long after it stops being valid.
Exceptions
Throwaway exploratory runs need not carry the full template if they are labelled as such.
Trade-offs
A little more writing per result; a template makes it routine.
Pitfalls
Reading "not significant" as "equivalent"; a small set cannot rule out a large loss.
Lessons and concepts
Claims and their sources
Dodge et al. argue that test scores alone cannot reliably show which model is best and propose reporting expected validation performance as a function of the budget of hyperparameter trials.