LLMs, GenAI, RAG, agents, evaluation · insight-1591
Gate on a non-inferiority margin for the paired difference, not on a raw score threshold
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A CI gate fails the build when the candidate scores below the baseline by any amount on a 400-item set.
Action
Decide a margin you will tolerate per metric (for example 2 points), compute the paired difference with an interval on the same items, and pass when the interval's lower end is above minus the margin, fail when its upper end is below it, and otherwise report "unproven" (add items or decide on cost) instead of rerunning until green.
Reason
The gate should encode a decision under uncertainty: how much loss is acceptable, and how sure you must be that it was not exceeded.
Boundary
Prompt, retrieval and model changes gated on a fixed eval set.
Symptom or risk
Run-to-run noise trips the gate on harmless changes, teams learn to rerun until green, and genuine regressions hide among the false alarms.
Exceptions
Safety-critical must-pass cases need a zero-failure rule, not a margin.
Trade-offs
Proving a small margin takes more items; otherwise report "unproven" and decide on cost.
Pitfalls
Choosing the margin after seeing the result.
Example
Hypothetical: 40 of 400 items flip between versions, so the paired difference has a standard error near 1.6 points; a 0.5-point drop is inside noise, yet a 2-point margin cannot be demonstrated either.