Engineering, production, governance, cost, communication · insight-0275
Gate releases on a frozen golden set with tolerances sized to run-to-run noise
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model-quality CI check fails randomly about one run in five, and engineers start rerunning until it passes.
Action
Measure metric variation across several seeds on the frozen set, set the tolerance from that spread, compare against the incumbent rather than a fixed number, and require investigation, not reruns, on failure.
Reason
A gate is only useful if failures are rare and meaningful, which requires thresholds above noise.
Boundary
Model and prompt regression suites, retraining pipelines.
Symptom or risk
Exact-match or zero-tolerance thresholds turn normal training noise into flaky failures, and reruns teach the team to ignore the gate.
Exceptions
Deterministic components (feature functions) should still use exact tests.
Trade-offs
Wide tolerances let small regressions through; track the trend across releases.
Pitfalls
Letting the golden set grow with recent production data, which moves the baseline.