Engineering, production, governance, cost, communication · insight-1272
Add hysteresis and a cooldown to automated retraining triggers
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A retraining job starts whenever a drift statistic crosses a threshold, and the statistic oscillates around the line for a week.
Action
Require the condition to hold for a minimum period, use different thresholds to arm and to reset the trigger, impose a minimum interval between retrains, and send repeated triggers to a human review.
Reason
A single threshold on a noisy signal flaps; separate arm and reset levels with a dwell time are the usual control-systems remedy for it.
Boundary
Drift-, volume- or performance-triggered retraining and auto-promotion.
Symptom or risk
The pipeline retrains and redeploys several times a week, creating cost, release churn and noisy comparisons, and nobody can tell which model is responsible for a metric change.
Exceptions
A confirmed severe failure should bypass the cooldown through a manual path.
Trade-offs
Slower reaction to real shifts; pair with a manual override.
Pitfalls
A cooldown that counts from the trigger, not from the last promoted model.