Engineering, production, governance, cost, communication · insight-1269
Re-derive decision thresholds whenever the model is retrained
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- telecom, banking, retail
Summary: situation, action, reason, boundary
Situation
A churn model's score cutoff of 0.35 was tuned for capacity in the first release and is hard-coded in the campaign job, while the model is retrained monthly.
Action
Store the threshold as a versioned artifact of the model, recompute it from the validation set against the stated capacity or cost ratio at each retrain, and review the change in selected volume at release.
Reason
A manually set threshold is valid only for the model it was tuned on; when the model updates, the old value may no longer be right.
Boundary
Any retrained classifier or scorer feeding an action with a capacity limit or a cost ratio.
Symptom or risk
The retrained model's score distribution shifts, so the same cutoff selects a different volume and a different precision, and the campaign over- or under-spends without any code change.
Exceptions
Rank-based rules such as 'top 5,000 per day' do not depend on score scale.
Trade-offs
Automatic thresholds move decision volume between releases; add a bound on the allowed change.
Pitfalls
Re-deriving the threshold on the same data that early stopping used.
Lessons and concepts
Claims and their sources
Sculley et al. note that manually set decision thresholds can become invalid when a model updates on new data and suggest learning thresholds from held-out evaluation.