Classical ML, validation, metrics, interpretation, debugging · insight-1147
Track alert volume at a fixed score threshold, because score drift moves it silently
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking, aviation, logistics
Summary: situation, action, reason, boundary
Situation
A model with a fixed 0.30 cut-off has run for months and operations complains that the queue is twice the planned size.
Action
Log the alert rate and the score quantiles per period, compare them with the validated alert rate, and recalibrate or move the threshold to a target alert rate or cost-based point.
Reason
A threshold fixes a score value, not a share of traffic; a drift in the score distribution changes the share without any code change.
Boundary
Queue-driven and capacity-bound models: fraud review, inspections, collections.
Symptom or risk
Inputs and seasonality shifted the score distribution, so the same threshold now flags a different share of cases even if the ranking quality is unchanged.
Exceptions
A threshold fixed by policy on a calibrated probability should stay put; then investigate the shift instead.
Trade-offs
Rate-based thresholds keep staffing stable but let actual risk per alert vary.
Pitfalls
Chasing the alert rate with the threshold while the underlying prevalence or model quality has changed.
Example
Hypothetical: a cut-off that flagged 5% of cases at validation flags 9% after a fare-structure change.