Classical ML, validation, metrics, interpretation, debugging · insight-0157
Separate prevalence changes from model decay before retraining on a production precision drop
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- banking, manufacturing
Summary: situation, action, reason, boundary
Situation
Production precision of an alerting model falls by a third in a month and the team wants to retrain.
Action
Decompose the drop: recompute recall (true positive rate) and false positive rate, which do not depend on prevalence, and the prevalence itself on labelled recent data.
Reason
Precision depends on prevalence; with stable TPR and FPR, fewer positives alone lowers precision.
Boundary
Fraud, defect, churn and incident alerting with delayed labels.
Symptom or risk
Retraining may not help if the cause is fewer positives in the population; it consumes time and can overfit a temporary mix.
Exceptions
If TPR falls or FPR rises at a fixed threshold, the model's behaviour has changed; investigate drift or data faults.
Trade-offs
Waiting for labels to decompose delays action; a temporary threshold change can buy time.
Pitfalls
Diagnosing with biased labels, such as only labelling alerted cases, which inflates apparent precision and hides misses.
Example
Hypothetical: TPR 0.8 and FPR 0.02 give precision 0.29 at 1% prevalence but 0.17 at 0.5%, with an unchanged model.