Classical ML, validation, metrics, interpretation, debugging · insight-1715
Compute the prevalence and cost range where a thresholded classifier beats alerting on everyone or no one
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- aviation, banking
Summary: situation, action, reason, boundary
Situation
A fixed-threshold classifier tuned at one prevalence and cost ratio will be reused where either differs, such as another route group.
Action
With PC(+) = p * cost_FN / (p * cost_FN + (1 - p) * cost_FP), compute where the operating point beats both trivial rules: FPR / (TPR + FPR) up to TNR / (FNR + TNR). Convert to prevalence and mark where each segment sits.
Reason
On a cost curve a classifier is a straight line in PC(+) and the trivial classifiers are the diagonals; its operating range is where it beats both.
Boundary
Alerting rules with asymmetric costs and varying base rates.
Symptom or risk
One AUC hides that outside some range the model costs more than never or always alerting.
Exceptions
A threshold re-chosen per segment has its own range.
Trade-offs
Assumes TPR and FPR hold across the range.
Pitfalls
Reading a high AUC as a wide range.
Example
Hypothetical costs, Gaussian two-class score (AUC 0.86) tuned at 5% prevalence, a miss costing 10 false alerts: TPR 0.63, FPR 0.12. It wins for prevalence 1.9% to 19%. At 0.5% never alerting is cheaper (0.05 versus 0.14 false-alert units per case); at 30%, alerting everyone is (0.70 versus 1.20).
Lessons and concepts
Claims and their sources
Drummond and Holte define normalized expected cost as FN * PC(+) + FP * PC(-), a straight line in PC(+), where PC(+) is p(+)C(-|+) divided by p(+)C(-|+) + p(-)C(+|-); the always-negative and always-positive classifiers are the two diagonals, and the operating range is where a classifier beats both.