Classical ML, validation, metrics, interpretation, debugging · insight-0139
Do not let a high ROC-AUC on rare events hide poor precision at the operating point
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- banking, finance
Summary: situation, action, reason, boundary
Situation
A model for a 0.5% event reports ROC-AUC above 0.9 and the team concludes it is ready.
Action
Show the precision-recall curve and the alert volume, precision and recall at the planned threshold, in counts per day or week.
Reason
ROC curves use rates conditioned on each class and ignore prevalence; precision depends on prevalence and reveals the alert burden.
Boundary
Fraud, intrusion, rare defects, rare diseases and any screening task with low base rates.
Symptom or risk
At any useful recall, false positives can outnumber true positives many times over, because a small false positive rate times a huge negative class is a lot of alerts.
Exceptions
ROC-AUC remains useful for ranking quality comparisons on the same data, especially when prevalence varies across deployments.
Trade-offs
PR metrics are prevalence-specific, so they must be recomputed when prevalence shifts.
Pitfalls
Comparing ROC-AUC between models trained on different resampled class ratios as if it reflected deployment precision.
Example
Hypothetical: 100,000 transactions, 500 frauds, recall 0.8 at FPR 2% gives 400 true and about 1,990 false alerts, precision near 0.17.
Lessons and concepts
Claims and their sources
On imbalanced data the precision-recall plot is more informative than the ROC plot, which can give an optimistic view of performance.