Classical ML, validation, metrics, interpretation, debugging · insight-0140

Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates

Situation
You compare PR-AUC of a churn model across countries, months or customer tiers.
Action
Report each PR-AUC next to its segment prevalence (the random-classifier baseline), and use a prevalence-independent metric such as ROC-AUC for cross-segment ranking comparisons.
Reason
A random classifier's precision equals the prevalence, so PR-AUC's floor moves with the base rate.
Boundary
Segment reports, monthly monitoring and model comparisons on different datasets.

Symptom or risk

A segment with higher churn prevalence shows higher PR-AUC even if the model ranks equally well there; teams chase the wrong segment.

Comparing models on the same evaluation set is fine; prevalence is then identical.

Normalized versions (lift over prevalence) are easier to compare but less familiar to stakeholders.

Calling a model worse after a month with fewer positives, when only the baseline moved.

The baseline of a precision-recall curve is the positive fraction P/(P+N), unlike the fixed diagonal baseline of the ROC curve.
The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets