Classical ML, validation, metrics, interpretation, debugging · insight-0138

Replace accuracy with prevalence-aware metrics when the positive class is rare

Situation
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
Action
Report precision and recall at the chosen threshold, PR-AUC, and the confusion matrix in counts, alongside the prevalence and a dummy baseline.
Reason
With 2% prevalence, predicting all negatives already gives 98% accuracy, so accuracy barely reflects performance on the class you care about.
Boundary
Binary and multiclass problems with skewed class frequencies.

Symptom or risk

Accuracy rises toward the majority share whatever the model does with positives; a useless model looks excellent.

When classes are balanced and errors cost about the same, accuracy is a reasonable summary.

Balanced accuracy fixes the majority bias but still ignores the cost asymmetry between errors.

Swapping accuracy for F1 without checking that F1's equal weighting of precision and recall matches the business costs.

Balanced accuracy averages per-class recall, so a classifier exploiting class imbalance falls toward chance level.
Metrics and scoring: quantifying the quality of predictions (scikit-learn User Guide 3.4)