Classical ML, validation, metrics, interpretation, debugging · insight-0149

Recalibrate after resampling or class weighting, because both inflate predicted risk

Situation
You trained with SMOTE, undersampling, class_weight='balanced', is_unbalance or scale_pos_weight, and now use predict_proba.
Action
Recalibrate on held-out data with the true class distribution, or correct for the known sampling ratio; if probabilities matter more than ranking, try training without rebalancing.
Reason
Rebalancing changes the effective prior the model learns, so posterior probabilities are shifted toward the minority class.
Boundary
Risk scores, no-show and fraud probabilities, any expected-value decision built on rebalanced models.

Symptom or risk

Probabilities reflect the artificial training prevalence; predicted risk for the rare class is overestimated and expected-cost thresholds misfire.

If only ranking or a tuned threshold is used, discrimination may be fine without recalibration.

Recalibration needs untouched held-out data; prior correction is cheap but assumes the sampling ratio is known exactly.

Calibrating on a resampled validation set, which reproduces the distortion.

Imbalance corrections (random over/undersampling, SMOTE) produced strongly miscalibrated, overestimated risks without improving ROC-AUC in a simulation and case study.
The harm of class imbalance corrections for risk prediction models: illustration and simulation using logistic regression
LightGBM's is_unbalance and scale_pos_weight can improve overall metrics but give poor estimates of individual class probabilities; XGBoost advises not rebalancing when correct probabilities are needed.
Parameters (LightGBM documentation)Notes on Parameter Tuning (XGBoost documentation)