Classical ML, validation, metrics, interpretation, debugging · insight-0150
Try the untouched data plus a cost-based threshold before class weights or resampling
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- banking, telecom
Summary: situation, action, reason, boundary
Situation
Positives are rare and the instinct is to rebalance the training data.
Action
Train on the natural distribution, choose the threshold from costs on validation data, and add class weights or resampling only if they improve the business metric under the same evaluation.
Reason
Cost-optimal decisions can come from probability estimates plus a threshold; rebalancing mainly moves the threshold implicitly.
Boundary
Logistic regression, gradient boosting and random forests with enough positives to learn from.
Symptom or risk
Rebalancing adds a tuning dimension, distorts probabilities and can leak if done outside folds, often for little gain in ranking.
Exceptions
With extremely few positives, or learners that ignore the minority class entirely, weighting can genuinely help the fit.
Trade-offs
Class weights avoid duplicating rows (unlike oversampling) and avoid discarding data (unlike undersampling), but still bias probabilities.
Pitfalls
Comparing a rebalanced model at its default 0.5 threshold with an unbalanced one at 0.5, which mostly compares thresholds.
Lessons and concepts
Claims and their sources
Elkan recommends learning from the training set as given and applying the cost matrix to probability estimates, noting that rebalancing has little effect on Bayesian and decision-tree learners.