Classical ML, validation, metrics, interpretation, debugging · insight-0156

Adjust predicted probabilities when the class prior shifts but the class-conditional patterns do not

Situation
Prevalence in production differs from training, for example no-show rates rising in a holiday season or fraud rates after a new attack wave.
Action
Estimate the new prior (from recent labels or with black-box shift estimation), reweight posteriors by the ratio of new to old priors and renormalize, then confirm calibration as labels arrive.
Reason
Under label shift, $p(x\mid y)$ is unchanged, so posteriors can be corrected by Bayes' rule with the new priors without retraining.
Boundary
Classification where the mix of classes changes faster than what each class looks like.

Symptom or risk

Probabilities stay anchored to the training prior; thresholds and capacity plans based on them misfire.

If the behaviour of each class also changes (concept drift), prior correction is insufficient; retrain.

Prior estimates from few recent labels are noisy; smooth them over time.

Applying prior correction to a model that was already rebalanced without accounting for the training sampling ratio.

Classifier posteriors trained under one class prior are not valid under new priors and can be adjusted to the new priors.
Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure
Label shift is defined as p(y) changing while p(x|y) stays fixed; black box shift estimation estimates the new label distribution from a fixed predictor's confusion matrix.
Detecting and Correcting for Label Shift with Black Box Predictors