Classical ML, validation, metrics, interpretation, debugging · insight-1713
Weight validation folds by the input density ratio when the scoring population differs in its inputs
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry, retail
Summary: situation, action, reason, boundary
Situation
You will score a population whose inputs differ from the training mix, such as a new route network, while the input-to-outcome relationship should hold, and cross-validation looks fine.
Action
Estimate the new-to-training density ratio (for example with a classifier separating the two input sets), clip it, weight each validation row's loss by it, and report both estimates with the effective sample size.
Reason
Under covariate shift ordinary cross-validation loses unbiasedness; importance-weighted cross-validation is unbiased given the true density ratio and an unchanged output relationship.
Boundary
Misspecified or limited-capacity models, with unlabelled inputs from the target population.
Symptom or risk
Cross-validation measures error on the training mix; a misspecified model can be much worse where the new population sits.
Exceptions
If the outcome relationship also changes, reweighting cannot fix it.
Trade-offs
Weights inflate variance; a few rows can dominate.
Pitfalls
Trusting weights from a classifier that separates the populations almost perfectly.
Example
Simulated (known densities, weights clipped at 20, linear fit to a curve, 300 rows, 100 repeats): ordinary 5-fold error 0.35, weighted 0.42, true error on the shifted population 0.44. The weighted estimate had nearly twice the spread (0.073 versus 0.040), effective sample size about 96.
Lessons and concepts
Claims and their sources
Sugiyama, Krauledat and Mueller show that under covariate shift, where the input distribution changes but the conditional distribution of outputs given inputs does not, ordinary cross-validation loses unbiasedness, and importance-weighted cross-validation is proved unbiased.