Feature engineering and preprocessing · insight-1127

Add a train-versus-live classifier to catch feature drift that per-feature tests miss

Situation
Every per-feature drift test is quiet, yet model quality is slipping, and the features are strongly related to one another.
Action
Train a simple classifier to separate a training sample from a recent live sample, score it on held-out rows, treat an AUC well above 0.5 as shift, and use its feature importances to locate the change.
Reason
A classifier sees combinations of features, so it detects shifts in dependence structure that univariate tests cannot, and its errors point at the responsible features.
Boundary
Tabular feature monitoring with enough rows in both samples and a stable feature schema.

Symptom or risk

The joint distribution has changed, for example a correlation between fare and lead time has flipped, while every marginal still looks the same.

Very small live samples, where the classifier has too little to learn, and shifts you already know are harmless.

A discriminator flags any difference, harmful or not, so it needs a harm check before alerting.

Letting the classifier see an ID or timestamp column, which separates the samples trivially.

Failing Loudly finds that domain-discriminating methods help characterize a shift and judge whether it is harmful, while two-sample tests on pretrained-classifier representations performed best at detection.
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift