Feature engineering and preprocessing · insight-1128
Judge feature drift by effect size, because p-values fire on negligible shifts at production volumes
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A daily drift job runs significance tests on millions of rows per feature and marks most features as drifted most days.
Action
Switch to an effect-size measure such as PSI or a normalised distance with a threshold set from the historical day-to-day variation, and report the share of rows or the size of the shift.
Reason
Test power grows with sample size, so a negligible shift becomes significant at large n; an effect size does not.
Boundary
High-volume scoring pipelines and per-feature dashboards.
Symptom or risk
Alert fatigue; real shifts drown among statistically significant but practically meaningless ones, and the team learns to ignore the monitor.
Exceptions
Small-sample monitors, such as a weekly job on a few hundred rows, where tests are appropriately cautious.
Trade-offs
Thresholds are judgments that must be calibrated on stable periods and revisited.
Pitfalls
Taking a default threshold as universal, or comparing PSI values across features with different binning.
Lessons and concepts
Claims and their sources
Evidently's drift defaults vary with sample size (KS test up to 1,000 rows, normed Wasserstein above), flag drift at p below 0.05, and offer PSI with a default threshold of 0.1.