Classical ML, validation, metrics, interpretation, debugging · insight-1190draft — in review
Set drift alerts on an effect size with a minimum sample, since p-values fire on trivial shifts at scale
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A per-feature Kolmogorov-Smirnov or chi-square drift test with p below 0.05 runs daily on a table with millions of rows.
Action
Alert on a distance or effect-size measure (PSI, normalized Wasserstein, share of rows moved), with thresholds calibrated on historical week-to-week variation. Keep p-value tests for small samples, and require a minimum row count per comparison.
Reason
Test statistics grow with the sample while the practical size of a shift does not.
Boundary
High-volume scoring tables and daily monitoring jobs.
Symptom or risk
With very large samples tiny differences are significant, so most features alert every day; the alerts get muted and a real shift is lost among them.
Exceptions
Low-volume segments, where a hypothesis test is the right tool.
Trade-offs
Distance thresholds have no built-in error rate and need calibration data.
Pitfalls
Taking a library's default PSI threshold as universal without checking it against your own noise level.
Lessons and concepts
Claims and their sources
Evidently's data-drift defaults use p-value tests below 1,000 objects (K-S for numerical, chi-square for multi-class categorical) and distance measures above (Wasserstein, Jensen-Shannon), with a default p-value threshold of 0.05 and a default PSI threshold of 0.1.