Engineering, production, governance, cost, communication · insight-0262
Monitor distributions to catch semantic changes that schemas cannot see
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer, Data analyst
- Industries
- retail, finance, e-commerce
Summary: situation, action, reason, boundary
Situation
A price column switches from cents to currency units in a release; schema, types and null rates are unchanged.
Action
Track per-feature summary statistics (quantiles, category frequencies) against a reference window at each load, with alerts on large ratio changes, and require producers to announce unit or code changes in the contract.
Reason
Semantic changes show up as distribution shifts even when structure is identical.
Boundary
Monetary, unit-bearing and code-list features, especially from external vendors.
Symptom or risk
Every structural check passes while the model sees values 100 times smaller and its predictions shift.
Exceptions
Expected seasonal shifts need seasonal reference windows to avoid false alarms.
Trade-offs
Distribution checks need thresholds that balance noise against sensitivity.
Pitfalls
Comparing against the previous day only, which misses slow creep.
Lessons and concepts
Claims and their sources
Evidently provides data-drift and data-quality reports that compare current data with a reference dataset.