SQL, data modeling, data quality, EDA, visualization · insight-0061
Track distinct-value counts per column as an early warning for upstream changes
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data analyst, ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model or dashboard depends on categorical columns fed by systems you do not own.
Action
Record the distinct count and the top values of key categorical columns per load, alert on large changes or unseen values, and route new categories to an explicit "other" bucket in models until reviewed. Before grouping by or one-hot encoding a column, check whether it is really an identifier with near one value per row.
Reason
Cardinality is cheap to compute and sensitive to many silent failures: mapping changes, truncated loads, default values and ID columns mistaken for categories.
Boundary
Feature pipelines, BI models and partner feeds.
Symptom or risk
A new code, a renamed category or a column that suddenly holds one value breaks encodings and segment KPIs without any schema error.
Exceptions
Columns expected to grow, such as customer ids; monitor growth rate rather than level.
Trade-offs
Alerts on every new value create noise; tune thresholds per column.
Pitfalls
Monitoring only the training-time category list, so new production values fall silently into an unknown category.