Feature engineering and preprocessing · insight-0109
Group infrequent levels before one-hot encoding a high-cardinality categorical
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- retail, e-commerce
Summary: situation, action, reason, boundary
Situation
One-hot encoding a feature such as travel agency or product SKU would create thousands of sparse columns.
Action
Set a minimum frequency or maximum number of categories so rare levels share an infrequent column, and benchmark against target or native categorical encoding.
Reason
Levels with few rows cannot support their own coefficient or split reliably; pooling them gives one well-estimated parameter.
Boundary
Linear and kernel models with one-hot inputs; also useful for trees when memory or latency matters.
Symptom or risk
Columns for rare levels add variance, memory and serving cost; new levels after deployment break the transform.
Exceptions
A rare level that matters to the decision (a fraud-prone merchant) should be kept explicitly, not pooled.
Trade-offs
Higher thresholds give smaller, stabler models but discard distinctions; tune the threshold inside cross-validation.
Pitfalls
Choosing the frequency threshold from full-data counts, which lets test rows influence the vocabulary.
Lessons and concepts
Claims and their sources
OneHotEncoder min_frequency and max_categories group infrequent categories; handle_unknown='infrequent_if_exist' maps unseen categories to the infrequent column.