Feature engineering and preprocessing · insight-1105
Replace a high-cardinality code with its descriptive attributes so new codes inherit meaning
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, banking
Summary: situation, action, reason, boundary
Situation
A model uses airport, aircraft-type or merchant codes with hundreds of levels, and new codes arrive with no history.
Action
Join a maintained reference table (region, hub size, runway length, merchant category) and use those columns, keeping the raw code only if it wins on a split that holds out unseen codes.
Reason
Attributes are shared across levels, so an unseen level borrows strength from similar ones, whereas an identity code carries no information about a level never seen.
Boundary
Codes for which a reference table exists and has an owner.
Symptom or risk
Level-specific encodings need history, so a new code gets the global average even though it closely resembles known ones.
Exceptions
Stale reference data, or attributes that are themselves outcomes of the process being predicted.
Trade-offs
The reference table must be versioned and kept up to date; attributes can also duplicate each other.
Pitfalls
Joining today's attributes to historical rows, such as a hub classification that changed last year.