Feature engineering and preprocessing · insight-0107
Shrink target encodings of rare categories toward the global mean
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
Many categories have only a handful of rows, for example new routes or long-tail products.
Action
Use a smoothed encoding that blends category mean and global mean by category size, and tune the smoothing strength inside cross-validation.
Reason
Small-sample means have high variance; partial pooling trades a little bias for much lower variance on rare levels.
Boundary
Target encoding with skewed category frequencies; scikit-learn TargetEncoder exposes this through its smooth parameter.
Symptom or risk
A category seen three times gets an extreme encoded value from noise; the model trusts it like a well-measured category.
Exceptions
If rare categories truly differ (a premium route), heavy shrinkage hides it; add a frequency feature so the model can learn reliability.
Trade-offs
More smoothing means more stable but less specific encodings; the right amount depends on noise and category count.
Pitfalls
Smoothing without cross fitting, which reduces but does not remove leakage.
Example
Hypothetical: a route with two bookings and one no-show encodes to 0.5 raw; shrunk toward a 0.08 network rate with a smoothing weight of 20 rows it lands near 0.12.
Lessons and concepts
Claims and their sources
scikit-learn TargetEncoder shrinks category encodings toward the global target mean, controlled by its smooth parameter; unseen categories receive the global mean.