Feature engineering and preprocessing · insight-1103
Shrink a thin route's target encoding toward its origin mean before the global mean
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
Route-level target encoding is shrunk toward the overall rate, yet routes sharing an origin behave alike and new routes keep appearing with a handful of flights.
Action
Encode the parent level first (origin, region), then use the parent's smoothed value as the prior for children: (n * child_mean + m * parent) / (n + m), with m chosen by cross-validation.
Reason
A prior from a related group has lower bias than a prior from everyone, so partial pooling along the hierarchy gives better estimates for small groups.
Boundary
Hierarchies with behavioural meaning: route to origin to region, SKU to brand to category, postcode to city.
Symptom or risk
Smoothing toward the global mean discards shared structure; a sparse route looks like an average route even when its siblings say otherwise.
Exceptions
Hierarchies that are administrative only (alphabetical or contract groupings), or too few parents to estimate a stable prior.
Trade-offs
Extra hyperparameter and custom code; the parent encoding also needs cross fitting.
Pitfalls
Computing parent means on all rows, validation labels included.
Example
Hypothetical: a route with 4 flights and 2 no-shows, m = 20, global rate 0.05, origin rate 0.12. Shrinking to the global rate gives 0.125; shrinking to the origin gives 0.183.
Lessons and concepts
Claims and their sources
scikit-learn's TargetEncoder shrinks each category mean toward the global target mean (smooth parameter) and encodes unseen categories with the global mean.