Feature engineering and preprocessing · insight-0108
Consider CatBoost's ordered target statistics when categorical encoding is the main leakage risk
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, e-commerce
Summary: situation, action, reason, boundary
Situation
A tabular problem is dominated by high-cardinality categoricals and your hand-built target encoding keeps leaking or overfitting.
Action
Benchmark CatBoost with categorical columns declared natively against your fold-safe encoding pipeline on the same splits and metric.
Reason
CatBoost computes target statistics using only rows that precede each row in a random permutation, a design intended to avoid the target leakage and prediction shift of naive encodings.
Boundary
Medium to large tabular data with many categorical columns, such as booking channels, agencies and routes.
Symptom or risk
Encoding bugs dominate the error budget; validation and production disagree because encodings were computed with each row's own label.
Exceptions
A strong, correctly cross-fitted encoding plus LightGBM or XGBoost can match it; native handling is not automatically better.
Trade-offs
Less custom code versus another library dependency, different defaults and slower training on some data.
Pitfalls
Still leaking through time: ordered statistics do not stop a feature computed after the prediction time.
Lessons and concepts
Claims and their sources
CatBoost's ordered target statistics and ordered boosting were introduced to avoid target leakage and prediction shift from standard target statistics.