Feature engineering and preprocessing · insight-0101
Skip feature scaling for tree ensembles and spend the effort on features that carry signal
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You are about to add StandardScaler or MinMaxScaler in front of a random forest, gradient boosting model or single decision tree.
Action
Leave numeric features unscaled for tree-only pipelines. Keep scaling only on branches that feed distance, kernel, penalty or gradient-trained models.
Reason
Trees split on thresholds of one feature at a time, so any strictly monotone transformation of a predictor leaves the possible partitions, and therefore the fit, unchanged.
Boundary
CART, random forests, extra trees, GBMs (XGBoost, LightGBM, CatBoost, HistGradientBoosting) on numeric inputs.
Symptom or risk
Extra pipeline steps, serving code and failure points that cannot change the fitted trees, plus a false belief that scaling explains a score change.
Exceptions
Histogram-based GBMs bin features by quantiles, so results can differ slightly by binning; a stacked linear meta-model or k-NN component still needs scaled inputs.
Trade-offs
Removing a step simplifies serving; keeping it can be harmless when one shared pipeline also feeds a linear benchmark.
Pitfalls
Concluding scaling helped because a rerun with a different random seed scored better.
Lessons and concepts
Claims and their sources
Tree-based models are invariant under strictly monotone transformations of individual predictors (ESL section 10.7).