Feature engineering and preprocessing · insight-0103
Standardize predictors before ridge, lasso or elastic net so the penalty is unit-free
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You fit a penalized linear or logistic model where features are measured in different units.
Action
Standardize numeric predictors inside the pipeline before the penalized estimator, then tune the penalty strength on a log-spaced grid with cross-validation.
Reason
Least-squares coefficients rescale with their feature, but penalized coefficients do not: the penalty acts on coefficient size, which depends on the feature's unit.
Boundary
Ridge, lasso, elastic net, penalized logistic regression and scikit-learn's LogisticRegression, which is regularized by default.
Symptom or risk
The penalty shrinks small-unit features hard and barely touches large-unit ones; selected features change when someone converts kilometres to miles.
Exceptions
One-hot dummies are often left unscaled so the penalty treats each level alike; document whichever convention you choose.
Trade-offs
Standardized coefficients compare features per standard deviation, which is less natural for business readers than per-unit effects; back-transform for reporting.
Pitfalls
Interpreting shrunken coefficients as unbiased effect sizes, or comparing coefficients across models with different penalties.
Lessons and concepts
Claims and their sources
Ridge coefficient estimates are not scale-equivariant, so predictors are usually standardized before fitting (ISL section 6.2.1).
scikit-learn's LogisticRegression applies regularization by default; no regularization corresponds to a very large C.