Feature engineering and preprocessing · insight-0134
Drop one dummy level for unpenalized linear regression, keep all levels for penalized models and trees
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- insurance, cross-industry
Summary: situation, action, reason, boundary
Situation
You one-hot encode categoricals for a linear model and must decide whether to set drop='first'.
Action
Drop one level per feature (drop='first' or a chosen reference level) when fitting unpenalized regression for inference; keep all levels with ridge, lasso or tree models and state the convention.
Reason
Full one-hot plus intercept is perfectly collinear; regularization resolves that collinearity, and symmetry across levels keeps shrinkage fair.
Boundary
Linear and logistic regression, GLMs and penalized variants in scikit-learn or statsmodels.
Symptom or risk
Keeping all levels with an intercept makes the design matrix singular in ordinary least squares; dropping levels in a penalized model makes the penalty depend on the arbitrary reference level.
Exceptions
With handle_unknown set to ignore, dropping a level makes unseen categories look like the dropped level; prefer an infrequent bucket instead.
Trade-offs
A reference level eases coefficient interpretation; full encoding eases comparisons across all levels.
Pitfalls
Reading a dropped-level model's coefficients as absolute effects instead of differences from the reference.
Lessons and concepts
Claims and their sources
OneHotEncoder's drop parameter avoids the collinearity that makes the covariance matrix non-invertible in non-regularized regression; with handle_unknown='ignore', unknowns encode like the dropped level.