Feature engineering and preprocessing · insight-0111
Regularize LightGBM's native categorical splits when categories are many and small
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, aviation
Summary: situation, action, reason, boundary
Situation
You pass a high-cardinality integer-coded categorical to LightGBM as categorical_feature.
Action
Tune min_data_per_group and cat_smooth, and compare against treating the codes as numeric or using a fold-safe target encoding on the same splits.
Reason
Optimal partitioning over many categories searches a huge space of groupings, which overfits when each category has few rows.
Boundary
LightGBM categorical support with dozens to thousands of levels and moderate data volume.
Symptom or risk
Training loss drops fast while validation stalls; the model carves bespoke category groupings that do not generalize.
Exceptions
Low-cardinality categoricals with plenty of rows per level usually benefit from native handling.
Trade-offs
Stronger smoothing reduces overfitting but can blur genuinely distinct categories.
Pitfalls
Encoding categories with negative integers or floats; LightGBM treats negatives as missing and truncates floats.
Lessons and concepts
Claims and their sources
LightGBM docs recommend tuning min_data_per_group and cat_smooth against overfitting with many categories, and note that high-cardinality features often work better treated as numeric or embedded.