Feature engineering and preprocessing · insight-1132
Declare which features are discrete before ranking them by mutual information
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You rank features with mutual_info_classif on a dense matrix that contains integer-coded categories next to continuous measurements.
Action
Pass a discrete_features mask or index list, fix random_state, and check the ranking's sensitivity to n_neighbors before using it to cut features.
Reason
The estimator is nonparametric and neighbour-based for continuous variables, and noise is added to remove ties; treating discrete variables as continuous gives incorrect estimates.
Boundary
Mutual-information filters on mixed-type tabular data as a first-stage screen.
Symptom or risk
The default treats every column of a dense matrix as continuous, so category codes are mis-estimated and the ranking is wrong in ways that look plausible.
Exceptions
All-continuous matrices, where the default is correct.
Trade-offs
Higher n_neighbors lowers variance but can add bias, so the ranking is a screen, not a verdict.
Pitfalls
Reading a mutual-information ranking computed on the full dataset as the final selection.
Lessons and concepts
Claims and their sources
mutual_info_classif defaults discrete_features to auto (False for dense input, True for sparse), warns that treating continuous variables as discrete or vice versa usually gives incorrect results, and adds noise to remove repeated values.