Classical ML, validation, metrics, interpretation, debugging · insight-0181
Cluster correlated features before trusting permutation importance
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- manufacturing, cross-industry
Summary: situation, action, reason, boundary
Situation
Permutation importance on a held-out set shows every feature in a correlated group as nearly useless.
Action
Cluster features by correlation (for example hierarchical clustering on rank correlation), keep or permute one representative or permute groups together, and recompute importance.
Reason
Permutation importance measures the loss from breaking one feature's link to the target given the others; redundancy masks that loss.
Boundary
Feature pruning, monitoring priorities and explanations with correlated inputs.
Symptom or risk
Permuting one feature leaves its twin intact, so the model loses little; a naive reading drops the whole group.
Exceptions
With nearly independent features, single-feature permutation is reliable.
Trade-offs
Grouped importance is more robust but less granular.
Pitfalls
Permuting a feature creates impossible rows when features are dependent, so even the remaining scores can be biased.
Lessons and concepts
Claims and their sources
When features are correlated, permuting one leaves the information available through the others, lowering both importances; the docs suggest clustering correlated features and keeping one per cluster.
Permutation-based importance with dependent features evaluates the model on unrealistic data points (IML book, permutation feature importance chapter).