Classical ML, validation, metrics, interpretation, debugging · insight-0183

Establish that the model predicts well before discussing which features it relies on

Situation
An importance plot is requested for a model whose held-out score is close to the baseline.
Action
Show the held-out score against the baseline first; present importances only for models with clear lift, and caveat them as model-specific.
Reason
Importance measures what a particular model uses, not the intrinsic predictive value of a feature; a feature unimportant to a bad model may matter to a good one.
Boundary
Driver analyses, feature-importance slides and model documentation.

Symptom or risk

Importances of a weak model describe noise it fitted; stakeholders draw conclusions about drivers from a model that learned little.

Debugging a weak model can still use importances to find leakage or broken features.

Holding back importances can frustrate stakeholders; explain that driver claims need a model worth explaining.

Using importance from one model family to argue that a feature has no predictive value at all.

Features unimportant to a poorly scoring model may be very important to a good one; evaluate predictive power on held-out data before computing importances.
Permutation feature importance (scikit-learn User Guide 5.2)