Classical ML, validation, metrics, interpretation, debugging · insight-0178
Refuse to turn SHAP values into intervention advice without a causal design
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, saas
Summary: situation, action, reason, boundary
Situation
A stakeholder sees that a feature has large SHAP values and asks you to change that feature to improve the outcome.
Action
Present SHAP as model explanation only, name plausible confounders, and propose an experiment or quasi-experimental analysis for any intervention question.
Reason
Attributions decompose a prediction among inputs; they say nothing about what happens when an input is changed in the world.
Boundary
Pricing, marketing, churn, credit and operations decisions informed by predictive models.
Symptom or risk
SHAP describes what the model uses, including proxies and confounded associations; acting on it can do nothing or cause harm.
Exceptions
When features are randomized or causal assumptions are explicitly defended, model-based effects can inform decisions.
Trade-offs
Causal studies cost time; the alternative is acting on correlations.
Pitfalls
Reading a positive price SHAP value in demand data as evidence that raising price raises demand; in the synthetic airline data fares rise when demand is expected to be high.
Lessons and concepts
Claims and their sources
SHAP reveals associations a predictive model learned, which are not necessarily causal; intervention questions need causal assumptions and methods.
SHAP assigns each feature an additive contribution to a single model prediction.