Core day 17 of 60 · Week 4 · Part I · application depth

Feature Selection and Interpretation

A random forest scores bookings for no-show risk. Its importance chart puts `business_trip` first, and a product manager wants to waive change fees for business travellers to cut no-shows. Which plots and importances describe the model, which describe the world, and what would you check before anyone acts?

By the end: For a fitted tree model you choose between impurity importance, permutation importance, PDP, ICE, ALE and SHAP for a stated question, name what each hides, and state why none of them is an intervention effect.

At a glance

Importances, effect plots and SHAP describe what a fitted model relies on; pick the tool by the question and never read it as a cause.

Check the model first
Explain only a model that scores well on held-out rows; otherwise you explain noise.
Rank on held-out data
Permutation importance on held-out rows; impurity importance favours many-valued columns.
Group duplicates
Correlated features cover for each other; shuffle or report them as one block.
ICE beside every PDP
An average curve can be flat while individual curves are steep; ALE for tied features.
Reliance is not effect
Proxies and duplicates earn credit; only an experiment or causal design says what a change does.

Recall · 5 min

Without notes: (1) why can a coefficient or an importance not be read as the effect of changing a feature (Day 5)? (2) which preprocessing steps may look at the label, and where must they run in a cross-validated pipeline (Days 9 and 15)? (3) what does a random forest do with two nearly identical columns (Day 12)? Then clear your due cards.

Open due reviews

Concept · 10 min

An importance chart answers "what does this fitted model rely on?", not "what drives defaults, churn or no-shows?". Every tool here probes a function learned from correlations. A feature earns credit as a cause, a proxy, a duplicate or an artefact of counting. Like a weather vane, the tools tell the truth about the wind, not about turning the vane.

An average slope hides two segment effects, and a correlated bystander earns importance

Predict first

A feature's effect is +1 per unit for half the rows and -1 per unit for the other half. What does its PDP look like?

Text description of the figure

A ridge model fitted live to a toy table with an interaction (x1 by a two-valued segment x3) and an irrelevant feature x2 correlated with x1. A method switch shows a PDP with ICE curves, permutation importance, or exact additive attributions for one row. With the interaction on, the PDP shows one average slope while the ICE curves split into two families. Raising the correlation gives x2 importance and a coefficient although it has no effect.

Six tools, six different questions Overview

Impurity importance
How much split gain did training give each feature? Training rows only.
Permutation importance
How much does the held-out score fall when one column is shuffled?
PDP
Average prediction as one feature is set to each value for every row.
ICE
The same curve drawn for each row, so segments with opposite slopes show.
ALE
Accumulated local change within narrow windows, using only realistic rows.
SHAP
How one prediction's gap from the average prediction is shared among features.

Pick the tool by the question, after the model earns it Diagram

Fix the model firstexplaining a poor model explains noise
Good held-out score?
- no: back to Fix the model first
yes
What is the question?
- to An effect plot
- to SHAP for one row
Grouped permutationwhich features matter? held-out, repeated
An effect plothow does the prediction move with x?
- to x tied to others?
SHAP for one rowwhy this prediction? shares of its gap
x tied to others?
- no: to PDP with ICE
yes
ALEstays inside the data
PDP with ICEaverage plus every row

Every branch ends in a description of the fitted model. None of them ends in an intervention effect.

A flat average hiding two steep families Illustration

What a PDP report shows: an average slope of +0.07 per unit. Read alone, it suggests lead time is irrelevant.

The same PDP (red) drawn over its 48 ICE curves: most bookings react strongly, in one of two directions.

Centered at the left edge, the curves split by trip type: leisure -0.53 per unit, business +0.54. The average of the two is the flat PDP.

Synthetic bookings from the lesson's generator plus a mild fare effect (500 to fit, 48 held out); a 30-tree forest fitted with the app's own tree code. Curves are model predictions, not effects of changing lead time.

Left to right: the PDP alone, the PDP over one curve per held-out booking, and the same curves centered. The forest relies heavily on lead time; the average cancels it.

Describing the model versus changing the world Comparison

Criterion	Model reliancewhat explanation tools answer	Intervention effectwhat a decision needs
Question	What does this fitted churn model use to predict?	What happens to churn if we change this?
Evidence	Importances, PDP/ICE/ALE, SHAP on held-out rows	A randomized test or an explicit causal design (Day 5)
Honest wording	The model relies on contract_type	Annual-contract offers cut churn by x in the test
Typical mistake	Reading the top bar as the cause of the outcome	Skipping the test because the chart looked clear

Four reasons a feature earns credit Overview

Cause
It changes the outcome. Only this one supports an action, and only a test shows it.
Proxy
It travels with a cause, such as postcode standing in for income in a credit model.
Duplicate
It copies another column; credit between the pair is arbitrary.
Artefact
The tool rewards it: many split points, training rows, memorized noise.

Say "the model relies on X" and attach the experiment that would show what changing X does. The SHAP documentation works through confounded and redundant features whose attributions do not match the true effects.

Key terms: Impurity importance, Permutation importance, Partial dependence plot (PDP), ICE plot (individual conditional expectation), ALE plot (accumulated local effects), SHAP value, Multicollinearity and Causal effect. Which tool answers which question, and how each one fails, is in the pictures above and the optional technical depth below.

Assumptions

The model generalizes; importances of an overfit or poorly scoring model describe its memorization.
Held-out rows come from the deployment distribution, so permuting one column creates realistic rows only when it is not tightly tied to others.
Explanations are stable only if the model is; refit with another seed or an equally good model and the ranking can change among correlated features.

Application and workplace decision · 10 min

4,000 synthetic bookings, a random forest, and four probes of the same model. The generator has:

an interaction: lead time raises the no-show score for leisure trips and lowers it for business trips;
a small real weekend effect, a near-duplicate of lead time and a pure-noise booking reference.

The script prints impurity importance, held-out permutation importance, a joint shuffle of the duplicate pair, and PDP and ICE slopes. Before running it, decide which ranking you would trust for the weekend flag and the noise column.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance

rng = np.random.default_rng(17)
n = 4000
biz = rng.integers(0, 2, n)                                    # 1 = business trip, 0 = leisure (synthetic)
lead = rng.uniform(-1, 1, n)                                   # standardized booking lead time
X = pd.DataFrame({"business_trip": biz, "lead_time": lead,
                  "advance_purchase": lead + rng.normal(0, 0.1, n),   # near-duplicate field, unused by the generator
                  "weekend": rng.integers(0, 2, n),
                  "booking_ref": rng.uniform(0, 1, n)})        # pure noise with many distinct values
y = (2 * biz - 1) * lead + 0.25 * X["weekend"] + rng.normal(0, 0.3, n)  # no-show score: opposite lead-time slopes
tr, te = slice(0, 3000), slice(3000, n)
rf = RandomForestRegressor(200, min_samples_leaf=5, random_state=0, n_jobs=-1).fit(X[tr], y[tr])
print("held-out R2:", round(rf.score(X[te], y[te]), 3))
print("impurity importance:", dict(zip(X, rf.feature_importances_.round(3).tolist())))
r = permutation_importance(rf, X[te], y[te], n_repeats=10, random_state=0)
print("permutation drop in R2:", dict(zip(X, r.importances_mean.round(3).tolist())))
both = X[te].copy()
both[["lead_time", "advance_purchase"]] = both[["lead_time", "advance_purchase"]].to_numpy()[rng.permutation(1000)]
print("lead_time + advance_purchase shuffled together:", round(rf.score(X[te], y[te]) - rf.score(both, y[te]), 3))

grid = np.linspace(-1, 1, 21)                                  # ICE: one curve per held-out row; the duplicate moves with lead_time
ice = np.array([rf.predict(X[te].assign(lead_time=v, advance_purchase=v)) for v in grid])
slope = lambda cols: np.polyfit(grid, ice[:, cols].mean(axis=1), 1)[0]
is_biz = X[te]["business_trip"].to_numpy() == 1
print(f"PDP slope {slope(slice(None)):+.2f} | ICE slope leisure {slope(~is_biz):+.2f}, business {slope(is_biz):+.2f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 42 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Permutation importance: definition, algorithm, a shuffle by hand7 minWhat the number is exactly, how it is computed, four rows shuffled on paper, and the properties that decide when to trust it.

Permutation importance, defined Formula

\mathrm{PI}_j = \frac{1}{K}\sum_{k=1}^{K} e\big(\hat f;\ X^{(\pi_k, j)}, y\big) - e\big(\hat f;\ X, y\big)

$e(\hat f; X, y)$: the frozen model's error (here MSE) on held-out rows X with labels y; nothing is refitted
$X^{(\pi_k, j)}$: X with column j reordered by the k-th random permutation; every other column untouched
$K$: number of shuffles (n_repeats); the average estimates the expected error under a random shuffle
$\mathrm{PI}_j$: the rise in error; for a score such as R-squared, scikit-learn reports score before minus score after

Breiman (2001) introduced it for forests; Fisher et al. (2019) call it model reliance. It is a property of one fitted function, one score and one set of rows, not of the feature in the world.

Permutation importance, step by step Sequence

Fit
Fit on training rows; freeze the model.
Score held-out rows
Compute the reference score s on rows the model never saw.
Shuffle one column
Permute column j across held-out rows; this breaks its link with the target.
Rescore
Score again with the same frozen model; no refitting.
Repeat and average
Repeat K shuffles; report the mean drop and its spread.

Permutation importance by hand: four held-out rows, a model that reads only x1 Table

Row	x1	y	Prediction 2 x1	After shuffling x1	Squared error after
1	1	2	2	4 (x1 = 2)	4
2	2	4	4	6 (x1 = 3)	4
3	3	7	6	8 (x1 = 4)	1
4	4	8	8	2 (x1 = 1)	36
MSE	-	-	0.25	-	11.25

Frozen model f = 2 x1 for a grocery customer's weekly orders (illustrative). The shuffle moves x1 one row down: importance 11.25 - 0.25 = 11.0. The unused x2 scores 0 because this model never reads it, not because x2 has no effect on orders. Over all 24 orderings the mean permuted MSE is 10.75; the identity ordering (1 in 24) gives 0.

Held-out permutation importance (drop in R-squared) Chart

business_triplarger than R2 0.768: shuffled model is worse than the mean1.411 R2 drop
lead_time0.893 R2 drop
advance_purchasecovered by its duplicate0.091 R2 drop
weekend0.072 R2 drop
booking_refnoise correctly near zero0.002 R2 drop
lead_time + duplicate togethermore than 0.893 + 0.0911.4 R2 drop

Output of the application script, 10 repeats on 1,000 held-out rows.

One shuffle is one draw Illustration

a real but small effect
a pure-noise ID column
dots: single shuffles; black tick: mean; band: mean ± 2 sd

Synthetic bookings from the lesson's generator plus a uniform noise column; a 20-tree forest from the app's tree code, scored with its permutation-importance helper on fresh held-out samples of 100 and 800 rows.

At 100 held-out rows the two columns overlap: some shuffles of the noise ID beat some shuffles of the real weekend flag. At 800 rows they separate cleanly. Report the mean and the spread over repeats, never one shuffle.

Each dot is one shuffle of one column. Fewer held-out rows widen the spread until a small real effect and a noise ID overlap. Report the mean drop with its spread; scikit-learn's default of 5 repeats is a floor, not a guarantee.

Four ways to take a column away, four different questions Comparison

Criterion	Permutation	Grouped permutation	Conditional permutation	Drop-column
What changes	One column shuffled across held-out rows	Tied columns shuffled with one shared permutation	Shuffle only within groups of similar correlates (Strobl et al., 2008)	Column removed from the data
Refit?	No: the frozen model	No	No	Yes, once per feature
Extra predictions	K x n per feature	K x n per group	More: groups must be built first	A full refit each
Answers	This model's reliance; a tied copy covers for it	Reliance on the block; rows stay realistic	Reliance beyond what correlates carry; a duplicate earns less	What a different model loses without it

Shuffling one of two tied columns creates rows outside the data while the copy covers for it: check a correlation matrix, then shuffle the group. Fisher et al. (2019) report reliance as a range over all near-optimal models (model class reliance) instead of one model's number.

PDP, ICE and ALE: three averages, worked on a tiny grid8 minThe exact definitions, a three-row grid pushed through all three by hand, and the assumption each one needs.

Partial dependence and ICE Formula

\hat f_j(v) = \frac{1}{n}\sum_{i=1}^{n} \hat f\left(v,\ x_{i,-j}\right)

$\hat f(v, x_{i,-j})$: ICE curve of row i: its other features kept, feature j set to v
$\hat f_j(v)$: PDP; the average of all ICE curves at v
$x_{i,-j}$: row i's other features; realistic only if they do not depend on x_j

Centered ICE (each curve shifted to start at zero) shows slopes. If f = g(x_j) + h(others), every ICE curve is g shifted and PDP and ALE both equal g up to a constant: non-parallel ICE curves signal an interaction. The lesson's script moves lead time and its duplicate together, so its ICE rows stay realistic.

Accumulated local effects, written out Formula

\hat f_{j}^{\mathrm{ALE}}(v) = \sum_{k=1}^{k(v)} \frac{1}{n_k}\sum_{i:\, x_{ij} \in N_k} \big[\hat f(z_{k}, x_{i,-j}) - \hat f(z_{k-1}, x_{i,-j})\big] - c

$N_k = (z_{k-1}, z_k]$: the k-th window of x_j, usually between quantiles so each holds rows
$i: x_{ij} \in N_k$: only the n_k rows whose own x_j falls in that window are moved
$\hat f(z_k, \cdot) - \hat f(z_{k-1}, \cdot)$: a local change across one narrow window, with the row's other features kept
$c$: a constant that centers the curve to average zero over the data

The PDP moves every row across the whole range of x_j; ALE moves each row only across its own window (Apley and Zhu, 2020).

PDP, ICE and ALE as plain averages on a three-row grid Table

Grid value v of x1	ICE row B (x2 = 1)	ICE row C (x2 = 2)	PDP (mean of ICE)	ALE (uncentered)
0	0	0	0	0
1	1	2	1	0
2	2	4	2	1
3	3	6	3	3

Toy fitted model f = x1 x2: extra units sold = discount depth x1 times loyalty tier x2 (illustrative retail promotion). Rows A (0.5, 0), B (1.5, 1), C (2.5, 2): deeper discounts go to higher tiers, so x1 and x2 are tied. Verified in Python.

The three-row grid, worked line by line Sequence

Windows
Cut x1 into (0, 1], (1, 2] and (2, 3]. Each window holds exactly one row: A, B and C.
ICE
Keep each row's own x2 and set x1 = v. Row B: f(v, 1) = v, so 0, 1, 2, 3. Row C: 2v. Row A, with x2 = 0: flat at 0.
PDP
Average the three ICE values at each v: (0 + v + 2v) / 3 = v. A straight line of slope 1.
The impossible rows
At v = 3 the PDP uses f(3, 0) for row A: the deepest discount for a tier-0 customer, a combination the data never show.
ALE local changes
Move each row only across its own window. A: f(1, 0) - f(0, 0) = 0. B: f(2, 1) - f(1, 1) = 1. C: f(3, 2) - f(2, 2) = 2.
Accumulate
Running sum at the window edges: 0, 0, 1, 3. The local slope grows 0, 1, 2 because, where the rows really are, x2 rises with x1.
Center
Subtract one constant so the curve averages zero over the data (Apley and Zhu, 2020). The shape does not change.
Read both
PDP: slope 1 everywhere. ALE: flat at shallow discounts, steep at deep ones. Both describe f; only ALE stays on realistic rows.

Tied inputs: the PDP extrapolates, ALE stays on the data Illustration

To draw the PDP at x1 = 0.9, every row's x2 is kept: 222 of 300 query points (red) lie more than 0.15 from the data diagonal, where the model never saw a row.

Dashed: the rule that generated y, +3 per unit of x1. Largest gap from it: PDP 1.20, ALE 0.28.

Largest gap from it: PDP 0.67, ALE 0.27.

Largest gap from it: PDP 1.07, ALE 0.29.

Synthetic data: 300 rows per refit, x1 uniform, x2 = x1 + noise (sd 0.05), y = 3 x1 + noise. Model: least squares on x1, x2, their squares and product. PDP uses the app's partial-dependence code; ALE uses 10 quantile windows. Curves start at 0. Panels show refits 2, 6 and 8 of 8 fresh samples; across all 8 the PDP strays up to 3.56 from the generating line and the ALE up to 0.58.

With x1 and x2 tied (correlation about 0.985) the PDP averages predictions at points no row resembles, so noise in the fitted coefficients there sets its shape. ALE averages local changes inside narrow windows of real rows.

PDP, ICE and ALE side by side Comparison

Criterion	PDP	ICE	ALE
Shows	The average response to x_j	Every row's response; segments and interactions	Accumulated local effect of x_j
Hides	Opposite effects cancel; flat can mean strong	Overplots; no single summary	Heterogeneity between rows (it is an average)
Assumes	x_j independent of the other features	Same as PDP, row by row	Enough rows per window
With tied features	Averages predictions at impossible combinations	Same impossible rows, one per curve	Uses only rows near their real values; noisy in sparse windows
Cost in predictions	n x G (rows times grid points)	The same n x G, kept row by row	About 2n: two per row

All three describe the fitted function; ALE is the safer choice when the feature is correlated with others (Apley and Zhu, 2020). Check ALE by counting rows per window, with quantile windows. Two-feature PDPs show interactions directly but get unreliable in sparse corners of the grid.

Shapley values: the axioms and a two-feature computation by handadvanced8 minThe formula, the four axioms that make it unique, an insurance toy model computed on paper, and what the axioms do not promise.

Where the shares come from (Shapley value) Formula

\phi_j = \sum_{S \subseteq F \setminus \{j\}} \frac{|S|!\,(|F|-|S|-1)!}{|F|!}\,\big[v(S \cup \{j\}) - v(S)\big]

$S$: a coalition of features already known
$v(S)$: expected prediction when only the features in S are known
$v(S \cup \{j\}) - v(S)$: what adding feature j to that coalition changes
$weights$: average over every order in which features could be added

How v(S) fills in the unknown features (marginal or conditional) is a choice: the values are exact for that v, and a conditional v gives different numbers when features are tied. Shares are in model output units, often log-odds, so a share of 0.4 is not 0.4 of probability.

Four axioms, and the Shapley value is the only rule meeting all of them Overview

Efficiency
The shares add up exactly to the prediction minus the baseline v of the empty set.
Symmetry
Two features that add the same to every coalition get the same share.
Dummy
A feature that never changes v(S), whatever S it joins, gets zero.
Additivity
For a model that is a sum of two models, each share is the sum of the two shares.

From cooperative game theory; Lundberg and Lee (2017) restate them for models as local accuracy, missingness and consistency. Every axiom is about v, not the world: a proxy the model reads gets credit; a cause it ignores gets none.

Shapley values by hand: two features, four background rows Table

Coalition S	v(S)	How it is computed
empty	3.5	Mean of f over the background: (0 + 2 + 3 + 9) / 4
young	5.5	young = 1 fixed, urban taken from the background: (2 + 9) / 2
urban	6.0	urban = 1 fixed, young from the background: (3 + 9) / 2
young and urban	9.0	The prediction itself, f(1, 1)

Toy insurance model f = 2 young + 3 urban + 4 young x urban, expected claim cost in EUR 100s (illustrative). Background: the four combinations of two binary features. Explained row: a young urban driver. Verified with the Exact explainer in shap 0.52.

From coalition values to two shares Sequence

Two orders
With two features there are two orders in which they can join: young first, or urban first. The Shapley value averages over both.
Young joins first
v(young) - v(empty) = 5.5 - 3.5 = 2.0.
Young joins second
v(young and urban) - v(urban) = 9.0 - 6.0 = 3.0.
Share of young
(2.0 + 3.0) / 2 = 2.5.
Share of urban
First: 6.0 - 3.5 = 2.5. Second: 9.0 - 5.5 = 3.5. Average: 3.0.
Efficiency check
Baseline 3.5 + 2.5 + 3.0 = 9.0, exactly the prediction.
Where the interaction went
The 4 x young x urban term sits 3 above its background mean; Shapley splits it 1.5 and 1.5, on top of 1.0 and 1.5 from the main terms.
Cost
Exact values need v(S) for all 2^M coalitions: 4 here, about a million at M = 20. TreeSHAP uses tree structure; KernelSHAP samples.

From the baseline to this driver's prediction, share by share Bridge

Baseline: mean over background3.5 EUR 100s
young = 11.0 main term + 1.5 of the interaction+2.5 EUR 100s
urban = 11.5 main term + 1.5 of the interaction+3 EUR 100s
Prediction f(1, 1)9 EUR 100s

The shares from the steps above, stacked. The bridge closing exactly at the prediction (9.0) is the efficiency axiom; the split itself is unique only because all four axioms hold together.

SHAP is additive around a baseline Formula

\hat f(x) = \phi_0 + \sum_{j=1}^{M} \phi_j(x), \qquad \phi_0 = \mathbb{E}\big[\hat f(X)\big]

$\phi_0$: baseline; the average prediction over the background data
$\phi_j(x)$: feature j's share of this row's gap from the baseline, in model output units
$\sum \phi_j$: local accuracy (efficiency): the shares add up exactly to the prediction minus the baseline

Change the background data and the baseline and every share changes. For classifiers the units are often log-odds, not probability.

Impurity importance and why it favours many-valued columns3 minWhere the default feature_importances_ number comes from, and why a random ID can tie with a real effect.

Impurity (mean decrease in impurity) importance Formula

\mathrm{MDI}_j = \frac{1}{T}\sum_{t=1}^{T} \sum_{\nu \in t,\ \text{split on } j} \frac{n_\nu}{n}\,\Delta i(\nu)

$\Delta i(\nu)$: impurity drop at node nu (Gini or squared error) measured on the training rows
$n_\nu / n$: share of training rows that reach the node
$T$: number of trees; scikit-learn normalizes each tree first, then rescales the average to sum to 1

Free with the fit, but computed on training rows, so memorized noise and many-valued columns collect gain.

Impurity importance in the lesson's forest Chart

lead_time0.521 share
business_trip0.241 share
advance_purchase (unused)no role in the generator0.156 share
weekend (real effect)0.042 share
booking_ref (noise)almost equal to a real effect0.04 share

Output of the application script: a pure-noise ID scores like the real weekend effect. A many-valued column offers many thresholds, and the best of many random cuts still shows gain (Strobl et al., 2007); the gain is measured on training rows, so memorized noise collects credit. A debugging hint, never the ranking you report.

SHAP in practice: explainers and the background choiceadvanced4 minTreeSHAP versus KernelSHAP, and the interventional versus observational choice that decides how tied features split credit.

TreeSHAP versus KernelSHAP Comparison

Criterion	TreeSHAP	KernelSHAP
Models	Tree models and tree ensembles	Any model (model-agnostic)
Exactness	Exact for the tree model	Approximation: weighted regression on sampled coalitions
Cost	Polynomial in tree size and depth; fast	Slow; grows with features and background size
Missing features filled by	tree_path_dependent (training cover) or interventional (background)	Background rows, sampled as if features were independent

From the SHAP papers and API documentation. Pass a small, representative background set and state which option you used.

Interventional versus observational SHAP: true to the model or to the data Comparison

Criterion	Interventional (marginal)	Observational (conditional)
Unknown features drawn from	Background rows, ignoring correlation	Rows consistent with the known features
Credit for an unused duplicate	Zero if the model never reads it	Can be non-zero: it carries the same information
Risk	Evaluates impossible combinations, like a PDP	Hard to estimate; credit spreads over correlates

Here "interventional" means intervening on the model's inputs; it is still not the effect of changing the feature in the world. The usual global summary, the mean absolute SHAP value per feature, averages away signs as a PDP averages slopes.

Importance is not a causal effect3 minWhy proxies and duplicates earn credit, and how to place a feature on the reliance and effect axes.

Reliance and effect are two different axes Map

Hidden lever

The model reads a correlate instead; an experiment can still find the effect.

Candidate lever

Worth an experiment to size the effect before acting.

Irrelevant

Can be dropped if validated selection agrees.

Proxy

Useful for targeting and prediction; changing it does nothing.

An importance chart places a feature on the horizontal axis only. The vertical axis needs a design (Day 5).

Feature selection without fooling yourself3 minEvery label-aware selection step is part of the fit and must run inside the training folds.

Feature selection: three families, one rule Table

Family	How it chooses	Sees the label?	Where it must run
Filter	Univariate score per feature (correlation, F-test, mutual information)	Yes	Inside each training fold
Wrapper	Search over subsets scored by a model (forward, backward, RFE)	Yes, many times	Inside each fold; tune in an inner loop
Embedded	Selection during the fit (L1 penalty, tree splits)	Yes	Part of the model; already inside the fold
Unsupervised	Variance or missingness thresholds, correlation clustering	No	Safer, but still fit on training rows

Selecting 20 of 2,000 noise features before cross-validation Chart

Selection on all rowsselector saw every label0.94 AUC
Selection inside each fold0.49 AUC
Fresh datathe honest answer: no signal0.5 AUC

Output of the hands-on lab script (100 rows, pure noise). The leaky number is the selector's luck.

Choosing between models honestly3 minSame folds, paired differences, the simplest model within noise, one test, and explanations checked across equally good models.

An honest model comparison Sequence

Same folds for all
Every candidate, including the baseline, is scored on the same time-ordered folds.
Tune inside
Selection and tuning run in an inner loop; the outer folds only score.
Paired differences
Compare fold by fold; report the spread, not only the mean.
Simplest within noise
Prefer the simpler model within one standard error of the best.
Test once
Score the chosen model once on the untouched block (Day 15).
Check explanations
Equally good models can rely on different correlated features: repeat across seeds or models, report ranges for groups.

Library defaults that change the answeradvanced3 minWhat scikit-learn and shap compute when you pass nothing, verified on the installed versions, and what to set instead.

Library defaults that change the answer (installed versions) Table

Call	Default	What to set
sklearn permutation_importance	n_repeats=5; scoring=None uses the model's score (R-squared for regressors)	Held-out X and y, more repeats, scoring in the decision's metric
sklearn partial_dependence	kind='average'; 100 grid points between the 5th and 95th percentiles	kind='both' to get ICE beside the PDP
PartialDependenceDisplay	centered=False; ICE drawn for a subsample of 1,000 rows	centered=True to compare slopes
Forest feature_importances_	Impurity (MDI) on training rows, rescaled to sum to 1	Use as a debugging hint only
shap TreeExplainer (0.52)	No background data: tree_path_dependent; with background: interventional	Pass a small background set and state the choice
ALE	Not in scikit-learn 1.9	Implement it or use a dedicated package; check rows per window

Checked against scikit-learn 1.9.1 and shap 0.52.0 in this repository's environment. The default PDP grid stops at the 5th and 95th percentiles, so the tails are not drawn at all; permutation importance scored in the decision's metric reads in business units.

Memory anchor

Importance describes the model, not the world; draw ICE beside every PDP and group correlated features.

Useful for

Debugging a credit-risk, churn or readmission model by spotting leakage-like features, duplicates and unstable reliance before launch.
Explaining to stakeholders what a model uses, with honest limits.
Reducing a feature set with selection that is validated correctly.

Not suitable for

Deciding what intervention will change an outcome; use an experiment or a causal design.
Auditing fairness or compliance on its own; an explanation of the model is not evidence about the process behind the data.

Traps

Reading a ranking as a list of drivers

The top importance goes into a plan ("cut the loyalty discount", "move every customer to an annual contract") as if shuffling a column were the same as changing it in the world.

How to avoid it

Phrase findings as "the model relies on X" and attach the experiment or causal analysis that would test the action.

Trusting one number from one model

With correlated features, the ranking depends on seed and model family, and a single permuted column understates a block that is jointly critical.

How to avoid it

Group or cluster correlated features, repeat with several seeds or equally good models, and report ranges.

Senior reasoning

A senior establishes that the model predicts well before explaining it, uses held-out permutation importance for ranking and grouped permutation for correlated blocks, draws ICE beside PDP, and states the question each tool answers. They keep description and intervention apart: an explanation says what the model uses, and only a design says what would happen if the feature were changed.

Interview question

"Your SHAP plot says feature X drives churn. Product wants to change X. What do you say?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

Name one failure condition each for impurity importance, permutation importance, PDP and SHAP.
Where must label-aware feature selection run in a validated workflow, and why?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Bias in random forest variable importance measures: Illustrations, sources and a solution — Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn, 2007core
Shows impurity-based random forest importance is biased toward predictors with many categories or split points, supporting the weekend-versus-noise comparison.
Permutation feature importance (scikit-learn User Guide 5.2) — scikit-learn developerscore
Defines permutation importance, recommends held-out data, warns that strongly correlated features give misleading values and suggests clustering them. (Misleading values on strongly correlated features)
Be careful when interpreting predictive models in search of causal insights (SHAP documentation) — Eleanor Dillon, Jacob LaRiviere, Scott Lundberg, Jonathan Roth, Vasilis Syrgkaniscore
Shows with worked examples why SHAP values on a predictive model are not causal effects when confounding or redundancy is present.
Peeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional Expectation — Alex Goldstein, Adam Kapelner, Justin Bleich, Emil Pitkin, 2015optional
Introduces ICE plots to show per-observation curves that an averaged PDP can hide.
Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models — Daniel W. Apley, Jingyu Zhu, 2020optional
Introduces ALE plots, which avoid evaluating the model at unlikely combinations when features are correlated.
Conditional variable importance for random forests — Carolin Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, Achim Zeileis, 2008optional
Proposes conditional permutation within groups of correlated predictors, so importance is less inflated for correlated features.
Random Forests — Leo Breiman, 2001optional
Introduces random forests and the permutation (shuffle a column, measure the drop) importance used in the permutation chapter.
All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously — Aaron Fisher, Cynthia Rudin, Francesca Dominici, 2019optional
Defines model reliance (permutation importance) and model class reliance, the range of reliance across near-optimal models.
Partial Dependence and Individual Conditional Expectation plots (scikit-learn User Guide 5.1) — scikit-learn developersoptional
Documents partial dependence and ICE in scikit-learn, including kind='both' and centered ICE curves.
A Unified Approach to Interpreting Model Predictions — Scott M. Lundberg, Su-In Lee, 2017optional
Defines SHAP values as additive feature attributions based on Shapley values.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd edition) — Trevor Hastie, Robert Tibshirani, Jerome Friedman, 2009optional
Section 7.10.2 shows why selecting predictors using all labels before cross-validation gives a misleadingly good error estimate. (7.10.2 The Wrong and Right Way to Do Cross-validation)

Optional sections

Deep dive · 10 min

Why impurity importance favours many-valued features. A tree searches for the split with the largest impurity reduction. A continuous variable offers many candidate thresholds, and the best of many random thresholds still shows some gain, so it is selected more often and its gain is added to the importance. The gain is also measured on the training rows, so an overfit tree credits noise it has memorized. Held-out permutation importance avoids both.

Why permuting a correlated column is awkward. Shuffling lead_time alone produces rows where lead time and advance_purchase disagree, a combination that never occurs. The model is then asked about inputs it has not seen. Two remedies exist: permute a correlated block together, as the script does, or use a conditional scheme that shuffles within windows of its correlates.

PDP, ICE, ALE in one paragraph. PDP at value $v$ is the average prediction when every row's feature is set to $v$ : $\frac1n\sum_i f(v, x_{i,-j})$ . ICE is the summand for each row. ALE estimates, within narrow windows of the feature, the average change in prediction between the window's edges for the rows that fall in it, and accumulates those changes, so it only uses combinations that occur in the data (Apley and Zhu, 2020).

What SHAP adds and what it keeps. SHAP assigns each feature a share of the gap between the baseline prediction and this prediction, with additivity and consistency properties. It gives local explanations and a global summary by averaging absolute values. It does not remove correlation or confounding: with correlated features the split of credit depends on how the background distribution is handled, and none of it says what an intervention would do.

Hands-on lab: Selection outside versus inside cross-validation, on data with no signal · 15 min

Run the script. It draws 100 rows and 2,000 pure-noise features, selects 20 features with a univariate F-test, and scores a logistic regression by 5-fold cross-validation twice: with selection done once on all rows, and with selection inside each training fold through a pipeline. A fresh sample then checks the honest answer. Then answer: (a) why is the first AUC far above 0.5 when nothing is predictive; (b) what would happen to the first number if you raised the feature count to 20,000; (c) where else in Days 9-15 did a step that sees the label sit outside the fold?

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(170)
n, p, k = 100, 2000, 20
X, y = rng.normal(size=(n, p)), rng.integers(0, 2, n)               # pure noise: no feature relates to y
X_new, y_new = rng.normal(size=(5000, p)), rng.integers(0, 2, 5000)  # fresh data from the same process
cv = StratifiedKFold(5, shuffle=True, random_state=0)
clf = LogisticRegression(max_iter=1000)

Xs = SelectKBest(f_classif, k=k).fit_transform(X, y)                 # wrong: selection has seen every label
wrong = cross_val_score(clf, Xs, y, cv=cv, scoring="roc_auc")
pipe = make_pipeline(SelectKBest(f_classif, k=k), clf)               # right: selection refit in each training fold
right = cross_val_score(pipe, X, y, cv=cv, scoring="roc_auc")
fit = pipe.fit(X, y)
print(f"selection before CV (leaky): AUC {wrong.mean():.2f}")
print(f"selection inside CV:         AUC {right.mean():.2f}")
print(f"fresh data, same pipeline:   AUC {roc_auc_score(y_new, fit.predict_proba(X_new)[:, 1]):.2f}")

Interview practice · 10 min

A hospital's quality team asks for "the top five features that matter" from a gradient-boosted readmission model with 200 features, many of them correlated lab values. How do you answer?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on scikit-learn, shap

1 Recall · 5 min

2 Concept · 10 min