Core day 9 of 60 · Week 2 · Part I · application depth

Feature Engineering and Preprocessing

A no-show model reaches a validation ROC AUC of 0.77 thanks to a "travel agent no-show rate" feature, but the same feature is useless in production. Where exactly did the validation score learn from the answers, and how do you build preprocessing that cannot do that?

By the end: You draw the holdout boundary before fitting anything, build a Pipeline with a ColumnTransformer whose every step is fit on training rows only, and identify which preprocessing step leaked in a given workflow.

At a glance

Split first, fit every preprocessing step on training rows only, and treat any step that looks at the target or at future rows as the main leakage risk.

Split first
Draw the time boundary before any step learns anything from the data.
Pipeline
One object fits imputers, scalers and encoders on training rows only.
Scaling
Matters for distances and penalties; trees ignore monotone rescaling.
Target encoding
A noise column scored 0.771 when its rates included the test outcomes.
Cross-fitting
Encode each training row with rates learned on other folds.

Recall · 5 min

Without notes: define the decision, the baseline, the metric unit and one action cost for a no-show prediction (Day 2). Today is the first day you will split data, and the baseline you name must be computed on the training side.

Open due reviews

Concept · 10 min

Every preprocessing step learns something from data: an imputer a median, a scaler a mean and spread, a target encoder each category's outcome rate. Anything learned from rows you later evaluate on is information from the exam paper. So the order is fixed: draw the holdout boundary first, fit every step on the training side only, and apply the fitted steps unchanged to the test side.

A noise feature looks predictive when its encoder has seen the test outcomes

Predict first

A category column has one row per category. You target-encode it in-sample. What does the encoded value equal for each row?

Text description of the figure

A lab with a fit-scope switch (all rows vs training rows), an encoding selector (naive in-sample target means vs out-of-fold encoding vs one-hot) and a cardinality slider. Validation scores rise sharply when the target encoder is fit on all historical rows, especially at high cardinality, while deployment scores stay flat; they barely move when only the scaler is fit on all rows. An in-sample encoder fit on training rows only does not inflate validation: it makes the model trust a feature that is partly the label, and validation falls below the reference.

The order is fixed Diagram

All bookingsin time order
Draw the boundarycutoff date
Fit on train onlymedians, means, rates
Apply unchangedto test rows
Score onceagainst AUC 0.5

The base-rate baseline gets the same rule. Any statistic computed before the boundary is drawn has seen the test rows.

One pipeline, three column groups Diagram

Raw columns
- to Market: one-hot
- to Agent id: target-encode
Numeric: imputemedian + missing indicator
- to Scale
Market: one-hothandle_unknown = ignore
- to ColumnTransformer
Agent id: target-encodecross-fitted, smoothed
- to ColumnTransformer
ScaleStandard or Robust
ColumnTransformerjoins the groups
Modelfit on train only

Calling fit on the pipeline with training rows fits every step there; predict applies the same fitted steps to new rows, in training and in serving.

Today's run: test ROC AUC Chart

Base-rate baseline0.5 AUC
Honest pipelinethe number to report0.654 AUC
Agent rate, train rows onlynoise, as designed0.495 AUC
Agent rate, all rowsleak: contains the test outcomes0.771 AUC

20,000 simulated bookings; the agent id has no relationship with no-shows.

A noise column, encoded three ways, as categories multiply Illustration

Fit on all rows, each later row's own outcome sits in its category's mean. With few rows per category that is most of the mean: AUC 0.96 from noise.

In-sample encoding makes the feature look strong exactly where it is weakest; the model learns to trust it and pays on new rows. Cross-fitted rows never see their own label.

Smoothing barely moves it: 0.79 at m = 0, 0.76 at m = 100. It shrinks small categories toward the overall rate, but each row's own label stays in its mean. Only cross-fitting removes it.

Seeded synthetic bookings: 5,000 rows in time order (first 4,000 train), no-show rate 0.12, category assigned at random (uneven sizes) so it carries no signal. Scores are ROC AUC of the encoded value alone.

The category carries no signal at all. Every point above 0.5 is the encoder copying labels into the feature, and the copy grows as rows per category shrink.

The leak that hurts is the one that looks at the target Scale

Scalerfit on all rows: 0.6629 either way in the lab
Imputera median of a feature
Rolling statisticsleak if the window reaches future rows
Feature selectionchosen by correlation with the target
Resamplingoversampled copies land on both sides of the split
Target encodingthe category's outcome rate

Qualitative order. Fitting a scaler on all rows usually moves results very little; a step that uses the target or future rows can turn a noise column into a headline score.

A scikit-learn Pipeline with a ColumnTransformer fits each column group's steps only on the data passed to fit.
Key terms: missing indicator, target encoding, cross-fitting and leakage.
The base-rate baseline scores ROC AUC 0.5: the floor any model must clear. Domain features obey the Day 2 availability rule.

Assumptions

The split mimics deployment (time order for time-ordered data, group order when the same entity repeats); a random split with a perfect pipeline can still leak through time or entities (Day 15).
Rows are already cleaned with logged, re-runnable rules (Day 6); fixes decided by looking at test rows are themselves a leak.
The preprocessing used in training is the exact object used in serving; a re-implemented copy creates training-serving skew (Day 19).

Application and workplace decision · 10 min

Run the script. It simulates 20,000 bookings in time order with a 12% no-show rate, missing lead times, a market column and a 2,000-level travel-agent id that has no relationship with no-shows. It splits first, fits one pipeline on the training rows, then shows what happens when the agent no-show rate is computed on all rows. Decide which number you would report.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import KFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler, TargetEncoder

rng = np.random.default_rng(9)
n = 20000  # synthetic bookings, in booking-date order
df = pd.DataFrame({"days_ahead": rng.gamma(2, 15, n), "fare": rng.lognormal(5, 0.5, n),
                   "market": rng.choice(["domestic", "intl-short", "intl-long"], n),
                   "agent_id": rng.integers(0, 2000, n).astype(str)})  # 2,000 agents: pure noise
logit = -2.2 + 0.02 * df["days_ahead"] - 0.004 * df["fare"] + 0.4 * (df["market"] == "domestic")
df["no_show"] = rng.random(n) < 1 / (1 + np.exp(-logit))
df.loc[rng.random(n) < 0.1, "days_ahead"] = np.nan  # 10% missing lead time
cut = int(n * 0.8)  # 1) holdout boundary FIRST: earlier bookings train, later bookings test
train, test = df.iloc[:cut], df.iloc[cut:]
X_cols, y = ["days_ahead", "fare", "market", "agent_id"], "no_show"
prep = ColumnTransformer([  # 2) every step below is fit on train rows only, inside the pipeline
    ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), ["days_ahead", "fare"]),
    ("cat", OneHotEncoder(handle_unknown="ignore"), ["market"]),
    ("agent", TargetEncoder(cv=KFold(5, shuffle=True, random_state=0)), ["agent_id"]),  # cross-fitted; a splitter as cv needs scikit-learn 1.9+
])
model = make_pipeline(prep, LogisticRegression(max_iter=1000)).fit(train[X_cols], train[y])
print(f"no-show rate train/test: {train[y].mean():.3f} / {test[y].mean():.3f}")
print(f"pipeline test ROC AUC: {roc_auc_score(test[y], model.predict_proba(test[X_cols])[:, 1]):.3f}")
# 3) agent no-show rate as a lone score: fit on train rows vs fit on ALL rows (test outcomes included)
honest = test["agent_id"].map(train.groupby("agent_id")[y].mean()).fillna(train[y].mean())
leaky = df.groupby("agent_id")[y].transform("mean").iloc[cut:]
print(f"agent rate from train only, test ROC AUC: {roc_auc_score(test[y], honest):.3f}")
print(f"agent rate from all rows (leak), test ROC AUC: {roc_auc_score(test[y], leaky):.3f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Standardization: the algebra, a worked distance, and which models care7 minz-scores defined with training statistics, what they do to coefficients, penalties and distances, and a 4-row kNN example where the nearest neighbour flips.

Standardization as algebra: what changes and what does not Formula

\begin{gathered}z_{ij} = \frac{x_{ij} - \hat\mu_j}{\hat\sigma_j} \\ b_0 + b\,x = (b_0 + b\,\hat\mu) + (b\,\hat\sigma)\,z \\ d_z(u,v)^2 = \sum_j \left(\frac{u_j - v_j}{\hat\sigma_j}\right)^2\end{gathered}

$\hat\mu_j, \hat\sigma_j$: mean and standard deviation of feature j on the TRAINING rows; scikit-learn's StandardScaler uses the population SD (divide by n)
$z_{ij}$: the standardized value; on training rows each column has mean 0 and SD 1, on test rows it need not (and drift shows up there)
$b\,\hat\sigma$: the coefficient on z: an unpenalized linear model gives identical predictions, only the coefficients are rescaled
$\lambda \sum b^2$: a ridge or lasso penalty is not rescaled with them: in raw units it punishes features measured in small units, which need large b
$d_z$: Euclidean distance after scaling weights feature j by 1 / sigma_j squared; in raw units the widest-ranging feature decides

Standardizing is an affine change of units. Models that are invariant to units (unpenalized OLS, trees) do not care; distances, penalties and gradient steps do.

Worked by hand: which customer is nearest, before and after scaling Table

Quantity	Basket value (EUR)	Days since last order	Distance to query q
Training rows	40, 60, 80, 100	2, 4, 6, 8	-
Mean, population SD	70, sqrt(500) = 22.36	5, sqrt(5) = 2.236	-
Query q (raw)	62	3	-
Customer A (raw)	60	8	sqrt(2^2 + 5^2) = 5.39 - nearest raw
Customer B (raw)	80	4	sqrt(18^2 + 1^2) = 18.03
q standardized	(62 - 70) / 22.36 = -0.36	(3 - 5) / 2.236 = -0.89	-
A standardized	-0.45	1.34	sqrt(0.09^2 + 2.24^2) = 2.24
B standardized	0.45	-0.45	sqrt(0.81^2 + 0.45^2) = 0.92 - nearest scaled

Hypothetical e-commerce customers, verified with scikit-learn's StandardScaler. The basket column spreads 10 times wider, so raw distance is almost only basket value; after scaling, recency gets a vote and the nearest neighbour flips.

What a distance listens to Illustration

The neighbours share a fare but their lead times span 45 days: tens of euros outweigh weeks.

Now both columns count: lead times span 11 days, fares 305 EUR.

If no-shows depend on lead time, a kNN model in raw units averages over bookings that differ on exactly that.

Seeded synthetic bookings (220): log-normal long-haul fares, lead times in days, averaged over every booking, 10 neighbours each. Standardised with the column mean and SD.

The same bookings and the same query booking. In raw units the fare decides who the neighbours are; after standardising, lead time gets a vote.

Unscaled features stretch the loss into a valley Illustration

Curvature 36 times larger across the valley than along it. The step that is safe across it (zig-zag) moves only 4.4% of the way along it per step.

Both directions have curvature 1, so one learning rate fits both: the path stays on one straight line through the minimum, overshooting and settling in a few steps.

Exact gradient descent on a two-coefficient least-squares bowl; feature SDs 1 and 6; learning rate 1.6 / (largest curvature) in both panels. Contours at 100%, 50%, 20% and 5% of the starting loss.

The same least-squares loss before and after standardizing, and gradient descent with the largest safe step for each. Standardizing is the cheapest preconditioner: one learning rate now suits every direction.

Which models care about scaling? Comparison

Criterion	kNN, SVM, k-means	Regularized linear, neural nets	Trees and tree ensembles
Why	Distances mix units: an income in euros swamps a utilisation share	Penalties and gradient steps depend on units	Splits depend on order, not scale
Scale them?	Yes	Yes	No effect from monotone rescaling

Three scalers Table

Scaler	Learns	Use when
StandardScaler	mean and standard deviation	roughly symmetric features
RobustScaler	median and interquartile range	heavy tails or outliers
MinMaxScaler	minimum and maximum	a fixed range is required; sensitive to outliers

Target encoding: shrinkage by hand, then cross-fitting7 minThe smoothed estimator derived as a posterior mean, six brokers encoded by hand, why shrinkage alone still leaks, and the cross-fitting fix.

Smoothed target encoding Formula

\text{enc}(c) = \frac{n_c\,\bar y_c + m\,\bar y}{n_c + m}

$\bar y_c$: outcome rate of category c in the rows used for fitting
$n_c$: number of those rows; small categories are pulled toward the overall rate
$\bar y$: overall outcome rate
$m$: smoothing strength; scikit-learn's TargetEncoder sets it by an empirical-Bayes rule by default

Smoothing limits noise from small categories. It does not stop a row's own outcome entering its value; cross-fitting does.

Target-encoding shrinkage, and where the formula comes from Formula

\begin{gathered}\text{enc}(c) = \lambda_c\,\bar y_c + (1-\lambda_c)\,\bar y \\ \lambda_c = \frac{n_c}{n_c + m} \;\;\text{(smooth} = m) \\ \lambda_c = \frac{n_c\, v}{n_c\, v + s_c^2} \;\;\text{(smooth = 'auto')}\end{gathered}

$\lambda_c$: the weight on the category's own rate: near 1 for big categories, near 0 for tiny ones
$n_c/(n_c+m)$: the posterior mean of a rate under a Beta prior worth m pseudo-rows at the global rate: (k_c + m ybar) / (n_c + m)
$v$: variance of the target over all training rows
$s_c^2$: variance of the target inside category c; scikit-learn's 'auto' rule (Micci-Barreca) shrinks noisy categories more
$\bar y$: the global rate: what an unseen category, or one with no rows in the fitting fold, receives
$1-\lambda_c$: the bias accepted for less noise: small categories are pulled toward the global rate, as in an empirical-Bayes or random-intercept model

Both rules read off scikit-learn 1.9.1's implementation. Note the 'auto' corner case: a category whose rows all share one label has s_c^2 = 0, so lambda_c = 1 and the encoding is that label - for a one-row category, exactly its own outcome.

With smooth = m, a category gets half its own rate at n_c = m Chart

m = 10
limit: own rate only

lambda_c = n_c / (n_c + m), computed. m is the row count at which a category's own rate and the global rate weigh the same: tune it on validation folds, or use 'auto', which sets the weight per category from within- and between-category variance.

Worked by hand: claim rate by insurance broker, smooth = 10 Table

Broker	Rows n_c	Claims k_c	Raw rate	lambda_c = n_c / (n_c + 10)	Encoding (k_c + 10 x 0.15) / (n_c + 10)
A	2	1	0.500	0.17	2.5 / 12 = 0.208
B	10	4	0.400	0.50	5.5 / 20 = 0.275
C	27	1	0.037	0.73	2.5 / 37 = 0.068
D	1	0	0.000	0.09	1.5 / 11 = 0.136
New broker E	0	-	-	0	0.150 (the global rate)
D if its one label were 1	1	1	1.000	0.09	2.75 / 11 = 0.250 (global rate becomes 0.175)

Hypothetical 40 policies, 6 claims, global rate 0.15; matches TargetEncoder(smooth=10).fit(X, y). Shrinkage tames broker A, but D's encoding still moves with D's own label - the leak cross-fitting removes.

Cross-fitting a target encoder Sequence

Split the training rows into folds
For example 5 folds.
Learn rates on 4 folds
Category outcome rates without the held-out fold.
Encode the 5th fold
Its rows never see their own outcomes.
Rotate
Repeat until every training row is encoded.
Test rows
Encoded with rates learned on all training rows; TargetEncoder.fit_transform does this.

Three treatments of one noise column2 minIn-sample, cross-fitted and absent, scored on train and test.

Lab run: the same noise column, three treatments Table

Agent feature	Train AUC	Test AUC
In-sample target means	0.889	0.528
Cross-fitted TargetEncoder	0.731	0.646
No agent feature (baseline)	0.725	0.643

In-sample encoding on training rows makes the model trust a column that is partly the label; test AUC falls below the model without it.

Pipelines and the defaults that bite3 minOne object keeps every fitted step on the training side; its defaults decide more than you think.

Inside cross-validation, the whole pipeline is refit every fold Loop

Hold out fold kthe rows this round scoresfit on the rest ↓
Fit on K-1 foldsimputer, scaler, encoder and modeltransform fold k ↓
Transform fold kwith the statistics just learnedpredict ↓
Predict fold krows no step was fit onscore ↓
Score and storeK scores give a mean and a spreadk + 1 ↓
↺ back to 1 · Hold out fold k

cross_val_score and GridSearchCV refit the full pipeline per fold. Preprocessing fitted once, before the loop, has seen every validation fold.

scikit-learn 1.9 defaults that change results Table

Object	Default	What it means for you
SimpleImputer	strategy='mean', add_indicator=False	mean is pulled by tails; the fact that a value was missing is dropped
OneHotEncoder	handle_unknown='error'	a new route at serving time raises an error; set 'ignore' or 'infrequent_if_exist'
ColumnTransformer	remainder='drop'	columns you forgot to list vanish silently
TargetEncoder	cv=5, smooth='auto'	fit_transform cross-fits; fit(X, y).transform(X) on the same rows does not
StandardScaler	with_mean=True, with_std=True	centering destroys sparsity; use with_mean=False or MaxAbsScaler on sparse input
KNeighborsClassifier	n_neighbors=5, Minkowski p=2	Euclidean distance: the widest-ranging unscaled column decides the neighbours; scale inside the pipeline

Checked against the installed scikit-learn 1.9.1 signatures and the user guide. Write the non-default choices explicitly so a reviewer can see them, and serve the fitted pipeline itself: a re-implemented copy creates training-serving skew (Day 19).

Choosing an encoding3 minOne-hot, frequency, target and hashing compared, and a decision rule by cardinality.

The encoding menu for a categorical column Comparison

Criterion	One-hot	Frequency / count	Target (cross-fitted)	Hashing
Uses the target?	No	No	Yes	No
High cardinality	Wide and sparse; group rare levels (min_frequency)	One column	One column per class or target	Fixed width, no vocabulary to store
Unseen category	All zeros with handle_unknown='ignore'	Zero or a floor count	The training target mean	Hashed like any other value
Main risk	Collinearity with an intercept in unpenalized models	Different categories with equal counts merge	Leakage unless cross-fitted; time order needs past-only rates	Collisions; features cannot be read back

Ordinal codes only for genuinely ordered levels (fare class L < M < H, a claim severity band, a credit grade); learned embeddings come on Day 38.

Encoding a categorical column Diagram

Many levels?
- no: to One-hot
yes
Target encodingcross-fitted, smoothed
- to Check against baseline
One-hotignore unseen levels
Check against baselinetest AUC with and without
Keep only if it helpson the time-ordered test

Encodings on time-ordered dataadvanced3 minWhy random folds are not enough when outcomes arrive over time, and the point-in-time alternative.

Target encoding on time-ordered rows Diagram

Rows ordered in time?
- no: to K-fold cross-fitting
yes
Expanding windowagent a at time t: its bookings before t only
- to Outcome known when?
K-fold cross-fittingTargetEncoder.fit_transform
Outcome known when?a no-show is known after departure
Lag the windowdeparted before t; shrink to the rate known at t
plus smoothing
Add a count featureearly rows have few past outcomes: noisy rates

Random folds let a row be encoded with outcomes from its future. A point-in-time rate uses only outcomes that existed at decision time (Days 2 and 24).

Domain features2 minFeatures from the business that generic transforms do not find.

Domain features that beat generic transforms Overview

Debt-to-income ratio
Banking: compares a loan with what the applicant can carry; neither column alone does.
Hours since last service
Manufacturing: wear accumulates between maintenance events, not with calendar age.
Price per unit
Retail: compares pack sizes; fare per kilometre does the same across routes.
Lead time x segment
Early bookings behave differently on long-haul; early renewals differ by plan in SaaS.
Same availability rule
Built only from what exists at decision time (Day 2).

Memory anchor

Split first, fit on train, transform everything - and never let a feature see its own label.

Useful for

Any tabular model with mixed numeric and categorical inputs, missing values and high-cardinality ids - merchant ids in fraud, product SKUs in retail, provider ids in health claims.
Code review of notebooks before a validation number is reported, in any sector.
Shipping preprocessing and model as one serialised object, so serving cannot drift from training.

Not suitable for

Fixing a split that does not match deployment - a perfect pipeline on a random split of time-ordered data still leaks (Day 15).
Replacing domain judgment on features - pipelines prevent leaks; they do not tell you which features are available or meaningful.

Traps

Statistics of the target before the split

Group means, rates or feature rankings computed on the full table copy test outcomes into features; the validation score rises and production does not follow.

How to avoid it

Split first; compute target statistics only inside the pipeline, cross-fitted on training folds.

Unknown categories at serving time

A new route, merchant or product SKU appears after training; a one-hot encoder without handle_unknown="ignore" raises an error, and a hand-written mapping returns NaN that a later step silently fills.

How to avoid it

Decide the unseen-category behaviour explicitly (ignore, "other" bucket, global-mean fallback) and test it.

Senior reasoning

A senior reads a pipeline in the order the data flow: where is the split, what does each step learn, from which rows, and does any step see the target or the future. They distinguish leaks that rarely matter (a scaler fit on all rows) from leaks that decide the result (target encoding, feature selection, resampling before the split), and they ship the fitted pipeline as one artefact so training and serving cannot diverge.

Interview question

"How would you encode a categorical feature with 50,000 levels for a churn model?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

State the order of operations for preprocessing, and what a Pipeline guarantees.
Which preprocessing leaks are usually small and which are usually large? When is scaling unnecessary?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Common pitfalls and recommended practices (scikit-learn User Guide, section 12) — scikit-learn developerscore
Never fit on test data; preprocessing learned on the full data inflates scores (feature-selection example on random targets); use Pipeline. (12.2 Data leakage, 12.2.2 Data leakage during pre-processing)
scikit-learn User Guide: Target Encoder (Preprocessing data, 8.3.4.2) — scikit-learn developerscore
Cross fitting in TargetEncoder.fit_transform, why fit(X, y).transform(X) on the same rows leaks, and shrinkage toward the global mean. (8.3.4.2 Target Encoder)
Pipelines and composite estimators (scikit-learn User Guide 8.1) — scikit-learn developersoptional
Pipeline and ColumnTransformer for applying per-column preprocessing fit on training data only. (8.1 Pipelines and composite estimators)
sklearn.preprocessing.TargetEncoder (scikit-learn API reference) — scikit-learn developersoptional
TargetEncoder defaults (cv=5, smooth='auto') and the documented difference between fit_transform (cross-fitted) and fit(X, y).transform(X).
Preprocessing data (scikit-learn User Guide 8.3) — scikit-learn developersoptional
Why regularized linear models and kernel methods assume similarly scaled features; sparse inputs and scaling; OneHotEncoder options.
Leakage in data mining: Formulation, detection, and avoidance — Shachar Kaufman, Saharon Rosset, Claudia Perlich, Ori Stitelman, 2012optional
Leakage defined as information about the target that is not legitimately available at prediction time, and learn-predict separation as the avoidance principle.

Optional sections

Deep dive · 10 min

Missing values as information. If lead time is missing mainly for agency bookings and agency bookings no-show more, the missingness is informative (a MAR or MNAR pattern from Day 6). A median plus a missing indicator lets the model use that; median alone erases it. Tree libraries with native missing-value handling learn a direction for missing values, which has the same effect.

Outliers in features. Clip or transform heavy-tailed features (log fare) for linear and distance-based models, with thresholds learned on training rows. Do not clip a value that Day 6 classified as an error - correct it in the cleaning step, where the rule is logged.

Encoding menu. One-hot (few levels); ordinal (genuinely ordered levels such as fare class L < M < H); frequency encoding (cheap, no target, but merges different categories with equal counts); target encoding (strong, needs cross-fitting and smoothing); hashing (fixed width, collisions); embeddings (Day 38). For linear models, one-hot with a dropped reference level avoids perfect collinearity; tree models do not need the drop.

Time-aware target encoding. For time-ordered data, cross-fitting on random folds still lets a row be encoded with outcomes from its future. An expanding-window encoding (only outcomes before the row's timestamp) is the point-in-time version; Day 24 builds the same idea for lag features.

Hands-on lab: Measure which preprocessing leaks matter · 20 min

Run the script. Part A fits the scaler on all rows vs training rows. Part B target-encodes a pure-noise, 2,000-level agent id three ways and trains a gradient-boosted model on each. Then:

Explain why in-sample target means give a train ROC AUC near 0.89 and a test ROC AUC near 0.53.
Why does the cross-fitted encoder land close to the "no agent feature" baseline? What would you conclude about keeping the feature?
Part A shows no difference. Describe a case where fitting the scaler on all rows would matter (hint: a time-ordered split with a trend in fares).

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler, TargetEncoder

rng = np.random.default_rng(9)
n = 20000
df = pd.DataFrame({"days_ahead": rng.gamma(2, 15, n), "fare": rng.lognormal(5, 0.5, n),
                   "agent_id": rng.integers(0, 2000, n).astype(str)})  # agent_id carries no signal
logit = -2.2 + 0.02 * df["days_ahead"] - 0.004 * df["fare"]
df["no_show"] = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
cut = int(n * 0.8)
train, test = df.iloc[:cut].copy(), df.iloc[cut:].copy()
y_tr, y_te = train["no_show"], test["no_show"]
num = ["days_ahead", "fare"]

# A) Scaler fit on all rows vs train rows: a real leak, usually a small one
for scope, fit_rows in [("all rows", df), ("train rows", train)]:
    sc = StandardScaler().fit(fit_rows[num])
    lr = LogisticRegression().fit(sc.transform(train[num]), y_tr)
    auc = roc_auc_score(y_te, lr.predict_proba(sc.transform(test[num]))[:, 1])
    print(f"A) scaler fit on {scope:10s}: test AUC {auc:.4f}")

# B) Target encoding of a pure-noise, high-cardinality column, three ways
def score(enc_train, enc_test, label):
    Xtr = np.column_stack([train[num], enc_train])
    Xte = np.column_stack([test[num], enc_test])
    gb = HistGradientBoostingClassifier(max_iter=200, random_state=0).fit(Xtr, y_tr)
    tr_auc = roc_auc_score(y_tr, gb.predict_proba(Xtr)[:, 1])
    te_auc = roc_auc_score(y_te, gb.predict_proba(Xte)[:, 1])
    print(f"B) {label:34s} train AUC {tr_auc:.3f} | test AUC {te_auc:.3f}")

means = train.groupby("agent_id")["no_show"].mean()  # in-sample: each row sees its own outcome
prior = y_tr.mean()
score(train["agent_id"].map(means), test["agent_id"].map(means).fillna(prior), "in-sample target means")
te = TargetEncoder(cv=KFold(5, shuffle=True, random_state=0))  # cross-fitted: each train row encoded by the other folds (splitter as cv: scikit-learn 1.9+)
score(te.fit_transform(train[["agent_id"]], y_tr).ravel(), te.transform(test[["agent_id"]]).ravel(),
      "cross-fitted TargetEncoder")
score(np.zeros(len(train)), np.zeros(len(test)), "no agent feature (baseline)")

Interview practice · 10 min

Walk me through how you would review a teammate's notebook that reports a surprisingly good validation score on a tabular classification problem.

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11 · depends on scikit-learn, pandas, numpy

1 Recall · 5 min

2 Concept · 10 min