Core day 12 of 60 · Week 3 · Part I · application depth

Trees and Ensembles

A gradient-boosted model cuts the Day 2 baseline error from 7.7 to 5.6 seats on last quarter's departures. In March a wide-body swap puts D-14 bookings far above anything in the training table, and its forecasts stall near 180. What did the ensemble learn, what can it never do, and what must it beat before you adopt it?

By the end: You compare a forest and a boosted model with the Day 2 baseline and a linear model on a time-ordered split, show that both flatline outside the training range, and reframe the target so that they no longer do.

At a glance

Many simple tree rules combined; averaging cuts variance, boosting cuts bias, and no tree ensemble continues a trend beyond the range it was trained on.

One tree
Easy to read but unstable; depth decides memorizing versus underfitting.
Bagging
Average deep, decorrelated trees: less variance, free out-of-bag error.
Boosting
Add shallow trees that fix remaining errors: less bias, stop early.
The limit
Flat outside the training range: predictions stay near the targets seen.
Adopt only if
it beats the Day 2 baseline and a linear model on a time-ordered split.

Recall · 5 min

Without notes: say what an OLS coefficient is conditional on, and which residual pattern threatens standard errors but not the fitted line (Day 11). Then name the baseline in seats that today's ensembles must beat and the rows it was computed on.

Open due reviews

Concept · 10 min

A tree asks yes/no questions and predicts the average of the rows in the leaf it ends in. One deep tree memorizes; many trees averaged (bagging) or added one by one (boosting) fix that, but only inside the range of data they have seen.

Averaging deep trees cuts variance, boosting shallow trees cuts bias, and every tree ensemble goes flat outside the training range

Predict first

A forest is trained on x between 0 and 10 where y rises linearly. What does it predict at x = 14?

Text description of the figure

A fitted curve over 80 noisy points for a linear model, a single tree, a random forest and a boosted model, with controls for tree count, depth, boosting depth and learning rate. A toggle extends the x axis beyond the training range, where every tree model runs flat while the truth continues. A bar table lists squared bias and variance per model.

A tree is a set of boxes Illustration

The tree: each question sends a flight left (yes) or right (no) until it reaches a leaf.

The same tree as boxes: every box predicts one number, the average of the flights inside it.

Synthetic example data (70 flights, seeded); the tree is fitted by the same code as the Day 12 lab.

One question per node; each leaf is a box that predicts the average of the rows inside it, be they flights, loans or machines.

Two ways to combine trees Diagram

Bagging: databootstrap resamples
- per resample: to Deep trees
Boosting: dataall rows
- fit errors: to Shallow tree
Deep treesgrown in parallel
- to Average
Shallow treefits current errors
- to Sum, small steps
Averagevariance falls
Sum, small stepsbias falls
- repeat: back to Shallow tree

Bagging builds independent trees at once. Boosting builds them in sequence, each on what the previous ones got wrong.

Where each model sits (qualitative) Scale

Shallow treetoo simple
Boosted shallow treesbias falls with rounds
Random forestdeep trees, averaged
One deep treememorizes noise

Outside the training range every tree model goes flat Illustration

truth (dashed)
linear fit
random forest
gradient boosting

Synthetic data (80 points, seeded); fitted with the Day 12 lab code.

Inside the training range (white) every model follows the data. Beyond it (shaded) the tree models repeat their last leaf average; only the line keeps going.

The flat line is not caution: it is the last leaf average repeating. Reframe the target (pickup, ratio, difference) or add a component that continues the trend.

Key terms: Impurity, bootstrap sample, out-of-bag error, gradient boosting and early stopping. Details are in the pictures above and in the optional technical depth below.

Assumptions

Future inputs and targets stay inside the range, and the same interactions, seen in training; outside it the ensemble stalls silently.
Rows are independent enough for bootstrap and out-of-bag estimates; repeated customers, machines, flights or neighbouring dates make them optimistic (Day 15).
The stopping block is separate from the final test block; stopping on the test block turns it into a tuning set.

Application and workplace decision · 10 min

The script builds 6,000 synthetic departures in date order. The first 3,000 train, the next 1,500 are the same aircraft type, and the last 1,500 are a wide-body swap (300 seats). Pickup (seats sold between D-14 and D-0) has a weekend-by-holiday interaction and does not depend on aircraft size. It scores the Day 2 baseline, a linear model, a forest and boosting on the level, and a forest fitted to the pickup. Before running it, predict which model degrades most on the wide-body.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.ensemble import HistGradientBoostingRegressor, RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error as mae

rng = np.random.default_rng(12)
n = 6000                                         # synthetic departures in date order
cap = np.where(np.arange(n) < 4500, 180, 300)    # last 1,500: a wide-body swap, bookings far above the training range
dow, hol = rng.integers(0, 7, n), rng.random(n) < 0.10
wk = dow >= 4
bk14 = cap * np.clip(rng.normal(0.50 + 0.08 * wk + 0.08 * hol, 0.10), 0.10, 0.90)             # bookings at D-14
pickup = 14 + 8 * wk + 36 * (wk & hol) - 8 * (hol & ~wk) + 0.12 * np.maximum(95 - bk14, 0) + rng.normal(0, 7, n)
final = np.minimum(bk14 + pickup, cap)           # target: bookings_final
tr, same, new = slice(0, 3000), slice(3000, 4500), slice(4500, n)   # Day 9 boundary, then the regime change
X, Z = np.c_[bk14, wk, hol], np.c_[cap - bk14, wk, hol]             # Z: seats still free, not the level

def forest():
    return RandomForestRegressor(100, min_samples_leaf=5, random_state=0)

pred = {"pickup baseline (Day 2)": bk14 + pickup[tr].mean(),
        "linear regression": LinearRegression().fit(X[tr], final[tr]).predict(X),
        "forest on the level": forest().fit(X[tr], final[tr]).predict(X),
        "boosting on the level": HistGradientBoostingRegressor(random_state=0).fit(X[tr], final[tr]).predict(X),
        "forest on the pickup": bk14 + forest().fit(Z[tr], (final - bk14)[tr]).predict(Z)}
for name, p in pred.items():
    print(f"{name:24s} MAE same aircraft {mae(final[same], p[same]):5.2f} | wide-body {mae(final[new], p[new]):5.2f} | max prediction {p[new].max():5.1f}")
print(f"largest wide-body actual {final[new].max():.0f}; largest training target {final[tr].max():.0f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 31 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

How a tree is grown: the split search by hand5 minImpurity, the full split search worked on six rows, and how depth trades bias for variance.

The model a tree fits: a constant per box Formula

\hat f(x) = \sum_{m=1}^{M} c_m \, \mathbf{1}\{x \in R_m\}, \qquad \hat c_m = \operatorname{mean}\{y_i : x_i \in R_m\}

$R_m$: the M leaves: axis-aligned boxes that partition the input space, cut out by the yes/no questions
$c_m$: the prediction in box m; for squared error the mean of its training targets, for classification the class shares
$\mathbf{1}\{x \in R_m\}$: 1 if x falls in box m, else 0: exactly one term is non-zero for any x

Every prediction is an average of training targets, so it can never leave their range. Finding the best boxes jointly is computationally intractable, so trees grow greedily, one split at a time.

How one tree is grown Diagram

Training rowsfeatures + target
Try every spliteach feature, each threshold
Keep the bestlargest impurity drop
Stop?depth, leaf size, gain
- no: split: back to Try every split
yes
Leafmean or class share

On "no", each child node goes back through the same search with only its own rows.

Impurity of a classification node Formula

G = 1 - \sum_{k} p_k^{2}

$p_k$: share of rows in the node that belong to class k
$G = 0$: a pure node (one class)
$\Delta G$: parent impurity minus the weighted child impurities; the biggest drop wins

For regression the same search minimizes squared error inside each child.

The split search by hand: six customers, five candidate splits Table

Split on months	Left \| right labels	Gini left \| right	Weighted	Decrease
< 1.5	1 \| 1,1,0,1,0	0.000 \| 0.480	0.400	0.044
< 2.5	1,1 \| 1,0,1,0	0.000 \| 0.500	0.333	0.111
< 3.5	1,1,1 \| 0,1,0	0.000 \| 0.444	0.222	0.222 - the winner
< 4.5	1,1,1,0 \| 1,0	0.375 \| 0.500	0.417	0.028
< 5.5	1,1,1,0,1 \| 0	0.320 \| 0.000	0.267	0.178

Hypothetical telecom rows: churn by months on contract (1, 1, 1, 0, 1, 0). Parent Gini 2 x (4/6)(2/6) = 0.444; weighted child Gini = (left share x left Gini) + (right share x right Gini). sklearn picks the same threshold, 3.5.

The winning split, drawn as the tree it creates Hierarchy

All 6 customerschurn 1,1,1,0,1,0 · Gini 0.444
- months < 3.53 customers: 1,1,1Gini 0 · predict churn
- months ≥ 3.53 customers: 0,1,0Gini 0.444 · predict stay

Each leaf predicts its majority class (or its share, 1.0 and 0.33). The weighted child Gini drops from 0.444 to 0.222; a deeper tree would keep splitting the impure right leaf.

What the greedy split search implies Overview

Exhaustive per node
Every feature, every threshold between sorted values: about p x n log n per node. Histogram libraries bin first.
Greedy
Locally best only: a weak split that enables two strong ones below is never tried. Depth limits and ensembles answer it.
Unstable
Near-ties are broken arbitrarily, and a few rows near the top split can reshape the whole tree and its importances.
Scale-free
Invariant to monotone transforms of each input, so no scaling. Deep trees: low bias, high variance.

Depth is the overfitting dial Illustration

training error
validation error (new draws)

Synthetic data: 60 training and 400 validation points from the same curve.

Validation error is lowest at depth 3; deeper trees fit the training set better and new data worse.

Random forests, step by step4 minBootstrap, random feature subsets, averaging and the out-of-bag estimate.

Why averaging helps Illustration

One deep tree chases every noisy point: a jagged, unstable fit. Error against the true curve: 0.13.

Eight trees, each on its own bootstrap resample. The shaded band is how far they disagree at each input.

Their average (25 trees) is smooth and close to the truth (dashed). Error against the true curve: 0.08, 40% lower than one tree.

Synthetic data: 45 noisy points around a smooth curve. Fitted with the same bagging code as the interactive lab; errors are mean squared distance to the true curve on a 140-point grid.

The variance of an average of trees Formula

\operatorname{Var}\left(\bar f\right) = \rho\,\sigma^{2} + \frac{1-\rho}{B}\,\sigma^{2}

$\sigma^{2}$: variance of one tree
$\rho$: correlation between trees
$B$: number of trees

More trees shrink only the second term. Lower correlation (bootstrap rows, random feature subsets) lowers the first.

The formula with numbers (one tree's SD 6, correlation 0.3) Table

Trees B	Variance of the average	SD of the average
1	36.0	6.00 - one tree
10	0.3 x 36 + 0.7 x 36 / 10 = 13.3	3.65
100	10.8 + 0.25 = 11.1	3.32
Infinitely many	rho x sigma^2 = 10.8	3.29 - the correlation floor

Illustrative numbers. Going from 100 trees to infinitely many buys almost nothing; only decorrelating the trees (bootstrap rows, feature subsets per split) lowers the 10.8 floor itself.

More trees help, then plateau Illustration

Synthetic data: 80 training points, 400 validation points; trees of depth 10.

One tree: error 0.70. Twenty trees: 0.50. A hundred: 0.49. Most of the gain arrives in the first 20 trees; after that, depth and leaf size decide the result, not tree count.

Where a bootstrap sample's rows come from Chart

In the tree's sample (unique rows)63.2 %
Left out = out-of-bagused as a free test set for that tree36.8 %

For a large dataset each tree leaves out about 1/e of the rows (36.8%).

Gradient boosting, step by step6 minFit the residuals, take a small step, repeat - worked by hand for two rounds - and the round where more trees start to hurt.

Boosting, round by round Illustration

Training error 0.84 · new data 1.04

Training error 0.55 · new data 0.71

Training error 0.24 · new data 0.35

Training error 0.02 · new data 0.36

Synthetic data (60 training points, 400 new points from the same curve); depth-2 trees, learning rate 0.3. Each panel adds trees fitted to the errors left by the previous ones.

Inside one boosting round: fit what is still wrong, take a small step Illustration

Round 1: training MSE 0.92 → 0.37; on new draws 0.92 → 0.41.

Round 2: training MSE 0.37 → 0.22; on new draws 0.41 → 0.28.

Round 3: training MSE 0.22 → 0.17; on new draws 0.28 → 0.23.

Round 25: training MSE 0.02 → 0.02; on new draws 0.22 → 0.22. The trees now chase noise: training error keeps falling while new-data error sits above its low of 0.19 at round 11. That turn is what early stopping finds on a validation block.

tree fitted to the residuals (top)
model before the round (dashed)
model after adding 0.5 x the tree

Synthetic data: 50 noisy points around a smooth curve; depth-2 regression trees, learning rate 0.5, squared-error loss (so the negative gradient is the residual). New-draw MSE = squared distance to the known curve plus the noise variance 0.12. Fitted on the page with the Day 12 lab code.

Top row - the residuals and the depth-2 tree fitted to them. Bottom row - the model before (dashed) and after the step.

One boosting round Formula

F_m(x) = F_{m-1}(x) + \nu\, h_m(x)

$h_m$: a shallow tree fitted to the current errors (the negative gradient of the loss)
$\nu$: learning rate; small steps need more rounds but generalize better
$F_m$: the ensemble after m rounds

For squared error the "errors" are simply the residuals.

One boosting round, as a loop Loop

Current model Fstarts as the meanpredict ↓
Residuals y − Fnegative gradient of the lossfit to them ↓
Small tree hdepth 3, a few leavesshrink by ν ↓
F ← F + ν·ha small step, ν ≈ 0.1score ↓
Validation lossstop at the best roundnext round ↓
↺ back to 1 · Current model F

Every round re-reads what the current model still gets wrong and corrects a fraction of it. Stopping is a tuning choice: pick the round on a validation block, not the test block.

Two boosting rounds by hand: five days of energy demand Table

Step	What happens	Numbers
Data	temperature index x = 1..5; daily load y (GWh)	y = 10, 12, 18, 20, 30
F0 = mean	the best single constant for squared error	F0 = 18; residuals -8, -6, 0, 2, 12; MSE 49.6
Round 1 stump	best split of the residuals: x < 4.5	left mean -3, right mean +12
Step, rate 0.5	F1 = F0 + 0.5 x stump	F1 = 16.5 (x up to 4), 24.0 (x = 5)
New residuals	y - F1; the next stump sees only these	-6.5, -4.5, 1.5, 3.5, 6.0; MSE 22.6
Round 2 stump	best split of the new residuals: x < 2.5	left mean -5.5, right mean +3.67
Step again	F2 = F1 + 0.5 x stump	F2 = 13.75, 13.75, 18.33, 18.33, 25.83; MSE 7.5

Hypothetical grid-operations numbers. sklearn's GradientBoostingRegressor (2 rounds, rate 0.5, depth 1) returns exactly these predictions; each round fits a small tree to what is still wrong and takes half the step.

Losses, learning rate and early stoppingadvanced4 minWhy boosting is gradient descent in function space, what the trees fit under other losses, and how the rate trades rounds for safety.

What each tree is fitted to, by loss (Friedman, 2001) Table

Loss	Pseudo-residual (negative gradient)	Leaf value	Use when
Squared error	y - F(x), the plain residual	Mean of the residuals in the leaf	The mean is the target; few extreme errors
Absolute error	sign(y - F(x)), +1 or -1	Median of the residuals in the leaf	Median target; robust to outliers
Huber	Residual, clipped at a threshold	Robust compromise of the two	Mostly squared error, some wild rows
Log loss (0/1)	y - p, with p = sigmoid(F)	One Newton step: sum(y - p) / sum p(1 - p)	Classification; F is a log-odds score

Boosting is gradient descent in function space: each round steps along the negative gradient at the training rows, and the tree carries that step to new x. The tree is always grown on the pseudo-residuals; only the leaf values and what F estimates change with the loss.

Learning rate trades rounds for a wider, safer minimum (computed) Chart

learning rate 0.3
learning rate 0.1
learning rate 0.03

GradientBoostingRegressor, depth 2, 300 synthetic training rows, 4,000 validation rows (noise variance 0.25). All three minima are 0.278-0.280; the smaller rate needs 10 times the rounds and punishes a late stop far less.

XGBoost, LightGBM, CatBoost: what is differentadvanced4 minDesign points from the original papers and documentation, and the defaults each library hands you.

XGBoost, LightGBM, CatBoost: documented design points Comparison

Criterion	XGBoost	LightGBM	CatBoost
Main design point	Regularized objective: penalized leaves, shrinkage, column subsampling	Histogram binning with leaf-wise (best-first) growth	Ordered boosting and ordered target statistics
Speed device	Sparsity-aware splits; weighted quantile sketch	Gradient-based one-side sampling; exclusive feature bundling	Built for categorical features without target leakage
Watch out for	Many parameters: start with depth, min_child_weight, learning rate	Leaf-wise growth can overfit small data: cap num_leaves, min_data_in_leaf	Check training time on large data; compare on your own split

Facts from the original papers and documentation; no ranking is implied.

Defaults you inherit (installed versions, checked) Table

Library	Size of each tree	Rate and rounds	Early stopping and sampling
scikit-learn 1.9.1 RandomForest	Unlimited depth, min_samples_leaf 1	100 trees	Bootstrap on; max_features 1.0 (regressor) or sqrt (classifier)
scikit-learn 1.9.1 HistGradientBoosting	max_leaf_nodes 31, min_samples_leaf 20	learning_rate 0.1, max_iter 100	On automatically above 10,000 rows: 10% validation, 10 rounds patience
XGBoost 3.4.1	max_depth 6, min_child_weight 1, lambda 1	eta 0.3, 100 rounds (sklearn API)	Off unless an eval set and early_stopping_rounds are given; hist tree method
LightGBM 4.7.0	num_leaves 31, no depth limit, min_child_samples 20	learning_rate 0.1, 100 rounds	Off unless requested (callback or parameter); no row or column sampling by default

Read from the installed packages (signatures and a saved XGBoost config). A default forest grows unpruned trees; a default HistGradientBoosting quietly holds out 10% of a large table. Defaults differ this much, so a fair comparison tunes each library on the same time-ordered folds.

Tuning, in the right order3 minWhich knob moves which risk.

What to change when Diagram

Training error high?
- yes: to More capacity
no
Large train-val gap?
- yes: to More regularization
- no: to Done
More capacitydeeper trees, more rounds
- re-check: to Done
More regularizationshallower, lower rate, subsample, stop early
then
Donecheck seeds and time-ordered folds

Which knob controls what Table

Goal	Forest	Boosting
Less overfitting	Shallower trees; larger minimum leaf size	Shallower trees; lower learning rate; subsampling; early stopping
Less variance	More trees (plateaus); fewer features per split	Row subsampling per round (stochastic boosting, a bagging-like cut); column subsampling; regularization
Less bias	Deeper trees	More rounds or slightly deeper trees
Speed	Fewer trees; parallel by tree	Histogram methods; fewer rounds with a higher rate

The stopping round is a tuning try: pick it on a validation block, never on the test block (Day 15).

Where tree ensembles fail3 minSix failure modes to test before adopting, the misleading importance chart among them.

Six ways tree ensembles fail Overview

Extrapolation
Flat outside the range; reframe the target.
Optimism
Repeated or neighbouring rows inflate out-of-bag and shuffled-CV scores.
Calibration
Scores are not probabilities until checked.
Leakage
Stopping or tuning on the test block turns it into a tuning set.
Importance bias
Impurity importance favours many-valued columns (Strobl et al., 2007); use held-out permutation importance (Day 17).
Shared credit
Two correlated inputs split the credit arbitrarily between refits, as coefficients did on Day 11.

A sensible order of work2 minFive steps from baseline to stress test.

A sensible order of work Sequence

Baseline
Name the Day 2 baseline and the metric unit.
Linear model
A regularized linear model is the honest second bar.
Random forest
Strong default with few knobs; read the out-of-bag error.
Boosted trees
Tune depth and learning rate, stop early on a validation block.
Stress test
Time-ordered split, inputs outside the range, calibration.

Memory anchor

Forests average away variance, boosting chips away bias, and neither can leave the range it has seen.

Useful for

Tabular problems with interactions, thresholds, mixed types and missing values - credit risk, claims severity, machine failures, churn - where a linear model leaves a gap that a strong baseline cannot close.
A fast, robust second opinion on how much signal the columns hold, with out-of-bag estimates available for forests.
Models for the residual of a baseline, where the ensemble corrects a transparent forecast instead of replacing it.

Not suitable for

Series that trend or drift out of range unless the target is detrended or the trend is carried by another component (Day 24).
Very small samples and strict monotone or audit requirements, where a regularized linear model or a rule is easier to defend (Day 11).

Traps

Stopping on the block you report

Early stopping picks the round with the lowest validation error, so that number is optimistic by construction. Reporting it as the expected live error credits the model with the luck of the selection.

How to avoid it

Keep a block that played no part in stopping or tuning, and report on it once (Day 15).

Reading a flat forecast as caution

Out-of-range inputs produce predictions that look plausible and stable, because they sit at the edge of the training targets. Monitoring on error alone sees the problem only after the actuals arrive.

How to avoid it

Monitor input ranges against the training range and compare the maximum prediction with the maximum target.

Senior reasoning

A senior asks what the ensemble adds over the linear model and the baseline, in the decision's unit (seats, EUR of credit loss, kWh of forecast error), on a split that matches deployment, before looking at libraries. They know that depth, learning rate and rounds trade bias against variance and that none of them creates extrapolation. They reframe targets so that the part that trends is carried by a transparent component, add a range guard, and choose among XGBoost, LightGBM and CatBoost on documented properties such as categorical handling and the team's tooling, not on a leaderboard.

Interview question

"When would you not use a gradient-boosted tree model?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What does each of these control in tree ensembles - depth, number of trees, learning rate, rounds - and which one overfits?
Why can a tree ensemble not extrapolate, and what two design moves keep it useful when inputs trend?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Greedy function approximation: A gradient boosting machine — Jerome H. Friedman, 2001optional
Gradient boosting as steepest descent in function space; each tree is fitted to the negative gradient (pseudo-residuals), with per-loss leaf values (mean for squared error, median for absolute error) and shrinkage.
Bias in random forest variable importance measures: Illustrations, sources and a solution — Carolin Strobl, Anne-Laure Boulesteix, Achim Zeileis, Torsten Hothorn, 2007optional
Impurity-based variable importance in random forests is biased towards predictors with many possible split points.
XGBoost: A Scalable Tree Boosting System — Tianqi Chen, Carlos Guestrin, 2016core
Describes the regularized learning objective (penalty on leaves and leaf weights, section 2.1), shrinkage and column subsampling (2.3), the weighted quantile sketch (3.3) and sparsity-aware split finding (3.4). (Sections 2.1, 2.3, 3.3 and 3.4)
LightGBM: A Highly Efficient Gradient Boosting Decision Tree — Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, Tie-Yan Liu, 2017optional
Introduces gradient-based one-side sampling and exclusive feature bundling and states the reported speed-up under the authors' benchmarks. (Abstract and Sections 3 and 4)
Features (LightGBM documentation) — LightGBM developers (Microsoft)core
Documents histogram-based splitting, leaf-wise growth with its overfitting risk on small data, and optimal splits for categorical features. (Leaf-wise (Best-first) Tree Growth; Optimal Split for Categorical Features)
CatBoost: unbiased boosting with categorical features — Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, Andrey Gulin, 2018optional
Defines ordered target statistics (section 3.2) and ordered boosting against prediction shift caused by target leakage (section 4). (Sections 3.2 and 4)
Ensembles: Gradient boosting, random forests, bagging, voting, stacking (scikit-learn User Guide 1.11) — scikit-learn developerscore
Random forests average bootstrapped trees with random feature subsets, out-of-bag estimates, and HistGradientBoosting early stopping defaults. (1.11.2 Random forests; 1.11.1.1.1 Early stopping)
Notes on Parameter Tuning (XGBoost documentation) — XGBoost developersoptional
States which parameters (max_depth, eta, min_child_weight, gamma, subsample) control model complexity and noise robustness in XGBoost. (Control overfitting)

Optional sections

Deep dive · 10 min

Why leaves average. For squared error, the best constant in a leaf is the mean of its targets; for absolute error the median; for a classification leaf, the class share. A split search tries every threshold of every feature and keeps the one with the largest drop in impurity. Because a tree is piecewise constant, its predictions are bounded by the targets it saw, which is the whole extrapolation story.

Variance reduction in numbers. Averaging $B$ trees with variance $\sigma^2$ and pairwise correlation $\rho$ gives variance $\rho\sigma^2 + (1-\rho)\sigma^2/B$ . Bootstrap samples and random feature subsets lower $\rho$ ; more trees shrink only the second term, which is why forests plateau.

What boosting fits. At round $m$ the new tree is fitted to the negative gradient of the loss at the current predictions: residuals for squared error, a signed share for log loss. The library papers differ in how they find splits (exact, approximate quantile sketch, histograms) and whether they use second-order information; XGBoost's objective uses both gradient and Hessian statistics, which is why its leaf weights have a closed form.

Categorical features. One-hot encoding of a high-cardinality column produces many sparse columns and deep trees. LightGBM's documentation describes an optimal split of categories found by sorting them by gradient statistics; CatBoost's paper replaces the category with an ordered target statistic computed only from rows earlier in a random permutation, which is its answer to the leakage in naive target encoding (Day 9). Verify the exact options in the version you install.

What to report. Out-of-bag error and validation error measure different things: the first is free but assumes exchangeable rows; the second matches the deployment split you design (Day 15). Impurity-based importance is biased toward high-cardinality features and is not an effect (Day 17).

Hands-on lab: Depth, out-of-bag error and early stopping in three libraries · 20 min

Run the script. It models the pickup on one aircraft type. Part A fits single trees of increasing depth, part B a forest with out-of-bag scoring, part C boosts with early stopping in XGBoost, LightGBM and CatBoost on the same validation block. Then:

At which depth does the single tree start to overfit, and what does the training-versus-test gap look like?
How close is the out-of-bag MAE to the test MAE, and when would it not be?
The three libraries stop at different rounds with nearly the same test MAE. What does that say about choosing a library on accuracy alone?

import numpy as np
import lightgbm as lgb
import xgboost as xgb
from catboost import CatBoostRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error as mae
from sklearn.tree import DecisionTreeRegressor

rng = np.random.default_rng(12)
n = 4500                                          # synthetic departures, date order, one aircraft type
dow, hol = rng.integers(0, 7, n), rng.random(n) < 0.10
wk = dow >= 4
bk14 = 180 * np.clip(rng.normal(0.50 + 0.08 * wk + 0.08 * hol, 0.10), 0.10, 0.90)
pickup = 14 + 8 * wk + 36 * (wk & hol) - 8 * (hol & ~wk) + 0.12 * np.maximum(95 - bk14, 0) + rng.normal(0, 7, n)
X, y = np.c_[bk14, wk, hol, dow], pickup          # model the pickup (seats sold between D-14 and D-0)
tr, va, te = slice(0, 2400), slice(2400, 3000), slice(3000, n)   # train / validation (for stopping) / test

# A) One tree: depth is the variance dial. Training error keeps falling; test error turns up.
for d in (2, 4, 8, 16):
    t = DecisionTreeRegressor(max_depth=d, random_state=0).fit(X[tr], y[tr])
    print(f"A) depth {d:2d}: train MAE {mae(y[tr], t.predict(X[tr])):.2f} | test MAE {mae(y[te], t.predict(X[te])):.2f}")

# B) Bagging: average deep trees on bootstrap samples; out-of-bag rows give a free error estimate
rf = RandomForestRegressor(300, min_samples_leaf=3, oob_score=True, random_state=0).fit(X[tr], y[tr])
print(f"B) forest: OOB MAE {mae(y[tr], rf.oob_prediction_):.2f} | test MAE {mae(y[te], rf.predict(X[te])):.2f}")

# C) Boosting with early stopping on the validation block; three libraries, same data, same budget
xg = xgb.XGBRegressor(n_estimators=2000, learning_rate=0.05, max_depth=4, early_stopping_rounds=50, verbosity=0)
xg.fit(X[tr], y[tr], eval_set=[(X[va], y[va])], verbose=False)
dtr = lgb.Dataset(X[tr], y[tr])
booster = lgb.train({"learning_rate": 0.05, "num_leaves": 15, "verbose": -1}, dtr, 2000,
                    valid_sets=[lgb.Dataset(X[va], y[va], reference=dtr)], callbacks=[lgb.early_stopping(50, verbose=False)])
cb = CatBoostRegressor(iterations=2000, learning_rate=0.05, depth=4, early_stopping_rounds=50, verbose=0)
cb.fit(X[tr], y[tr], eval_set=(X[va], y[va]))
for name, rounds, p in [("xgboost", xg.best_iteration, xg.predict(X[te], iteration_range=(0, xg.best_iteration + 1))),
                        ("lightgbm", booster.best_iteration, booster.predict(X[te], num_iteration=booster.best_iteration)),
                        ("catboost", cb.get_best_iteration(), cb.predict(X[te]))]:
    print(f"C) {name:8s} best round {rounds:4d} of 2000 | test MAE {mae(y[te], p):.2f}")

Interview practice · 10 min

Your gradient-boosted model forecasts store demand for a grocery chain and is excellent on backtests, but planners notice that it never forecasts a record high, even in peak weeks when demand clearly exceeds history. Explain and propose a fix.

Answer as you would in the interview.

Content version 7 · last verified 2026-10-11 · depends on scikit-learn, xgboost, lightgbm, catboost, numpy

1 Recall · 5 min

2 Concept · 10 min