Core day 11 of 60 · Week 3 · Part I · application depth

Linear and Logistic Regression

A regression reports that a EUR 1 higher fare goes with 0.9 more final bookings. Before you read that aloud: does the model beat the Day 2 pickup baseline on departures it has not seen, which residual pattern would undermine it, and why is the number not an answer to "what if we raise fares"?

By the end: You fit a regression and a logistic regression behind the Day 9 boundary, compare them with the pickup baseline in seats, read a coefficient as a conditional association, and name which residual pattern breaks which claim.

At a glance

A weighted sum fitted by a loss; read predictions against the baseline, coefficients as conditional, and residuals for what they break.

Weighted sum
Each weight moves the prediction with the other inputs held fixed.
Log-odds
Logistic regression is linear in log-odds; a sigmoid turns it into a probability.
Not causal
Coefficients shift when columns change; a price effect needs a design (Day 5).
Residuals
Curve, funnel, runs, one far point: each pattern threatens something different.
Loss vs optimizer
The loss defines the model; the solver only finds it. 0.5 is not a decision rule.

Recall · 5 min

Without notes: write the Day 2 framing card for the D-14 fare-class decision in one line per item, then say which rows a scaler or an imputer may be fit on (Day 9). Today every model you fit must be scored against that card's baseline on rows after the boundary.

Open due reviews

Concept · 10 min

A linear model is a weighted sum of the inputs. Each weight says how the prediction moves when one input rises by one unit while the other inputs in the model stay put. Logistic regression reads the same sum as log-odds and squeezes it into a probability. Fitting picks the weights; the skill is knowing what they do not say.

One far-away point tilts the line, and curvature or widening noise appear only in the residuals

Predict first

With a straight truth and constant noise, you add one point at x = 14 (the others lie between 0 and 10) with a very low y. What happens to the slope?

Text description of the figure

A scatter of 40 points with the least-squares line, the true relationship and, optionally, one draggable outlier. Below it a residual plot against fitted values with a binned mean. Toggles switch the truth between straight and curved and the noise between constant and growing with x. A readout lists slope, R squared, classical and robust standard errors, leverage and Cook's distance.

One weighted sum, two readings Diagram

Inputs at D-14bk14, fare, weekend
Weighted sum zintercept + weights x inputs
- squash to 0-1: to Logistic: sigmoid(z)
read as seats
OLS: z is the forecastloss: squared residuals
- to Seats
Logistic: sigmoid(z)loss: log loss
- to Probability
Seatscompare with the baseline
Probabilitythreshold from costs (Day 13)

Same linear part, different loss and output scale. Neither output is an action until a decision rule is attached.

Log-odds in, probability out Chart

z runs from -6 to 6, p from 0 to 1. Equal steps in z are unequal steps in p: steep in the middle, flat in the tails.

The fare coefficient moves with the column list, so it is not a price effect Chart

Without bk14absorbs demand that fares respond to0.91 seats per EUR
With bk14most demand signal removed-0.007 seats per EUR
True effect (simulator)known only because we built the data-0.1 seats per EUR

Output of the application run. The same model can still forecast well: predict with it, and leave the price effect to an experiment or a design (Day 5).

Four residual plots, four different verdicts Illustration

Healthy: a flat band; the binned mean stays on zero. R² 0.92. This is the only panel where the classical standard errors and one common error band are both fine.

Curved truth: R² is 0.91, yet the binned mean bends into a U, positive at both ends and negative in the middle. The form is wrong: add terms or move to trees, then refit.

Funnel: residual SD 1.0 in the lowest quarter of fitted values, 4.2 in the highest. The line is fine; the classical standard errors and a constant-width interval are not.

One point at x = 16 with a low y: leverage 0.16 against an average of 0.025, Cook's distance 5.56. The slope drops from 1.50 to 1.22: one row out of 81 carries it.

Synthetic data (80 seeded points per panel); straight-line OLS fits computed on the page. Thick lines in the residual panels are the mean residual in six equal-count bins of the fitted value; the funnel band is 1.6 times the mean absolute residual per bin.

Each panel is a real OLS fit. Read the binned mean and the spread, not R squared, before you quote anything.

Key terms: residuals, Log-odds (logit), log loss, Heteroscedasticity, Multicollinearity, Confounder and Regularization. The pictures above carry the mechanism; the optional technical depth below has the formulas, diagnostics and penalties.

The Day 2 card still governs: close class L at D-14, one departure per row, target bookings_final, pickup baseline, seats of absolute error, about EUR 80 per empty seat.
Draw the Day 9 boundary first, then fit.

Assumptions

The relationship is close to linear in the columns given; otherwise add terms or move to trees (Day 12).
Rows are roughly exchangeable; repeated customers, patients, flights or neighbouring dates make standard errors too small (Day 15).
Predictors exist at the cutoff; a coefficient on a column unknown at D-14 is a leak, not a finding (Day 2).

Application and workplace decision · 10 min

The script builds 4,000 synthetic departures in date order: the first 3,000 train, the last 1,000 test (the Day 9 boundary).
Fares are quoted higher when expected demand is high, a confounder the model never sees.
It scores the pickup baseline, an OLS regression and a logistic regression for "150 or more bookings".
Before running it, guess the sign of the fare coefficient with and without bk14.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import log_loss, mean_absolute_error

rng = np.random.default_rng(11)
n, cut = 4000, 3000                      # synthetic departures in date order; Day 9 boundary: first 3,000 train
demand = rng.normal(size=n)              # expected demand: drives fares AND bookings, unseen by the model
weekend = rng.integers(0, 7, n) >= 4
fare = 110 + 25 * demand + rng.normal(0, 8, n)                # RM quotes higher fares when demand is high
bk14 = 95 + 22 * demand + 8 * weekend + rng.normal(0, 6, n)   # bookings on hand at the D-14 cutoff
pickup = 28 + 6 * demand + 4 * weekend - 0.10 * (fare - 110) + rng.normal(0, 1, n) * (1 + 0.08 * bk14)
final = bk14 + pickup                    # target; the true fare effect on pickup is -0.10 seats per EUR
X, tr, te = np.c_[bk14, fare, weekend], slice(0, cut), slice(cut, n)

base = bk14[te] + pickup[tr].mean()      # Day 2 baseline: bookings on hand + average past pickup
print(f"pickup baseline MAE {mean_absolute_error(final[te], base):.2f} seats/departure")
ols = LinearRegression().fit(X[tr], final[tr])
alone = LinearRegression().fit(X[tr][:, [1, 2]], final[tr])
print(f"OLS MAE {mean_absolute_error(final[te], ols.predict(X[te])):.2f}; fare coefficient "
      f"{ols.coef_[1]:+.3f} with bk14, {alone.coef_[0]:+.3f} without it (true effect -0.10)")
fit = ols.predict(X[te]); res = final[te] - fit; order = np.argsort(fit)
print(f"residual SD, lowest vs highest 250 fitted values: {res[order[:250]].std():.1f} vs {res[order[-250:]].std():.1f}")

sold_out = final >= 150                  # classification target: a near-full departure
clf = LogisticRegression(C=1e6, max_iter=1000).fit(X[tr], sold_out[tr])   # C=1e6: almost no penalty
p = clf.predict_proba(X[te])[:, 1]
i = np.abs(p - 0.3).argmin()             # one departure whose probability is near 0.3
z = clf.decision_function(X[te][i:i + 1])[0]   # its log-odds (the linear part of the model)
print(f"logit {z:+.2f} -> probability {1 / (1 + np.exp(-z)):.3f} (predict_proba {p[i]:.3f})")
print(f"log loss {log_loss(sold_out[te], p):.3f} vs base rate {log_loss(sold_out[te], np.full(len(p), sold_out[tr].mean())):.3f}")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 32 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Least squares: the model, the normal equations, a fit by hand7 minWhat OLS estimates, how the estimator is derived, a three-point fit to redo on paper, and when an expert reaches elsewhere.

The linear model, with its assumptions written as math Formula

\begin{gathered} y = X\beta + \varepsilon, \quad \operatorname{rank}(X) = p \\ \mathbb{E}[\varepsilon \mid X] = 0, \quad \operatorname{Var}(\varepsilon \mid X) = \sigma^2 I \end{gathered}

$X\beta$: n rows by p columns (intercept included) times p unknown coefficients: the conditional mean of y
$\mathbb{E}[\varepsilon \mid X] = 0$: the form is right and no omitted input moves with X; breaks with curvature or confounding
$\sigma^2 I$: one error variance, no correlation between rows; breaks with funnels, repeated customers or dates
$\operatorname{rank}(X) = p$: no column is an exact combination of the others; otherwise beta is not identified

Mean-zero errors protect the coefficients, the variance condition only their standard errors, the rank condition whether a unique answer exists. Normal errors are an extra, optional assumption for small-sample tests.

What least squares minimizes Formula

\hat b = \arg\min_b \sum_i \left(y_i - x_i^\top b\right)^2 = (X^\top X)^{-1} X^\top y

$y_i - x_i^\top b$: residual of row i: actual minus fitted
$(X^\top X)^{-1} X^\top y$: closed-form solution; libraries use QR or SVD, not an explicit inverse
$\hat y$: estimates the conditional mean of y given the inputs

Squared error is the Gaussian log-likelihood up to constants, which is why it targets the mean (absolute error targets the median, Day 16).

From the loss to the normal equations Formula

\begin{gathered} \nabla_b \sum_i (y_i - x_i^\top b)^2 = -2\,X^\top (y - Xb) = 0 \\ \Rightarrow\; X^\top X\, \hat b = X^\top y \end{gathered}

$-2X^\top(y - Xb)$: gradient of the RSS; zero means every column is uncorrelated with the residuals
$X^\top X\,\hat b = X^\top y$: the normal equations: p linear equations in the p coefficients
$X^\top X \text{ singular}$: a column is a combination of others: no unique solution (collinearity chapter)

The RSS is a convex bowl in b, so the zero-gradient point is the minimum. Solving costs about n p^2 operations via QR or SVD.

Least squares by hand: three points, one line Table

Step	Formula	Value
Data (x, y)	ad spend (k EUR), sign-ups (hundreds)	(1, 2), (2, 2), (3, 4)
Sums	Sx, Sy, Sxy, Sxx	6, 8, 18, 14
Slope	(n Sxy - Sx Sy) / (n Sxx - Sx^2)	(54 - 48) / (42 - 36) = 1.0
Intercept	(Sy - slope x Sx) / n	(8 - 6) / 3 = 0.667
Fitted values	0.667 + 1.0 x	1.67, 2.67, 3.67
Residuals	y - fitted; with an intercept they sum to 0	+0.33, -0.67, +0.33
RSS	sum of squared residuals	0.111 + 0.444 + 0.111 = 0.667
Noise variance	s^2 = RSS / (n - p), with p = 2 coefficients	0.667 / 1 = 0.667
SE of the slope	sqrt(s^2 / sum of (x - mean x)^2) = sqrt(0.667 / 2)	0.577: one degree of freedom, so very wide

Hypothetical marketing numbers, verified with numpy.linalg.lstsq. Any other line does worse: slope 1 with intercept 1 has RSS 1.0 against 0.667. The SE is the diagonal of s^2 (X'X)^-1.

Each assumption you add buys OLS one more property Stack

+ normal errors (optional)Exact small-sample t and F tests. Nothing below needs it, and with many rows the tests hold approximately anyway.
+ one variance, independent rowsGauss-Markov: the smallest variance of any linear unbiased estimator. Classical SE = square root of the diagonal of σ²(X'X)⁻¹.
+ mean-zero errors, E[ε | X] = 0Unbiased, and consistent: with more rows b̂ settles at the population β, which is still a conditional, not a causal, quantity.
rank(X) = pA unique b̂ exists. Near-duplicate columns make X'X nearly singular and the coefficients unstable (collinearity chapter).

Each layer rests on the ones below it. The SE is the spread of b̂ over repeated samples from the same process: it shrinks like 1/√n and grows when an input barely varies. None of the layers stops one high-leverage row from carrying the slope (diagnostics chapter).

Linear model or a neighbour? Comparison

Criterion	OLS / logistic	Splines / GAM	Tree ensembles (Day 12)
Shape it can fit	Additive and straight in the inputs you give it	Smooth curves per input, still additive	Interactions and thresholds found automatically
Outside the training range	Continues the line: right if the trend holds, wild if not	Usually linear beyond the last knot; check it	Flat: repeats the last leaf averages
What you can read off it	Conditional coefficients and their standard errors	One curve per input instead of one number	Only through post-hoc tools (Day 17)
Reach for it when	Few rows, a transparent challenger, effects roughly additive	Residuals bend but interactions are few	Many rows, strong interactions, prediction only

An expert skips the linear model when residuals show interactions it cannot express and the goal is prediction; keeps it when rows are few or a reviewer must read it.

Logistic regression: the MLE and its gradient6 minThe model as maximum likelihood, the log-loss gradient with one step worked by hand, and why the solver is not the model.

Logistic regression in three scales Formula

z = b_0 + \sum_j b_j x_j, \quad \text{odds} = e^{z}, \quad p = \frac{1}{1 + e^{-z}}

$z$: log-odds: linear in the inputs, any real number
$e^{b_j}$: odds multiplier for one more unit of x_j, others fixed
$p$: probability; the change from one unit of x_j depends on where p starts

Quote e^b (an odds multiplier) or a probability change at a stated starting p, never b itself as a probability. The 0.5 cut is a decision rule, not part of the model (Day 13).

Logistic regression is maximum likelihood Sequence

Model
Each row is a coin flip: P(y_i = 1) = sigmoid(x_i' b), a probability that moves with the inputs.
Likelihood
Probability of the observed labels: the product of p_i over events and (1 - p_i) over non-events.
Take -log / n
Products become sums: maximizing the likelihood is minimizing the mean log loss.
Gradient
d loss / d b = X'(p - y) / n: each row's error (p_i - y_i), weighted by its inputs.
Solve
No closed form, but the loss is convex: gradient descent, BFGS and L-BFGS reach one minimum.
Uncertainty
The MLE is consistent and roughly normal; SE from (X'WX)⁻¹, W = diag(p_i(1 - p_i)). Newton's method here is reweighted least squares.

Squared error plays the same role for OLS under Gaussian noise; both losses are a likelihood in disguise.

Log loss, the logistic objective Formula

\ell(b) = -\frac{1}{n}\sum_i \left[y_i \log p_i + (1 - y_i)\log(1 - p_i)\right]

$y_i$: 1 if the departure reached 150 bookings, else 0
$p_i$: the model's probability for row i
$\nabla \ell$: X^T (p - y) / n; the loss is convex, so every descent method finds the same minimum

Log loss is the Bernoulli negative log-likelihood. The run: 0.193 for the model against 0.501 for the base rate.

One gradient step by hand (three loans, one weight) Table

Quantity	Formula	Value
Data (x, y)	card utilization score x; default y	(1, 0), (2, 1), (3, 1)
Start b = 0	p_i = sigmoid(0) for every row	p = 0.5, 0.5, 0.5; log loss 0.693
Gradient	mean of (p_i - y_i) x_i	(0.5 - 1.0 - 1.5) / 3 = -0.667
Step (rate 1)	b minus 1 x gradient	b = 0.667
New p	sigmoid(0.667 x)	0.661, 0.791, 0.881
New loss	mean log loss at b = 0.667	0.481, down from 0.693

Hypothetical credit-scoring numbers, verified in numpy. The two defaulters sat at high x, so the gradient pushed b up; every solver repeats this until the gradient is zero.

Objective versus optimizer Comparison

Criterion	Objective (loss)	Optimizer (solver)
Answers	What counts as a good fit	How to reach the minimum
Example	Squared residuals (OLS); log loss (logistic); plus a penalty	Closed form, gradient descent, BFGS, L-BFGS (sklearn default)
Change it and	the model and its coefficients change	the same coefficients, faster or slower (convex losses)

In the lab, gradient descent, BFGS and L-BFGS all reach log loss 0.2136 with coefficients -3.69, 4.14 and 0.36.

The same +0.5 in log-odds, five starting points Table

Start p	New p	Change in p
0.05	0.080	+0.030
0.25	0.355	+0.105
0.50	0.622	+0.122
0.75	0.832	+0.082
0.95	0.969	+0.019

Computed: p' = sigmoid(logit(p) + 0.5). The odds always rise by e^0.5 = 1.65; the change in p is largest near 0.5 because dp/dz = p(1 - p). The same p(1 - p) weights each row in (X'WX)⁻¹: rows scored near 0 or 1 add almost nothing, so in a churn, fraud or default model the coefficients' precision comes from the uncertain middle.

Perfect separation: without a penalty the loss has no minimum Chart

no penalty
C = 10
C = 1 (default)

Six hypothetical loans, x = -2, -1, -0.5 repaid and 0.5, 1, 2 defaulted: a cut at 0 separates them. Mean log loss keeps falling as b grows; the penalty b²/(2Cn) turns it back up. scikit-learn returns b = 1.18 at C = 1, 2.96 at C = 10 and 21.5 at C = 1e6: a huge C brings the problem back.

Why a coefficient is conditionaladvanced3 minThe omitted-variable formula reproduces the +0.91 from the simulator's own numbers.

Why the fare coefficient came out at +0.91 Formula

\hat b_{\text{fare}} \approx b_{\text{fare}} + b_{\text{demand}} \cdot \frac{\operatorname{Cov}(\text{demand}, \text{fare})}{\operatorname{Var}(\text{fare})}

$b_{\text{fare}}$: true effect of fare on final bookings: -0.10 seats per EUR
$b_{\text{demand}}$: direct effect of the omitted demand: 22 via bk14 + 6 via pickup = 28 seats per SD
$\operatorname{Cov}/\operatorname{Var}$: how strongly fare moves with demand: 25 / (25^2 + 8^2) per EUR

Omitted-variable formula: -0.10 + 28 x 25 / 689 = 0.92, against 0.91 fitted. Adding bk14 removes most of the demand term, but not all of it.

One coefficient, two questions: the data answer only the first Comparison

Criterion	Conditionalwhat a coefficient measures	Interventionalwhat a pricing decision needs
The question	Among departures with equal bk14 and weekend, how do bookings differ with fare?	If we raise the fare by EUR 10, how many seats change?
What answers it	The observational rows, priced as they were	An experiment or a design (Day 5)
Today's number	+0.91 without bk14, -0.007 with it	-0.10 per EUR, known only from the simulator
Adding columns	Changes the number: that is what conditional means	Searching until the sign looks right is not identification

Forecasting needs only the first question. No column list turns it into the second.

Residual diagnostics, assumption by assumption4 minWhich residual pattern threatens which claim, and what to do first.

Which assumption protects what Table

Assumption	Residual sign when it fails	What it threatens	Remedy
Linear form	Curve in residuals against fitted	Bias in predictions and coefficients	Add terms, transform, or use trees (Day 12)
Independent errors	Runs by date, or within a patient, store or route	Standard errors, not the line	Cluster by the repeating unit
Constant spread	Funnel shape (heteroscedasticity)	Standard errors and intervals (Day 26)	Robust (HC) errors, log target, weights
Normal residuals	Heavy tails in a QQ plot	Small-sample p-values only	Usually nothing with large n
No dominant point	One point with high leverage	The whole slope	Check Cook's distance; refit without it

Reading a residual plot, in order Diagram

Curved residuals?
- yes: to Fix the form first
no
Funnel or runs?
- yes: to Fix the standard errors
no
One far point?
- yes: to Check influence
- no: to Quote the fit
Fix the form firstterms, transform, trees
Fix the standard errorsrobust or clustered
Check influenceleverage, Cook's distance
then
Quote the fitconditional, against the baseline

Standard errors that match your rowsadvanced3 minClassical, robust and clustered errors share one line and differ only in the middle of one formula.

Three standard errors for one line Comparison

Criterion	Classical	Robust (HC)	Clustered
Assumes	Independent rows, one error variance	Independent rows, spread may vary	Rows independent across clusters only
Use when	Plain residual band, no repeated units	Funnel-shaped residuals	Many rows per customer, patient, store or route
Reliable when	Its assumptions hold, at any n	Enough rows: it is a large-sample correction	Many clusters; worry below a few dozen (practitioner heuristic)
Leaves alone	The coefficients	The coefficients	The coefficients

All three keep the same fitted line. With 90 rows per route and a within-route error correlation of 0.3 (hypothetical), the Moulton factor for a route-level input, 1 + 89 x 0.3 = 27.7, makes each route's 90 rows worth about 3.2 independent ones: cluster by the unit that repeats.

Where robust and clustered standard errors come from Formula

\widehat{\operatorname{Var}}(\hat b) = (X^\top X)^{-1}\Big(\sum_g X_g^\top \hat e_g \hat e_g^\top X_g\Big)(X^\top X)^{-1}

$(X^\top X)^{-1}$: the bread: the same matrix the classical formula uses
$\hat e_g$: residuals of group g; one row per group gives robust (HC0), one customer or route per group gives clustered
$\sigma^2 (X^\top X)^{-1}$: the classical formula: what the middle collapses to with independent rows and one error variance

The coefficients never change, only the middle term. Robust errors let each row's spread differ; clustered errors also let rows of one group move together.

Collinearity: what the data can and cannot identify4 minNear-duplicate inputs leave the sum of their effects and the predictions intact and make the split of credit arbitrary.

Correlated inputs - does it matter for your question? Map

Conditional reading

Coefficients are stable but still associations, not effects.

Credit not identified

Signs can flip between refits; combine, drop one, or report the sum.

Fine

OLS against the baseline; check residuals.

Predictions still fine

Held-out error stays put; ridge stabilizes coefficients.

Near-duplicate inputs - the sum is identified, the split is not Illustration

Each dot is one bootstrap refit. OLS dots slide along the line b1 + b2 = 2: every point on it fits almost equally well, so the x1 coefficient wanders from -1.1 to 5.6. Ridge (alpha 30) pins both near the middle, shrunk a little.

What the data can identify: the sum. SD of the x1 coefficient 1.30 (OLS) vs 0.05 (ridge); SD of the sum 0.09 vs 0.09 (ridge sum 1.65: the price is a little bias). Held-out MAE 0.76 vs 0.78.

Synthetic data: 160 rows, x2 = x1 plus small noise (correlation 0.997); only x1 drives y (true coefficient 2). Fits use the first 120 rows, MAE the last 40.

80 bootstrap refits of the same model. The data pin down b1 + b2, not how the credit is shared.

How much collinearity inflates a coefficient's variance Formula

\operatorname{Var}(\hat b_j) = \frac{\sigma^2}{n\,\operatorname{Var}(x_j)} \cdot \frac{1}{1 - R_j^2}

$R_j^2$: R squared of regressing input j on all the other inputs
$1/(1 - R_j^2)$: the variance inflation factor (VIF): 1 for an input unrelated to the others
$VIF about 166$: the demo's pair, correlation 0.997: the coefficient's SD is about 13 times what it would be alone

A common rule of thumb flags VIF above 5 or 10 (practitioner heuristic). It warns about reading coefficients, not about predictions.

Ridge, lasso, elastic net: paths and knobsadvanced5 minHow each penalty shrinks, how the strength is chosen, and the library defaults that silently set it for you.

Ridge, lasso and elastic net Formula

\begin{gathered} \min_b \sum_i (y_i - x_i^\top b)^2 \\ + \alpha\left[(1-r)\sum_j b_j^2 + r \sum_j |b_j|\right] \end{gathered}

$r = 0$: ridge: shrinks correlated coefficients together, none exactly zero
$r = 1$: lasso: shrinks and sets some coefficients exactly to zero
$\alpha$: penalty strength; sklearn's LogisticRegression uses C, the inverse (default C = 1.0)

Penalties depend on units, so scale inputs on training rows first (Day 9). Schematic form; libraries scale the terms differently.

Why lasso zeroes a coefficient and ridge only shrinks it Illustration

Ridge (alpha 55): the solution is where an RSS contour first touches the circle. A circle has no corners, so the touch almost never lands on an axis: across the whole computed path the x2 coefficient never gets below 0.250 in size - shrunk, never exactly zero.

Lasso (alpha 79): the diamond's corners sit on the axes, and a corner is where the contour usually touches first. Here the touch is at b2 = 0 exactly: the computed path sets the x2 coefficient to zero at alpha 75 and keeps it there, while b1 = 1.49 survives shrunk (OLS 2.18).

RSS contours (equal training error)
constraint set (circle / diamond)
solution path as the penalty grows

Synthetic data: 80 rows, two standardized inputs with correlation 0.52, y driven mostly by x1 (true coefficients 2.0 and 0.5). Contours, circles, diamonds, paths and solutions are all computed from the same sufficient statistics; nothing is sketched.

Computed RSS contours around one OLS solution. The L2 circle has no corners, so the touch point keeps both coefficients; the L1 diamond's corner sits on an axis, and the touch lands there: b2 becomes exactly zero.

Ridge and lasso paths, penalty chosen by cross-validation Illustration

Ridge: every coefficient shrinks smoothly towards zero and none reaches it. The near copy shares the load with bk14 at strong penalties. CV picks alpha 0.49.

Lasso: coefficients enter one at a time (bk14, weekend, holiday, fare, noise, bk15). At the CV choice bk15 (near copy) and noise are exactly zero. Of two near copies it keeps one; which one is close to a coin toss.

Synthetic data: 150 rows, six standardized inputs, true coefficients 3, 0, 1.5, 0.8, -0.4, 0; bk15 is correlated 0.9 with bk14. Paths and 5-fold CV computed on the page (ridge: squared error + alpha ||b||²; lasso: squared error / 2n + alpha ||b||₁, as in scikit-learn).

Read left to right as the penalty weakens. Ridge never reaches zero; lasso admits inputs one at a time.

Which penalty for which job Comparison

Criterion	Ridge (L2)	Lasso (L1)	Elastic net
Does	Shrinks all coefficients smoothly	Shrinks and zeroes some coefficients	Blends both penalties
Good for	Correlated inputs; stable prediction	Sparse models with many weak columns	Groups of correlated columns, some sparsity
Watch out for	Keeps every column; small bias	Arbitrary choice among near-duplicates	Two knobs to tune on validation rows

Elastic net is the usual answer when correlated groups and sparsity both matter (Zou and Hastie, 2005). Ridge is the MAP estimate under Gaussian priors on the weights; lasso under Laplace priors.

The knobs and their defaults (scikit-learn 1.9.1, checked) Table

Estimator	Objective (as documented)	Defaults that matter	Watch out for
Ridge	squared error summed + alpha x sum of squared weights	alpha=1.0	alpha acts on the sum, so its effect shrinks as n grows
Lasso	squared error / (2n) + alpha x sum of absolute weights	alpha=1.0, max_iter=1000	alpha acts on the mean: the same number is a different strength than Ridge's
ElasticNet	as Lasso, penalty mixed by l1_ratio	alpha=1.0, l1_ratio=0.5	tune both on validation rows; scale inputs first
LogisticRegression	C x summed log loss + penalty	C=1.0, l1_ratio=0.0 (L2), solver lbfgs, max_iter=100	penalized by default; penalty= is deprecated since 1.8 (use l1_ratio)
RidgeCV / LassoCV	pick alpha by CV or efficient LOO	RidgeCV alphas (0.1, 1, 10); LassoCV 100 alphas	the chosen alpha is a tuned value: score it on untouched rows (Day 15)

Read from the installed library's signatures and docstrings. Defaults change between versions; pin and record them.

Fitting a penalized model without leaking the choice of alpha Sequence

Scale on training rows
Inside the pipeline (Day 9). In cents a coefficient is 100 times smaller, so ridge penalizes it 10,000 times less.
Trace the path
Fit over a grid of alpha, strong to weak: ridge shrinks every weight, lasso admits inputs one at a time.
Choose alpha on validation folds
The lowest CV error, or the simplest alpha within one standard error of it (Day 15).
Refit, report once
Refit at that alpha on all training rows and score untouched rows once: the chosen alpha was one more tuning try.

RidgeCV and LassoCV automate steps 2 and 3. Step 4 stays your job.

Memory anchor

A coefficient is conditional, not causal; the loss is not the optimizer; a probability is not a log-odds.

Useful for

A transparent first challenger to the Day 2 baseline, with coefficients a reviewer can read in a minute.
Probability scores for decisions (claim review, fraud flag, churn call) before any more complex classifier.
A reference model for judging complexity - if a tree ensemble cannot beat a well-built regression, the extra complexity is not earning its keep.

Not suitable for

Estimating what would happen under an intervention; the coefficient is conditional on how the data were generated (Day 5).
Strongly nonlinear or interaction-heavy signals such as promotion by weekday by store, or temperature by hour in energy load, without engineered terms (Day 12).

Traps

Reading R squared as proof of form

A high R squared on a trending or curved relationship looks reassuring, while the residuals show a U shape, a funnel or one point carrying the slope. The model's errors are then systematic, not random.

How to avoid it

Always plot residuals against fitted values, by date and by group before quoting a coefficient.

Treating 0.5 and the optimizer as part of the model

Teams compare "L-BFGS versus SGD" as if they were different models and use 0.5 as the decision boundary, so the cost structure of the decision is never encoded.

How to avoid it

Name the objective and the decision rule separately; choose the threshold from costs (Day 13).

Senior reasoning

A senior fits the simplest model against the written baseline first and spends the time on what the output means: which columns existed at the cutoff, which residual pattern is present and which claim it undermines, and whether any coefficient is being asked to carry a causal story it was not designed for. They separate the predictive use (the line is fine) from the explanatory use (needs design), and they say which one the audience is about to act on.

Interview question

"A regression coefficient on discount depth is positive and significant. The head of sales wants to double discounts. What do you say?"

Outline your answer first.

Recall cards scheduled after completion

In logistic regression, what are the objective, the optimizer and the output scales, and what turns a probability into an action?
Match each residual pattern to the assumption it breaks and what it threatens.

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Linear Models (scikit-learn User Guide 1.1) — scikit-learn developerscore
Section 1.1.1 states that least-squares coefficient estimates rely on independent features and that near-dependent columns make them highly variable (multicollinearity); section 1.1.11 gives the regularized log loss and states that regularization is applied by default. (1.1.1 Ordinary Least Squares; 1.1.11 Logistic regression)
An Introduction to Statistical Learning: with Applications in Python — Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, Jonathan Taylor, 2023core
Chapter 3 covers the linear model and its practical problems (non-linearity, non-constant variance, outliers, high-leverage points, collinearity); Chapter 4 covers logistic regression; Chapter 6 covers ridge and lasso. (Chapters 3, 4 and 6)
To Explain or to Predict? — Galit Shmueli, 2010optional
Explaining and predicting need different variable choices, validation and evaluation, which is why a well-predicting coefficient is not an explanation.
Ridge Regression: Biased Estimation for Nonorthogonal Problems — Arthur E. Hoerl, Robert W. Kennard, 1970optional
Introduces the ridge penalty as a biased estimator that reduces variance when inputs are correlated.
Regression Shrinkage and Selection Via the Lasso — Robert Tibshirani, 1996optional
Introduces the lasso, an L1 penalty that shrinks coefficients and sets some exactly to zero.
Regularization and Variable Selection Via the Elastic Net — Hui Zou, Trevor Hastie, 2005optional
Introduces the elastic net, which combines the L1 and L2 penalties and tends to keep or drop groups of correlated inputs together.

Optional sections

Deep dive · 10 min

Closed form and gradient. Least squares solves $\hat b = (X^\top X)^{-1}X^\top y$ , in practice with a QR or SVD decomposition rather than an explicit inverse. For logistic regression the mean log loss $\ell(b) = \frac{1}{n}\sum_i [\log(1+e^{z_i}) - y_i z_i]$ has gradient $X^\top(p - y)/n$ and is convex, so any descent method finds the same global minimum; the lab shows gradient descent, BFGS and L-BFGS agreeing to four decimals. Convexity of the loss in the weights is a property of this model, not of neural networks (Day 33).

Where the losses come from. Squared error is the Gaussian log-likelihood up to constants; log loss is the Bernoulli log-likelihood. Choosing a loss is choosing which kind of error you punish, and it determines what the fitted value estimates: squared error targets the conditional mean, absolute error the median (Day 16).

Why the confounded coefficient was +0.91. In the simulator, one standard deviation of expected demand moves final bookings by about 25.5 seats (22 through D-14 bookings, 6 through pickup, minus 2.5 through the fare it triggers) and moves the fare by 25 EUR. The slope of final bookings on fare alone is then about $(25.5 \times 25 - 0.10 \times 64)/(25^2 + 8^2) \approx 0.92$ , the omitted-variable formula: the confounder's effect times how strongly it moves with the fare, divided by the fare's variance. Including bk14 removes most of the demand information and the coefficient collapses to nearly zero, still not the -0.10 truth.

Standard errors. With non-constant spread, the classical standard error assumes one error variance. A heteroscedasticity-consistent (HC) estimator, shown as "robust" in the widget, fixes the standard error without changing the line. Clustered standard errors do the same for repeated rows per flight or customer.

Penalties and priors. Ridge is the maximum a posteriori estimate under independent zero-mean Gaussian priors on the weights; lasso corresponds to Laplace priors. The scikit-learn parameter C is the inverse of the penalty strength: a very large C approximates no regularization.

Hands-on lab: Same objective, three optimizers; correlated inputs, three penalties · 20 min

Run the script. Part A minimizes one log loss with gradient descent, BFGS and scikit-learn's L-BFGS. Part B adds a near-duplicate snapshot (bk15) and refits OLS, ridge and lasso on 100 bootstrap resamples. Then:

Explain why the three optimizers print the same log loss and coefficients, and what would change the model.
OLS has the best held-out MAE yet its coefficients on bk14 and bk15 have a standard deviation of 5.2. Which question would you answer with OLS and which with ridge?
Lasso sets some coefficients to exactly zero. Why is that not evidence about which snapshot is the true driver?

import numpy as np
from scipy.optimize import minimize
from sklearn.linear_model import Lasso, LinearRegression, LogisticRegression, Ridge
from sklearn.metrics import mean_absolute_error
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(11)
n = 600                                          # synthetic departures, date order
demand = rng.normal(size=n)
bk14 = 95 + 22 * demand + rng.normal(0, 4, n)    # bookings at D-14
bk15 = bk14 + rng.normal(0, 2, n)                # a near-duplicate snapshot, correlation about 0.99
weekend = rng.integers(0, 7, n) >= 4
final = bk14 + 28 + 6 * demand + 4 * weekend + rng.normal(0, 1, n) * (1 + 0.08 * bk14)
cut = 450
X = StandardScaler().fit(np.c_[bk14, bk15, weekend][:cut]).transform(np.c_[bk14, bk15, weekend])
tr, te = slice(0, cut), slice(cut, n)

# A) One objective (log loss), three optimizers: the solutions agree, so the solver is not the model
y = final >= 150
Xa, ya = X[tr][:, [0, 2]], y[tr]
def log_loss_w(w):
    z = w[0] + Xa @ w[1:]
    return np.mean(np.logaddexp(0, z) - ya * z)       # mean negative log-likelihood
w_gd = np.zeros(3)
for _ in range(20000):                                 # plain gradient descent, fixed step
    p = 1 / (1 + np.exp(-(w_gd[0] + Xa @ w_gd[1:])))
    w_gd -= 0.5 * np.r_[np.mean(p - ya), Xa.T @ (p - ya) / len(ya)]
w_bfgs = minimize(log_loss_w, np.zeros(3), method="BFGS").x
skl = LogisticRegression(C=1e6, max_iter=5000).fit(Xa, ya)
for name, w in [("gradient descent", w_gd), ("BFGS", w_bfgs), ("sklearn lbfgs", np.r_[skl.intercept_, skl.coef_[0]])]:
    print(f"A) {name:16s} log loss {log_loss_w(w):.4f} coefficients {np.round(w, 2)}")

# B) Collinearity: predictions stay fine, coefficients on bk14/bk15 become unstable across resamples
def boot(model):
    coefs = []
    for _ in range(100):
        i = rng.integers(0, cut, cut)
        coefs.append(model.fit(X[tr][i], final[tr][i]).coef_[:2])
    return np.array(coefs)
for name, model in [("OLS", LinearRegression()), ("Ridge(alpha=50)", Ridge(alpha=50)), ("Lasso(alpha=3)", Lasso(alpha=3.0))]:
    c = boot(model)
    mae = mean_absolute_error(final[te], model.fit(X[tr], final[tr]).predict(X[te]))
    zero = np.mean(c == 0)
    print(f"B) {name:16s} held-out MAE {mae:.2f} | bk14 coef mean {c[:, 0].mean():5.1f} SD {c[:, 0].std():4.1f} | bk15 SD {c[:, 1].std():4.1f} | exact zeros {zero:.0%}")
print("B) corr(bk14, bk15) =", round(np.corrcoef(bk14, bk15)[0, 1], 3))

Interview practice · 10 min

Your linear model has a respectable R squared and the residual plot against fitted values shows a clear funnel. What does that change, and what does it leave alone?

Answer as you would in the interview.

Content version 7 · last verified 2026-10-11 · depends on scikit-learn, numpy, scipy

1 Recall · 5 min

2 Concept · 10 min