Skip to content
ML Academy

Introduced on day 11

Regularization

A penalty on model complexity added to the training objective: ridge shrinks coefficients, lasso can set them to zero, elastic net blends both.

In detail

Ridge (L2) adds the sum of squared coefficients and shrinks correlated inputs together; lasso (L1) adds the sum of absolute values and produces sparse solutions; elastic net mixes them. Penalties depend on feature units, so inputs are scaled first. The penalty strength is tuned on validation data, inside the folds. Regularization trades a little bias for lower variance; it does not turn associations into causal effects.