Introduced on day 15
Train / validation / test split
Three roles for data - training rows fit parameters, validation rows choose between candidates, and a test block, used once, reports the result.
In detail
Every use of a block to make a choice (a feature, a hyperparameter, a stopping round) makes its score less independent. Cross-validation folds can replace a single validation block, but the test block stays outside all choices. For time-ordered data the blocks are ordered in time, and for repeated entities they hold out whole entities, so the split mimics what is new at deployment.