Concept map: Train / validation / test

Prerequisites (two levels) and direct dependents of Train / validation / test. Show everything

2prerequisites shown
4direct dependents
4lessons use it

Taught and used in

day 15 day 30 day 43 day 56

Validation and metrics

Insights 6

Use sigmoid calibration on small calibration sets and isotonic only with plenty of data
Count every look at the test set as a selection step and refresh the holdout when looks pile up
Plot a learning curve before asking for more data
Give early stopping its own validation split and never point it at the test set
Split the eval set into a development set for iteration and a locked test set for decisions
Use many rolling origins that span the seasons you care about, not one final holdout

Projects 14

List view151 concepts

Day 1
- Baseline-first culture · needs When rules beat ML
- Business vs analytical problems
- Cost of being wrong · needs Business vs analytical problems
- Prediction · needs Business vs analytical problems
- When rules beat ML · needs Business vs analytical problems
Day 2
- Data contracts · needs Unit of analysis and grain
- Horizon · needs Target definition
- Leakage · needs Point-in-time correctness
- Point-in-time correctness · needs Horizon
- Target definition · needs Unit of analysis and grain
- Unit of analysis and grain · needs Business vs analytical problems
Day 3
- Common distributions · needs Random variables and expectation
- Conditional probability and Bayes · needs Random variables and expectation
- Random variables and expectation
- Sampling distributions · needs Variance and covariance, Common distributions
- Variance and covariance · needs Random variables and expectation
Day 4
- Bayesian and frequentist thinking · needs Confidence intervals, Conditional probability and Bayes
- Confidence intervals · needs Sampling distributions
- Effect size and power · needs Hypothesis tests and p-values
- Hypothesis tests and p-values · needs Sampling distributions
- Multiple comparisons · needs Hypothesis tests and p-values
Day 5
- Confounding · needs Prediction, Variance and covariance
- DAGs · needs Confounding
- Forecasting vs causal what-if · needs Confounding
- Quasi-experimental designs · needs DAGs
- Randomized experiments · needs Hypothesis tests and p-values, Confounding
- Selection and survivorship bias · needs Confounding
- Simpson's paradox · needs Confounding
Day 6
- Duplicates and integrity rules · needs Unit of analysis and grain
- Hypothesis-driven EDA · needs Unit of analysis and grain, Common distributions
- Missing data mechanisms · needs Hypothesis-driven EDA
- Outliers · needs Hypothesis-driven EDA
- Temporal gaps and irregular timestamps · needs Hypothesis-driven EDA
Day 7
- Event vs snapshot data and event vs processing time · needs Unit of analysis and grain, Point-in-time correctness
- Joins and join explosions · needs Unit of analysis and grain
- Slowly changing dimensions and backfills · needs Event vs snapshot data and event vs processing time
- Window functions · needs Joins and join explosions
Day 8
- Chart choice · needs Hypothesis-driven EDA
- Driver and contribution analysis · needs Chart choice
- Insight-led titles and storytelling · needs Chart choice
- Misleading charts and interpretation limits · needs Chart choice, Selection and survivorship bias
Day 9
- Categorical encoding · needs Holdout boundary before fitting
- Fold-safe target encoding · needs Categorical encoding
- Holdout boundary before fitting · needs Leakage
- Preprocessing pipelines · needs Scaling and when it is unnecessary, Categorical encoding
- Scaling and when it is unnecessary · needs Holdout boundary before fitting
Day 11
- Linear regression · needs Variance and covariance, Baseline-first culture
- Logistic regression and log loss · needs Linear regression, Conditional probability and Bayes
- Multicollinearity · needs Linear regression
- Regression assumptions and residuals · needs Linear regression
- Regularization · needs Linear regression, Scaling and when it is unnecessary
Day 12
- Bagging and random forests · needs Decision trees
- Decision trees · needs Baseline-first culture
- Gradient boosting · needs Decision trees
- XGBoost · needs Gradient boosting
Day 13
- Class imbalance · needs Logistic regression and log loss
- Probability calibration · needs Logistic regression and log loss
- Resampling · needs Class imbalance, Holdout boundary before fitting
- Thresholds from cost matrices · needs Class imbalance, Cost of being wrong
Day 14
- Anomaly detection · needs Outliers, Clustering
- Cluster validation and actionability · needs Clustering
- Clustering · needs Scaling and when it is unnecessary, Variance and covariance
- PCA and dimensionality reduction · needs Variance and covariance, Scaling and when it is unnecessary
Day 15
- Cross-validation schemes · needs Train / validation / test
- Leakage taxonomy · needs Leakage, Cross-validation schemes
- Learning curves and early stopping · needs Train / validation / test
- Nested CV · needs Cross-validation schemes, Multiple comparisons
- Train / validation / test · needs Holdout boundary before fitting
Day 16
- Business vs ML metric · needs Regression metrics, Cost of being wrong
- Classification metrics · needs Thresholds from cost matrices
- Forecast metrics · needs Regression metrics
- Regression metrics · needs Random variables and expectation
Day 17
- Feature importance · needs Bagging and random forests, Multicollinearity
- Feature selection inside CV · needs Feature importance, Cross-validation schemes
- PDP · needs Feature importance
- SHAP · needs Feature importance
Day 18
- Error slices and residual patterns · needs Regression assumptions and residuals, Classification metrics
- Root-cause analysis and failure taxonomies · needs Error slices and residual patterns
Day 19
- Batch vs online serving and deployment · needs Reproducibility and versioning
- Drift and monitoring · needs Batch vs online serving and deployment, Common distributions
- Experiment tracking and model registry · needs Reproducibility and versioning
- Reproducibility and versioning · needs Preprocessing pipelines
- Training-serving skew · needs Batch vs online serving and deployment, Preprocessing pipelines
Day 21
- Forecasting workflow · needs Trend, Target definition
- Stationarity · needs Trend, Variance and covariance
- Structural breaks and regimes · needs Trend
- Time series data types · needs Trend
- Trend · needs Hypothesis-driven EDA, Temporal gaps and irregular timestamps
Day 22
- Autocorrelation · needs Variance and covariance, Stationarity
- Differencing · needs Stationarity
- STL and decomposition · needs Trend
- White noise and Ljung-Box · needs Autocorrelation, Hypothesis tests and p-values
Day 23
- ARIMA and SARIMA · needs Autocorrelation, Differencing
- Exponential smoothing and ETS · needs Forecast baselines
- Forecast baselines · needs Forecasting workflow, Baseline-first culture
Day 24
- Direct · needs Point-in-time lag and rolling features
- Exogenous variables and availability · needs Point-in-time lag and rolling features
- Global ML forecasting models · needs Gradient boosting, Point-in-time lag and rolling features
- Point-in-time lag and rolling features · needs Window functions, Autocorrelation, Point-in-time correctness
Day 25
- Per-horizon error and stability · needs Rolling-origin backtesting
- Rolling-origin backtesting · needs Cross-validation schemes, Forecast baselines
Day 26
- Conformal prediction · needs Coverage, Train / validation / test
- Coverage · needs Prediction intervals and quantiles
- Forecast combinations · needs Forecast baselines
- Forecast decision economics · needs Prediction intervals and quantiles, Cost of being wrong
- Hierarchical forecasting and reconciliation · needs Time series data types, Forecast baselines
- Intermittent demand · needs Time series data types, Forecast metrics
- Prediction intervals and quantiles · needs Confidence intervals, Forecast baselines
Day 27
- Deep and foundation forecasting models · needs Global ML forecasting models, Rolling-origin backtesting
Day 28
- Capacity-censored demand · needs Target definition, Forecasting workflow
Day 29
- Clarifying ambiguous cases · needs Business vs analytical problems
Day 31
- Neurons · needs Linear regression
- Tensors · needs Neurons
Day 32
- Activation functions · needs Neurons
- Losses and output layers · needs Activation functions, Logistic regression and log loss
Day 33
- Backpropagation · needs Losses and output layers, Tensors
- Optimizers and learning rates · needs Backpropagation
Day 34
- Training dynamics and regularization · needs Optimizers and learning rates, Learning curves and early stopping
Day 35
- Neural vs classical baseline comparison · needs Training dynamics and regularization, Baseline-first culture
Day 36
- Convolutional networks · needs Tensors
Day 37
- Recurrent networks · needs Backpropagation
Day 38
- Embeddings and representation learning · needs Neurons, Variance and covariance
Day 39
- Attention and Transformers · needs Embeddings and representation learning, Activation functions
Day 40
- Transfer learning and fine-tuning decisions · needs Training dynamics and regularization
Day 41
- Tokenization · needs Categorical encoding
Day 42
- Decoding and sampling · needs Next-token prediction and context windows
- Hallucination and unsupported output · needs Next-token prediction and context windows
- Next-token prediction and context windows · needs Attention and Transformers, Tokenization
Day 43
- Prompting vs retrieval vs fine-tuning · needs Next-token prediction and context windows
- Versioned evaluation set before adaptation · needs Train / validation / test, Prompting vs retrieval vs fine-tuning
Day 44
- PEFT and LoRA · needs Transfer learning and fine-tuning decisions, Prompting vs retrieval vs fine-tuning
Day 45
- LLM evaluation · needs Versioned evaluation set before adaptation
- RLHF and DPO fundamentals · needs LLM evaluation
Day 46
- Generative model families · needs Embeddings and representation learning
Day 47
- Structured outputs and validation · needs Next-token prediction and context windows
- Untrusted input boundaries · needs Structured outputs and validation
Day 48
- Lexical · needs RAG pipeline
- RAG pipeline · needs Embeddings and representation learning, Tokenization, Untrusted input boundaries
Day 49
- Agents and tool use · needs Structured outputs and validation, Untrusted input boundaries
- Agents vs deterministic workflows · needs Agents and tool use
Day 51
- Faithfulness and groundedness · needs Retrieval metrics, Hallucination and unsupported output
- Retrieval metrics · needs RAG pipeline
Day 52
- LLM request lifecycle and observability · needs Batch vs online serving and deployment, Next-token prediction and context windows
Day 53
- Privacy · needs Data contracts
- Prompt injection and AI security · needs Untrusted input boundaries, Agents and tool use
Day 54
- AI cost · needs LLM request lifecycle and observability
Day 55
- Production GenAI design · needs RAG pipeline, Prompt injection and AI security, AI cost
Day 58
- Decision and assumption logs · needs Batch vs online serving and deployment
- ML system design · needs Drift and monitoring, Decision and assumption logs
Day 59
- Interview defense and revision · needs Clarifying ambiguous cases
Day 60
- Portfolio evidence · needs Interview defense and revision

Taught and used in

Insights 6

Projects 14

List view151 concepts

Day 1

Day 2

Day 3

Day 4

Day 5

Day 6

Day 7

Day 8

Day 9

Day 11

Day 12

Day 13

Day 14

Day 15

Day 16

Day 17

Day 18

Day 19

Day 21

Day 22

Day 23

Day 24

Day 25

Day 26

Day 27

Day 28

Day 29

Day 31

Day 32

Day 33

Day 34

Day 35

Day 36

Day 37

Day 38

Day 39

Day 40

Day 41

Day 42

Day 43

Day 44

Day 45

Day 46

Day 47

Day 48

Day 49

Day 51

Day 52

Day 53

Day 54

Day 55

Day 58

Day 59

Day 60