Classical ML, validation, metrics, interpretation, debugging · insight-0166

Plot a learning curve before asking for more data

Situation
Stakeholders ask whether another year of history or more labelling budget will improve the model.
Action
Plot training and validation scores against training size with the final pipeline; if the gap is open and validation still rising, more data should help; if both have converged low, improve features or model class.
Reason
The shape of the two curves separates variance problems, which data reduces, from bias problems, which it does not.
Boundary
Labelling budget decisions, history-length choices and model-capacity choices.

Symptom or risk

Money is spent on data when the model is bias-limited, or on model complexity when it is variance-limited.

For drifting data, older history may hurt; extrapolate learning curves only within a stable period.

Learning curves need repeated fits; subsample sizes on a log scale to keep cost manageable.

Building the curve with a random split on temporal data, which hides drift effects.

A learning curve shows whether more data helps: an open train-validation gap suggests yes, while curves converging to a similarly low score suggest little benefit.
Validation curves and learning curves (scikit-learn User Guide 3.5)