Classical ML, validation, metrics, interpretation, debugging · insight-0141

Do not compare R² across test sets with different target variance

Situation
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
Action
Compare segments and periods with absolute error metrics in business units (MAE, RMSE, WAPE), and use R² only within one fixed evaluation set.
Reason
R² is one minus error variance relative to target variance, so the same errors give a lower R² on a narrower population.
Boundary
Regression reports by segment, time period, or dataset; vendor model comparisons.

Symptom or risk

Stakeholders conclude the model is worse on the route, although absolute errors may be identical; the route simply has less variance to explain.

Within a single held-out set, R² is a fine relative summary of two models' squared errors.

Absolute metrics need a scale context for executives; pair them with a baseline error.

Pooling heterogeneous segments to inflate R², since between-segment variance is easy to explain.

Synthetic airline data: predicting revenue across domestic and intl-long flights earns a high R² mostly from separating the markets.

Because variance is dataset dependent, R² may not be meaningfully comparable across different datasets.
Metrics and scoring: quantifying the quality of predictions (scikit-learn User Guide 3.4)