Classical ML, validation, metrics, interpretation, debugging · insight-1136
Do not score pooled out-of-fold predictions as if they were one test set
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You call cross_val_predict, compute one AUC or RMSE over all pooled predictions and report it as the cross-validated score.
Action
Report the mean and spread of per-fold scores from cross_validate or cross_val_score, and keep pooled out-of-fold predictions for plots, error analysis and blending.
Reason
scikit-learn documents that cross_val_predict returns predictions from several distinct models undistinguished and is not an appropriate measure of generalization error.
Boundary
Any cross-validated report, especially AUC, PR-AUC, log loss and calibration summaries.
Symptom or risk
The pooled vector mixes predictions from several different fold models, so the number has no fold spread and, for ranking and calibration metrics, can differ from the average of fold scores.
Exceptions
For a metric that is a plain sum over rows, such as total cost, equal-sized folds make the pooled and fold-averaged values nearly the same.
Trade-offs
Fold scores on small folds are noisy; repeated cross-validation narrows that without pooling.
Pitfalls
Picking a threshold on pooled out-of-fold scores and then quoting the same scores' performance at that threshold.
Example
Hypothetical: five fold models score probabilities on different scales, so a pooled AUC can fall below the weakest fold's AUC.
Lessons and concepts
Claims and their sources
cross_val_predict returns, for each element, the prediction from the model for which it was in the test set; the predictions come from several distinct models undistinguished, so it is not an appropriate measure of generalization error.