Classical ML, validation, metrics, interpretation, debugging · insight-1159

Train a stacking meta-model on out-of-fold predictions, never on in-sample ones

Situation
A second-level model combines a boosted model, a forest and a linear model, and was fitted on their predictions for the training rows.
Action
Build the meta features from cross-validated predictions (StackingClassifier does this internally), refit base models on all training data for inference, and test the stack on untouched data.
Reason
The meta-model must see base predictions that look like those on new data, which are out-of-sample.
Boundary
Stacked and blended ensembles for tabular prediction.

Symptom or risk

In-sample predictions of flexible learners are overconfident, so the meta-model learns to trust the most overfit base model.

Simple averaging with fixed equal weights has no meta-model to leak.

Several refits per base learner; the gain over a tuned single model is often small.

Doing a second-level cross-validation with leaked out-of-fold predictions from folds that include the new validation rows.

In scikit-learn stacking the estimators are fitted on the whole training data, and to generalize and avoid over-fitting the final_estimator is trained on out-of-sample predictions obtained internally with cross_val_predict.
Ensembles: Gradient boosting, random forests, bagging, voting, stacking (scikit-learn User Guide 1.11)