Classical ML, validation, metrics, interpretation, debugging · insight-0136

Score a dummy baseline on the same split and metric before reading any model result

Situation
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
Action
Fit DummyClassifier (most_frequent, prior) or DummyRegressor (mean, median) in the same pipeline, split and metric, and report the model's lift over it.
Reason
A score is only meaningful relative to what an uninformed rule achieves on the same data; near-dummy scores point to weak features, bad tuning or imbalance.
Boundary
Every supervised task, including each segment you report separately.

Symptom or risk

Without a reference point, a model that barely beats predicting the majority class or the mean looks impressive.

For time series, use naive or seasonal-naive forecasts instead of a global mean.

A dummy is the floor, not the bar; the bar is usually the current business rule.

Computing the dummy on a different split or metric than the model, which makes the comparison meaningless.

Synthetic no-show data at 8% prevalence: predicting nobody gives 92% accuracy, so a 93% model has learned little.

Dummy estimators ignore inputs and serve as sanity baselines; a model scoring close to them suggests uninformative features, poor hyperparameters or class imbalance.
Metrics and scoring: quantifying the quality of predictions (scikit-learn User Guide 3.4)