Classical ML, validation, metrics, interpretation, debugging · insight-0145

Pick MAE or RMSE from whether the decision needs the median or the mean

Situation
You are choosing a training loss or headline metric for a regression model.
Action
Use squared error when totals or expected values drive the decision (budgets, revenue sums) and absolute error when typical-case accuracy matters; report both if stakeholders differ.
Reason
The expected squared error is minimized by the conditional mean and the expected absolute error by the conditional median.
Boundary
Regression with skewed targets such as revenue, claim size, delivery time or demand.

Symptom or risk

The model optimizes a different quantity than the decision uses; for skewed targets the median and mean can differ a lot.

With asymmetric costs neither is right; use a quantile (pinball) loss at the cost-optimal quantile.

RMSE is sensitive to a few large errors, which is desirable when large misses are costly and harmful when they are data errors.

Training on squared error and celebrating MAE, or vice versa, without noticing the mismatch.

Under squared-error loss the best predictor is the conditional mean; under absolute-error loss it is the conditional median (ESL section 2.4).
The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd edition)