Neural networks, deep learning, NLP · insight-0408
Leave regression outputs linear and choose the loss for the statistic you need
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- energy, insurance
Summary: situation, action, reason, boundary
Situation
A network predicts a non-negative quantity such as daily energy load or claim cost, and someone adds a ReLU on the output.
Action
Keep a linear output unit. For positive skewed targets, model the log or use a softplus output; choose MSE for the mean, absolute error for the median or a pinball loss for a quantile, matching the Day 16 metric.
Reason
With a linear output, minimizing squared error is maximum likelihood under a Gaussian, so the loss defines which summary of the conditional distribution you estimate.
Boundary
Regression heads in tabular, forecasting and vision models.
Symptom or risk
A ReLU output can get stuck at zero for many inputs and stop learning; MSE on a skewed target chases a few huge values.
Exceptions
Bounded targets such as load factor in [0, 1] can use a sigmoid output when values near the bounds matter.
Trade-offs
Log targets need back-transformation and give median-type predictions; correct for bias if the mean is required.
Pitfalls
Reporting MAPE while training on MSE and wondering why the metric does not improve.
Lessons and concepts
Claims and their sources
Goodfellow, Bengio and Courville (ch. 6.2.2) show that a linear output unit trained by maximum likelihood with a Gaussian output corresponds to minimizing mean squared error.