Neural networks, deep learning, NLP · insight-1415
Prefer a small deep ensemble over MC dropout when uncertainty must hold up
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A neural classifier's confidence will gate automated decisions, and you can afford several times the training and inference compute.
Action
Train five networks from different random initialisations (same data, different seeds), average predicted probabilities, and use their disagreement as an extra uncertainty feature. Check both on shifted data.
Reason
Deep ensembles are simple, parallel and needed little tuning in the original proposal. In the Ovadia et al. shift benchmark (full text read) ensembles performed best across most metrics, with MC dropout the strong runner-up and ahead of ensembles on Brier score at high shift in one ad-click experiment.
Boundary
Moderate-size networks where five training runs are affordable.
Symptom or risk
A single network is overconfident on unfamiliar inputs, and dropout-sampled spread can look reassuring without being validated.
Exceptions
Large models for which five copies are infeasible; use cheaper averaging such as checkpoint or weight averaging.
Trade-offs
Five times the training and serving cost, and a harder deployment story.
Pitfalls
Members that share a seed for data order or initialisation are not independent.
Lessons and concepts
Claims and their sources
Lakshminarayanan et al. propose ensembles of networks as an easy, parallel, low-tuning way to get well-calibrated uncertainty, with higher uncertainty on out-of-distribution examples.
Ovadia et al. found, under dataset shift, that post-hoc recalibration often falls short while methods that average over several models performed surprisingly well.