Classical ML, validation, metrics, interpretation, debugging · insight-1192
Do not present the spread of tree predictions in a forest as a prediction interval
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, logistics, energy
Summary: situation, action, reason, boundary
Situation
A random forest or bagged model is asked for uncertainty, and the standard deviation or 5th to 95th percentile of its tree predictions is shown as an interval.
Action
Measure empirical coverage and width on held-out data by slice. If coverage falls short, calibrate with split conformal prediction (quantiles of held-out residuals) or fit quantile models, and report the coverage achieved.
Reason
Tree disagreement captures model variance only; outcome noise sits on top of it, and conformal calibration targets coverage directly.
Boundary
Demand, duration and cost predictions that carry a range to a decision maker.
Symptom or risk
The spread among trees reflects how much the fitted model would change with resampled data, not how far outcomes fall from the prediction; on noisy targets the band is far too narrow and misses its stated coverage.
Exceptions
Using the spread as a relative signal of sensitivity, not as an interval, is legitimate.
Trade-offs
Conformal intervals have constant-looking width unless made adaptive.
Pitfalls
Calibrating on the same rows used to train the forest.
Example
Hypothetical: a nominal 90% band built from tree percentiles covers 55% of held-out flights.
Lessons and concepts
Claims and their sources
Conformal prediction wraps a pre-trained model to produce prediction sets intended to contain the outcome with a user-specified probability, valid in a distribution-free, finite-sample sense.