Classical ML, validation, metrics, interpretation, debugging · insight-1140
Treat the out-of-bag score as optimistic when rows are grouped or time-ordered
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A random forest reports oob_score and you plan to skip validation because the out-of-bag estimate is free.
Action
Use the out-of-bag score only for quick comparisons on independent rows, and report grouped or time-ordered validation for entity or temporal data.
Reason
Out-of-bag samples are left-out rows of a row-level bootstrap, so they estimate error on new rows of seen entities, not on new entities or later periods.
Boundary
Random forests and bagged trees on panel, repeated-measure or time-stamped tabular data.
Symptom or risk
Bootstrap samples draw rows independently, so out-of-bag rows have near twins from the same customer, flight or week inside the bag; the estimate ignores the group and time boundary.
Exceptions
With independent identically distributed rows, the out-of-bag error is a reasonable and cheap estimate.
Trade-offs
Out-of-bag is nearly free; grouped cross-validation costs several fits.
Pitfalls
Tuning forest hyperparameters on out-of-bag error and quoting the same number as final performance.
Lessons and concepts
Claims and their sources
In random forests the default is bootstrap sampling, and generalization error can be estimated on the left-out out-of-bag samples by setting oob_score=True.