Classical ML, validation, metrics, interpretation, debugging · insight-1158
Expect ensemble variance reduction to stall at the correlation between members
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A team keeps adding trees, or averaging near-identical models, to reduce variance and sees the gain flatten.
Action
Measure the average pairwise correlation of member predictions and reduce it, for example with feature subsampling or different learners and feature views, rather than adding more copies.
Reason
For B identically distributed members with variance sigma squared and pairwise correlation rho, the average has variance rho sigma squared + (1 - rho) sigma squared / B, so rho sets a floor.
Boundary
Bagging, random forests and model averaging of similar learners.
Symptom or risk
Compute grows while error does not fall, because the averaged members make the same mistakes.
Exceptions
Boosting reduces bias sequentially, so this variance formula is not its main mechanism.
Trade-offs
De-correlating members, for instance with smaller max_features, can raise the bias of each one.
Pitfalls
Averaging many models trained on the same features with different seeds and expecting independent errors.
Example
With rho 0.5 and sigma squared 1, 100 members give variance 0.505 against 0.01 under independence; adding members barely moves it from 0.5.
Lessons and concepts
Claims and their sources
Hastie, Tibshirani and Friedman give the variance of the average of B identically distributed trees with pairwise correlation rho as rho sigma^2 + ((1 - rho)/B) sigma^2 (Elements of Statistical Learning, chapter 15, eq. 15.1); random feature subsets reduce the correlation.
scikit-learn describes random forests as achieving reduced variance by combining diverse trees, sometimes at the cost of a slight increase in bias.