Classical ML, validation, metrics, interpretation, debugging · insight-1193
Validate that ensemble disagreement predicts error before routing cases to review on it
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- banking, healthcare, aviation
Summary: situation, action, reason, boundary
Situation
The disagreement among seeds or bootstrap members is proposed as a confidence signal for abstaining or sending cases to manual review.
Action
On held-out data, compute the rank correlation between disagreement and absolute error, chart error by disagreement decile, and compare errors caught at the review budget with a simple baseline such as the model's own score margin. Adopt it only when the lift is clear, and recheck after drift.
Reason
Disagreement measures variance across members, which is one part of error; bias shared by all members is invisible to it.
Boundary
Human-in-the-loop routing for scored cases with a fixed review capacity.
Symptom or risk
Disagreement is high where the model is merely sensitive and low where every member shares the same bias, for example in a regime absent from training; reviewing high-disagreement cases may find no more errors than random sampling.
Exceptions
When disagreement is used only as a diagnostic, not as a gate.
Trade-offs
Training several members multiplies cost and latency.
Pitfalls
Evaluating on rows that were also used to select the routing threshold.