Classical ML, validation, metrics, interpretation, debugging · insight-0175
Rank error slices by total cost, not by error rate alone
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
Action
For each slice, compute volume, error, and volume times error (or business cost), sort by the product, and add a confidence interval so small slices do not dominate by noise.
Reason
Impact equals how often a slice occurs times how bad it is; error rate alone ignores frequency.
Boundary
Classification and regression error analysis by segment, time, channel or input type.
Symptom or risk
Attention goes to tiny slices with dramatic error rates while a large slice with moderate errors drives most of the cost.
Exceptions
Slices with safety, legal or fairness implications may deserve priority regardless of volume.
Trade-offs
Predefined slices are explainable; automated slice discovery finds surprises but raises multiple-comparison risk.
Pitfalls
Defining slices with the model's own predictions, which creates circular patterns.
Example
Synthetic airline data: intl-long flights are about 10% of flight-days but about 47% of revenue, so the same percentage error ranks slices very differently when weighted by flights or by revenue.
Lessons and concepts
Claims and their sources
Slice-based evaluation reveals performance differences that aggregate metrics hide (DMLS chapter 6).