Classical ML, validation, metrics, interpretation, debugging · insight-1164
Judge a slice's error rate against sample-size control limits before calling it the problem
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A slice table sorted by error rate puts a 20-row segment at the top with 25% errors while the model averages 10%.
Action
Plot slice error against slice size with binomial control limits around the overall rate (a funnel plot), and investigate slices outside the funnel or large ones near it.
Reason
The spread of an observed rate shrinks with the square root of the slice size, so the same rate means different things at different sizes.
Boundary
Error analysis by segment, route, device or cohort with many slices of varied size.
Symptom or risk
Small slices have extreme rates by chance, so the worst-looking slice is often just the smallest; effort is spent on noise.
Exceptions
If a tiny slice carries a regulatory or safety requirement, follow it regardless of significance.
Trade-offs
Many slices means many limits; expect a few outside by chance and confirm on a held-out period.
Pitfalls
Defining slices after looking at the errors and then testing on the same rows.
Example
Overall error 10%: 95% limits are about plus or minus 13 points at 20 rows (5 of 20 is not clearly outside; exact binomial tails are asymmetric), 4 points at 200 and 1.3 at 2,000.
Lessons and concepts
Claims and their sources
Funnel plots present unit-level rates against unit size with control limits instead of league tables.