Classical ML, validation, metrics, interpretation, debugging · insight-0176
Compare candidate and incumbent per slice before shipping an aggregate improvement
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- banking, e-commerce
Summary: situation, action, reason, boundary
Situation
A new model beats the production model on the overall metric and a release is planned.
Action
Evaluate both models on the same holdout by key slices, set non-regression guardrails for critical slices, and block or segment the release if a guardrail fails.
Reason
Aggregate metrics are weighted by slice size, so trade-offs between slices are invisible in the total.
Boundary
Model upgrades, retrains and vendor switches in customer-facing or regulated settings.
Symptom or risk
The gain comes from the largest slices while an important minority slice gets worse; complaints follow from exactly that group.
Exceptions
Tiny slices with wide intervals need a looser guardrail or more data before blocking a release.
Trade-offs
Guardrails slow releases and need maintenance; skipping them moves risk to production.
Pitfalls
Choosing slices after seeing results, which finds spurious regressions or excuses real ones.
Lessons and concepts
Claims and their sources
Overall improvements can coexist with worse performance on subgroups, and slice-based evaluation is recommended before deployment (DMLS chapter 6).