Classical ML, validation, metrics, interpretation, debugging · insight-1142draft — in review
Compute ranking metrics within each group when the decision is to rank inside that group
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- aviation, e-commerce, banking
Summary: situation, action, reason, boundary
Situation
A model ranks flights, accounts or claims, but the team reports one AUC over all pooled rows while analysts only ever compare items within one day, route or queue.
Action
Compute AUC, precision at K or NDCG per group, average with group-size weights, and compare against a baseline that knows only the group-level rate.
Reason
The decision compares items inside a group, so only within-group order matters; pooled metrics also reward between-group separation.
Boundary
Daily work queues, per-route prioritization, search and recommendation lists, per-customer next-best action.
Symptom or risk
Group-level base rates dominate pooled AUC, so a model that only learns which days are busy scores well while ordering items within a day no better than chance.
Exceptions
If one global ranked list is worked top to bottom across groups, pooled metrics match the decision.
Trade-offs
Small groups give noisy per-group scores; use weights and report the spread.
Pitfalls
Dropping groups with a single class, which silently changes the population being evaluated.
Example
Hypothetical: a model that only encodes weekday can reach a pooled AUC near 0.65 while its within-day AUC is 0.50.