Classical ML, validation, metrics, interpretation, debugging · insight-1186
Evaluate a ranker per list and average, instead of pooling all items into one AUC
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, aviation, retail
Summary: situation, action, reason, boundary
Situation
A model orders items inside lists (offers for a user, fares for a search, cases for an analyst) and is scored with one AUC or precision across all pooled items.
Action
Use per-list metrics such as NDCG at k or precision at k, averaged across lists with list weights you can justify. Report how many lists have no relevant item, and compare with a popularity or recency order inside the list.
Reason
The product decision is the order within one list, so the metric must be computed within one list and then aggregated.
Boundary
Search, recommendation, case prioritization and offer ranking.
Symptom or risk
Pooling rewards separating easy lists from hard lists, so a model that only recognizes busy searches looks strong while leaving each list in the wrong order.
Exceptions
A single global list, where pooled and per-list metrics coincide.
Trade-offs
Per-list metrics are noisier when lists are short or have one relevant item.
Pitfalls
Ties among scores inside a list and a metric implementation that assumes there are none.
Example
Hypothetical: a score that rises with the day's overall demand pools well across 200 daily offer lists, yet orders the offers inside each list no better than chance.
Lessons and concepts
Claims and their sources
scikit-learn's ndcg_score treats each row as one ranking and returns the average of per-row NDCG, optionally sample-weighted; k restricts the sum to the top k, and tied scores are averaged by default.