Classical ML, validation, metrics, interpretation, debugging · insight-1143
Count tied scores before you read precision at K or NDCG from a shallow model
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A shallow tree or a model with few distinct leaf values produces the scores, and precision at 100 is reported from a sort.
Action
Count distinct scores and the number of rows tied at the K-th score, report an average over random tie-breaking or a second sort key, and use metrics that average ties.
Reason
Rank metrics need a total order; with ties the order inside a tied block is undefined and results change with data order.
Boundary
Shallow trees, rule-based scores, isotonic-calibrated outputs and any model with coarse score values.
Symptom or risk
Hundreds of rows share the score at the cut-off, so which of them land in the top K depends on row order; the reported number is one arbitrary draw.
Exceptions
Continuous scores from linear models or deep ensembles rarely tie at the cut-off.
Trade-offs
Random tie-breaking is fair on average but not reproducible per run without a seed.
Pitfalls
Passing ignore_ties=True to speed up scoring on data that has ties, which gives wrong values.
Example
Hypothetical: a depth-3 tree has at most 8 leaves, so at most 8 distinct scores; on 2,500 rows that is about 312 rows per score, so the 100th row typically shares its score with hundreds of others.
Lessons and concepts
Claims and their sources
scikit-learn's ndcg_score averages ties by default; ignore_ties=True assumes no ties and gives incorrect results when ties exist.