Classical ML, validation, metrics, interpretation, debugging · insight-1716
Summarise a precision-recall curve with average precision, not the trapezoidal area, when scores are coarse
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry, banking
Summary: situation, action, reason, boundary
Situation
A shallow tree, banded scorecard or isotonic-calibrated model gives only a handful of distinct scores, and its PR-AUC is the trapezoidal area under the curve.
Action
Report average precision, the step-wise sum of precision weighted by recall gained, read against the prevalence, and use the same summary for coarse and continuous scorers.
Reason
scikit-learn documents average precision as not interpolated and warns that the trapezoidal rule can be too optimistic; Davis and Goadrich say PR points must not be linearly interpolated.
Boundary
Any PR summary, above all scores with ties from leaves or bins.
Symptom or risk
Linear interpolation between few far-apart PR points credits operating points the model cannot reach, so a weak scorer shows several times the prevalence.
Exceptions
With thousands of distinct scores the two agree closely.
Trade-offs
Neither is an operating point; both depend on prevalence.
Pitfalls
Comparing one model's trapezoidal area with another's average precision.
Example
Simulated (2,000 rows, 100 positives (5%), 200 draws, a Gaussian signal cut into four levels): trapezoidal area 0.17, average precision 0.10; a random two-level score gives 0.29 versus 0.05. With the continuous signal both are about 0.14.
Lessons and concepts
Claims and their sources
scikit-learn's average_precision_score is the weighted mean of precisions with weights equal to the increase in recall, is not interpolated, and differs from the trapezoidal area under the precision-recall curve, which uses linear interpolation and can be too optimistic.
Davis and Goadrich state that, unlike ROC space, points in precision-recall space must not be linearly interpolated.