Classical ML, validation, metrics, interpretation, debugging · insight-1145
Size recall and precision intervals by the number of positives, not by the number of rows
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- banking, manufacturing, aviation
Summary: situation, action, reason, boundary
Situation
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
Action
Put a binomial or bootstrap interval on recall and precision using the positives (or flagged cases) as the sample size, and extend the test window or label more positives before ranking close models.
Reason
Recall is a proportion over positives and precision a proportion over alerts, so their uncertainty depends on those counts.
Boundary
Rare-event models: fraud, no-shows on thin routes, defects, incidents.
Symptom or risk
The large row count creates a false sense of precision; recall is estimated from 40 cases and a few more or fewer hits changes the story and the model ranking.
Exceptions
Metrics averaged over all rows, such as log loss or Brier score, have more effective samples.
Trade-offs
Wider honest intervals make selection harder, which is the point.
Pitfalls
Using the normal approximation at small counts or near 0 and 1, where Wilson or bootstrap intervals behave better.
Example
Wilson 95% interval for 32 hits out of 40 positives is roughly 0.65 to 0.90; for 320 of 400 it is roughly 0.76 to 0.84.