Classical ML, validation, metrics, interpretation, debugging · insight-1712
Correct a share estimated by counting classifier labels when the classifier is imperfect and prevalence moved
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- aviation, manufacturing
Summary: situation, action, reason, boundary
Situation
A report estimates the share of positive cases, such as cancellations, by counting a classifier's positive labels in a new period.
Action
Estimate TPR and FPR at the fixed threshold on held-out labelled rows and report (predicted positive share - FPR) / (TPR - FPR), clipped to [0, 1], with an interval that bootstraps both the new period and the rates.
Reason
The expected predicted-positive share is TPR * p + FPR * (1 - p), a straight line in true prevalence p; solving for p undoes the bias.
Boundary
Aggregate shares for monitoring and planning, not case-level decisions.
Symptom or risk
The count is biased whenever TPR is below 1 or FPR above 0, and the bias changes with prevalence, so a trend can mislead.
Exceptions
It assumes TPR and FPR stay put when prevalence moves, and is unstable when TPR is close to FPR.
Trade-offs
A weak classifier gives a noisy corrected count.
Pitfalls
Estimating TPR and FPR on training rows.
Example
Simulated (one Gaussian feature, 300 draws): trained at 20% prevalence; TPR about 0.55 and FPR 0.05 from 500 labelled rows; the new period has 2,000 rows, 50% positives. Counting labels gives 29.8%; the adjusted count 50.2%, with spread 0.058 versus 0.015.
Lessons and concepts
Claims and their sources
Saerens, Latinne and Decaestecker state that posteriors from a classifier trained under one class prior are not valid under new priors and can be adjusted.
Forman's paper on quantifying counts and costs via classification (Data Mining and Knowledge Discovery 17, 2008) is cited for the existence and topic of the quantification literature; only the bibliographic record was checked, and the adjustment formula here is derived from the definitions of TPR and FPR.