Classical ML, validation, metrics, interpretation, debugging · insight-1156
Treat unlabeled cases as unknown, not negative, when positives are only partly recorded
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- banking, manufacturing, insurance
Summary: situation, action, reason, boundary
Situation
Only investigated cases carry a fraud, defect or violation flag, and every uninvestigated case is coded as a negative.
Action
Estimate the share of positives that get recorded from a random audit sample, treat model scores as proportional to risk, and evaluate on the audited sample rather than on all unlabeled rows.
Reason
If recorded positives are a random subset of true positives, a classifier of recorded against unrecorded rows is proportional to the true probability by a constant factor.
Boundary
Fraud, compliance, safety and quality screens where only reviewed cases are labeled.
Symptom or risk
Missed positives sit in the negative class, so measured precision is understated, recall covers only the recorded positives, which may be an unrepresentative subset, and the model can learn the investigators' selection.
Exceptions
When investigation is driven by the features or by a previous model, recording is not random and the constant-factor result does not apply.
Trade-offs
A random audit costs reviewer time; it is also the only unbiased evidence of prevalence.
Pitfalls
Quoting precision from flagged-and-reviewed cases only, which ignores positives never reviewed.
Lessons and concepts
Claims and their sources
Under the assumption that labeled examples are selected randomly from the positive examples, a classifier trained on positive and unlabeled data predicts probabilities that differ only by a constant factor from the true conditional probabilities of being positive.