Neural networks, deep learning, NLP · insight-1413
Rank training examples by out-of-fold loss to find labels worth re-checking
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A classifier plateaus below what the data seems to allow, and the labels come from a rule, a vendor or a stressed annotation team.
Action
Produce out-of-fold predicted probabilities, flag examples whose given label has a low probability under a class-wise threshold, and hand the top few hundred to a reviewer. Relabel, then retrain and compare.
Reason
Confident learning estimates which labels are likely wrong by thresholded counting and ranking under a class-conditional noise assumption, and cleaning with it was reported to improve accuracy.
Boundary
Classification datasets with a few thousand to millions of labelled rows and reviewable examples.
Symptom or risk
The network memorises wrong labels, so training loss looks fine, validation is capped, and the same mistakes are copied into the next dataset version.
Exceptions
Heavily ambiguous classes, where the disagreement is the signal, not an error.
Trade-offs
Costs one cross-validated training pass and reviewer time; a flagged example is a candidate, not a verdict.
Pitfalls
Computing probabilities in-sample, where the model has memorised the wrong label.
Lessons and concepts
Claims and their sources
Confident learning finds label errors by pruning suspect examples, estimating noise through thresholded counting and ranking examples, assuming class-conditional noise.