Neural networks, deep learning, NLP · insight-1459
Reweight to the production class prior when labelling effort was split evenly across intents
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
A labelling budget was split equally over six intents so each has the same count, while production traffic has one dominant intent and several rare ones.
Action
Estimate production priors from a small random sample, rescale predicted odds by the ratio of production to training priors before thresholding, and recompute precision and expected cost under the production prior.
Reason
Lipton et al. define label shift as p(y) changing while p(x given y) stays fixed, and show it can be detected and corrected from a classifier's confusion matrix; sampling by class keeps p(x given y) intact. For a calibrated classifier, rescaling odds by the prior ratio is Bayes' rule.
Boundary
Intent, topic and defect classifiers whose labelled set was stratified by class.
Symptom or risk
Precision, calibrated probabilities and thresholds measured on the balanced sample do not transfer; rare-class precision in production is far lower than offline.
Exceptions
If the sample was drawn by keyword search, p(x given y) also changed and prior correction is not enough.
Trade-offs
Random-sample labelling for priors costs budget and rare priors stay noisy.
Pitfalls
Correcting twice when the loss was already class-weighted; reporting recall unchanged and calling the model validated.
Example
Hypothetical: recall 0.8 and false-positive rate 0.1 give precision 0.89 at a 50 percent prior but 0.47 when the class is 10 percent of traffic.
Lessons and concepts
Claims and their sources
Lipton, Wang and Smola define label shift as a change in p(y) with p(x given y) fixed, and give a black-box method to detect and correct it from a classifier's confusion matrix.