Classical ML, validation, metrics, interpretation, debugging · insight-1155
Measure how often two labelers agree before treating a score gap as progress
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Labels come from analyst judgement or agent coding and the team tunes models for another point of accuracy.
Action
Relabel a random sample with a second reviewer, report raw agreement and a chance-corrected statistic, and read metric differences against the disagreement rate.
Reason
A label is a measurement with error; the rate at which labelers disagree shows how much of any score difference could be label noise.
Boundary
Judgement labels: severity, root cause, complaint category, manual risk ratings.
Symptom or risk
If two reviewers disagree on 10% of cases, accuracy against one reviewer's labels partly measures that reviewer's noise and quirks; gains of a point or two are inside the noise.
Exceptions
Labels from system-of-record outcomes (a ledger, a boarding scan) are not reviewer judgements.
Trade-offs
Double labelling costs reviewer time; a sample of a few hundred is often enough.
Pitfalls
Computing agreement on cases chosen because the model was unsure, which understates it overall.
Example
Hypothetical: 200 double-labelled tickets agree on 182, that is 91%; the 9% contested tickets are larger than a 0.5-point model gain, so that gain is not demonstrated progress.
Lessons and concepts
Claims and their sources
scikit-learn documents Cohen's kappa as a statistic intended to compare labelings by different human annotators.