Classical ML, validation, metrics, interpretation, debugging · insight-0153
Derive the decision threshold from the cost matrix instead of using 0.5
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- banking, aviation
Summary: situation, action, reason, boundary
Situation
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
Action
Act when the probability exceeds (with zero cost for correct decisions), then confirm on validation data by plotting expected cost against threshold.
Reason
Acting is optimal when the expected cost of acting, , is below the expected cost of not acting, .
Boundary
Binary decisions with calibrated probabilities and estimable unit costs.
Symptom or risk
A 0.5 threshold silently assumes false positives and false negatives cost the same, which is rarely true.
Exceptions
Capacity limits, legal rules or non-linear costs override the formula; then optimize the constrained objective directly.
Trade-offs
Cost estimates are uncertain; show how the threshold moves across a plausible cost range.
Pitfalls
Applying the formula to uncalibrated or rebalanced scores, which shifts the effective threshold.
Example
Hypothetical fraud review: a needless review of a legitimate transaction costs 20, a missed fraud costs 400, and a caught fraud adds no further cost, so review above 20/420, about 0.048.
Lessons and concepts
Claims and their sources
Cost-sensitive decisions can be made by applying the cost matrix to the classifier's probability estimates.