Classical ML, validation, metrics, interpretation, debugging · insight-1148
Pick a threshold in the flat part of the expected-cost curve, not at its sharpest minimum
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- banking, insurance, aviation
Summary: situation, action, reason, boundary
Situation
Your cost curve over thresholds has a minimum at 0.19 but the unit costs behind it are rough estimates.
Action
Plot expected cost against threshold for a plausible range of cost ratios, then choose a threshold inside the region that stays near-optimal across that range.
Reason
The optimal threshold depends on the ratio of error costs, which is uncertain; robustness to that ratio matters more than the last decimal of cost.
Boundary
Any cost-based or profit-based threshold with estimated unit costs.
Symptom or risk
A threshold at the exact minimum is tuned to noise in validation counts and to cost numbers that are off by a factor of two; small changes in either move the optimum.
Exceptions
If costs are contractual and exact, the formula threshold can be used directly.
Trade-offs
Accepts a slightly higher expected cost in return for stability.
Pitfalls
Computing the curve on the data that chose the model, which makes the minimum look deeper.
Lessons and concepts
Claims and their sources
Elkan recommends making cost-sensitive decisions by applying the cost matrix to probability estimates, so the decision threshold is a function of the cost entries.