Classical ML, validation, metrics, interpretation, debugging · insight-1146
Let the stake of each case set its own action threshold when costs scale with value
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- banking, insurance, retail
Summary: situation, action, reason, boundary
Situation
A single probability cut-off is used for retention offers, refunds or fraud holds although the money at risk varies by a factor of fifty between cases.
Action
Act when probability times the loss the action would avert exceeds the action cost for that case, using calibrated probabilities, and tune the cost inputs on validation outcomes.
Reason
Expected-cost minimization compares the expected loss of acting and of not acting per case; when losses scale with value, the break-even probability scales inversely with it.
Boundary
Decisions with a known per-case amount: order value, claim size, fare, exposure.
Symptom or risk
A global threshold acts on cheap cases with a modest probability and ignores expensive ones just below it; the average cost looks fine while the large losses leak.
Exceptions
A fixed review capacity turns this into a ranking by expected value rather than a threshold.
Trade-offs
Needs calibrated probabilities and a reliable value estimate; errors in either move the decision.
Pitfalls
Applying the rule to probabilities from a rebalanced model without correcting them.
Example
Hypothetical: an intervention costs 15; a 300-value case is worth acting on above p of 0.05, a 60-value case only above 0.25.
Lessons and concepts
Claims and their sources
Elkan: cost-sensitive decisions can be made by applying the cost matrix to the classifier's probability estimates.