Skip to content
ML Academy

Engineering, production, governance, cost, communication · insight-1294

List who bears each error type before setting the threshold

Evidence type
practitioner heuristic
Difficulty
medium
Roles
Data scientist, Data analyst, Applied AI
Industries
insurance, banking, public-sector

Summary: situation, action, reason, boundary

  1. Situation

    A claims-flagging threshold is set from investigator hours alone, while flagged customers wait weeks for payment (hypothetical).

  2. Action

    List each error type and who bears it (company, customer, third party), estimate the burden on each party, add a cost term or a constraint such as a cap on time-to-resolution or on wrongly flagged share per group, and show the trade-off to the decision owner.

  3. Reason

    The cost matrix determines the optimal threshold, so a burden left out of it is silently priced at zero.

  4. Boundary

    Fraud, claims, eligibility, content moderation and screening models.

Symptom or risk

False positives fall on customers who face delay but never appear in the cost matrix, so the threshold looks optimal on internal cost only.

Exceptions

If the cost really is internal and reversible, the plain cost matrix is enough.

Trade-offs

Customer burden is hard to price; use caps rather than guessed monetary values.

Pitfalls

Pricing customer harm at the company's cost of the call centre contact.

Lessons and concepts