S12 · decision value simulation

Auto-resolve or review: a confidence threshold with 1,000 reviews a day

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

A support operation uses a language-model assistant to resolve routine tickets automatically. Below a confidence threshold, a human reviews the ticket first. All figures are simulated and costs are hypothetical.

Volume: 4,000 tickets a day. A human review costs 1.80 USD per ticket and is assumed to be correct.
An automatic resolution that is wrong costs 14.00 USD on average (rework, a follow-up contact and goodwill).
The assistant's confidence has been calibrated on last month's labelled tickets (the share of auto-resolutions that are correct in each bin).

Confidence bin	Share of tickets	Tickets a day	Accuracy
0.50 - 0.60	6%	240	55.0%
0.60 - 0.70	9%	360	66.0%
0.70 - 0.80	15%	600	76.0%
0.80 - 0.90	25%	1,000	86.0%
0.90 - 0.95	22%	880	93.0%
0.95 - 1.00	23%	920	97.5%

Currently all tickets are auto-resolved. The operations lead wants a threshold and a staffing request.

Stage 1 of 5

Reviewing a ticket avoids the expected cost of a wrong automatic resolution but costs the review.

At what accuracy is a human review exactly worth its cost?

50%: review whenever the assistant is more likely wrong than right, because that is when automation fails more often than it succeeds.12.9%: review whenever the expected error rate exceeds the ratio of review cost to error cost.98.6%: automate only when the accuracy leaves less than the review cost per hundred tickets, at 0.25 USD of expected error per ticket.About 87.1%: review is worth it when the expected error cost, 14 x (1 - accuracy), exceeds 1.80 USD.