S15 · rag eval simulation

Where should the abstention threshold of a baggage-policy assistant sit?

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

Call-centre agents at the synthetic airline ask an internal assistant about baggage rules. The assistant retrieves a passage, and a reranker gives the top passage a score between 0 and 1. If the score is below a threshold the query is escalated to the specialist desk instead of being answered.

You hold a synthetic labelled set of 500 queries: 400 are answerable from the policy pages and 100 are not. A reviewer checked every answered answerable query for correctness. All numbers, costs and capacities below are hypothetical:

A wrong answer given to an agent costs 10.00 USD (rework, rebooking, goodwill).
Escalating a query costs 2.00 USD of specialist time, whether or not the assistant could have answered it.
The specialist desk can absorb at most 35% of queries.
Answering an unanswerable query counts as a wrong answer.

Stage 1 of 4

The first prototype used a threshold of 0.5 "because it is half". A simulated engineer proposes keeping it. Another proposes asking the model for a self-reported confidence between 1 and 10 and abstaining below 7.

What do you establish before choosing any threshold?

Keep 0.5: reranker scores live on a 0 to 1 scale, and a score above one half means the passage is more likely relevant than not, which is the natural place to start answering and needs no extra labelling work.Label a set that includes unanswerable queries, then tabulate by score bin how many were unanswerable and how often answered ones were wrong; the threshold then follows from costs and desk capacity.Use the model's own 1 to 10 confidence; it knows when it is unsure, so abstaining below 7 puts the decision inside the component with the most information about the answer, and the cut-off can be tuned later.Maximise gold-passage recall on answerable queries, since abstention only matters when retrieval has nothing to find, and recall is the metric the retrieval team already reports every week.