S27 · interview simulation
Teach back calibration to a new analyst
People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.
- 4stages
- 25 minabout
- 0stages committed
The brief
An insurer (simulated) scores incoming claims for possible fraud. The investigation unit reviews claims whose expected saving exceeds the review cost. A new analyst (simulated) joins and asks you to explain how the scores are used.
Facts for this exercise (synthetic and hypothetical): fraud base rate 2% of claims; review cost 150 USD per claim; average loss avoided when fraud is caught 4,000 USD. The model was trained on data undersampled to 50% fraud.
Rule of the exercise: write your explanation before reading any feedback. The interviewer then probes twice.
Stage 1 of 4
Simulated analyst: "What does it mean for the model to be calibrated, and why do you care for this decision?"
Explain calibration and how it connects to the review threshold, in plain language a new analyst can follow. Include the threshold implied by the costs above.