S01 · incident simulation

Reconfirmation calls that went out twice

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A synthetic airline (simulated company) runs a nightly pipeline: a no-show model scores passengers on departures in the next 48 hours, and the top-scoring passengers are written as reconfirmation call tasks to a work queue that the call-centre team starts working at 06:30. The pipeline orchestrator retries a failed step automatically up to three times. The write step inserts tasks in chunks of 1,000 rows through the queue's API. All people, logs and counts are simulated.

Stage 1 of 4

Tuesday 07:15. The call-centre lead (simulated) writes: "Several passengers complained they were called two or three times since 06:30. The queue shows 21,400 tasks and your model flagged 12,400 passengers. 3,100 tasks are already marked completed."

Calls are still going out: the team is working the queue at about 70 tasks a minute.

What do you do first?

Delete the duplicate tasks directly in the queue table, keeping the lowest id per passenger, so the team has a clean list as quickly as possible and no passenger is called twice.Pause the queue so that no further calls go out, keep the task table exactly as it is for evidence, and tell the call-centre lead what you are doing and why.Roll the scoring model back to the previous version, since the model produced the passenger list that ended up in the queue and the queue is the model's output.Ask the team to ignore duplicates and handle them manually as they appear, while you work on the root cause in parallel and the queue keeps running.