S01 · incident simulation

The assistant that stopped finding answers overnight

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

You are on call for a fare-rules assistant (retrieval over about 50,000 policy chunks, then a generator). It is 08:00. All numbers are simulated for this exercise.

Alert at 07:40: thumbs-down rate 11.8% (usual 3.1%) and "no relevant passage found" replies 22% (usual 3%).
Latency, error rate, traffic and generator token counts are normal.
Release log: at 02:10 a release switched the query embedding model from model E1 to a newer model E2 and started a migration job that re-embeds chunks in batches of 2,000 and overwrites each vector in place. At 08:00, 17,750 of 50,000 chunks (35.5%) have been re-embedded.
A nightly snapshot of the vector index exists from 02:00 (before the migration). About 120 chunks were added or edited since.
Your golden set has 1,000 queries with a known relevant chunk (replay run on current production retrieval, in this exercise): hit@5 was 0.85 before the release.

Stage 1 of 5

The on-call channel has three suggestions: roll back the generator prompt (changed last week), check the retrieval layer against the golden set, or scale out the service.

What do you do first?

Roll back last week's prompt change straight away, because it is the most recent change to the part of the system that writes the answers customers see and rollback is quick.Replay the golden set through retrieval alone, without the generator, and compare hit@5 with the pre- release value, while listing every change since 00:00 against the onset.Scale out the retrieval service and the generator workers to absorb load, because users are retrying and the thumbs-down rate suggests a capacity problem on the hosted model.Page the vendor of the generator model and ask whether the model's behaviour changed overnight, since model updates are a common cause of answer-quality drops.