S20 · rag eval simulation

The golden answers went stale before the retriever did

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

A bank (simulated) runs a policy assistant that answers staff questions about card-dispute handling from internal policy documents. A golden set of 150 questions (version v3) pairs each question with the expected answer and the policy section it comes from. Staff reviewed the labels when policy revision 7 was current.

This week's release combined two changes: a retriever change (smaller chunks plus hybrid search) and the policy documents moving to revision 8 (the dispute-filing window and several fees changed; the figures are hypothetical).

Pass means the answer matches the expected answer on a rubric check by a reviewer (simulated). The release gate is "pass rate does not fall by more than two points". All numbers below are simulated.

Stage 1 of 5

Golden set v3 results:

	Before release	After release
Pass	126 / 150 (84.0%)	117 / 150 (78.0%)

Case level: 21 questions went from pass to fail, 12 from fail to pass. The release manager says: "Six points down. Revert the retriever."

What is your first move?

Revert the retriever as the gate says, and re-run the set on the old retriever with revision 8 documents to see whether the pass rate recovers before reintroducing anything.Change the expected answer of every failing case to the new output, then re-run, because the gate is meant to catch regressions and not to enforce yesterday's wording.Hold the release and wait for staff to report wrong answers in production, since the golden set is only a sample and users are the ground truth.Split the 21 regressions by whether the question's source section changed in revision 8, and by question type, before deciding what the six points are made of.