S11 · experiment simulation

Is the new assistant pipeline better, or luckier on 60 questions?

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
35 minabout
0stages committed

The brief

A fare-rules assistant (simulated) answers customer questions from a document store. The team changed three things at once for pipeline v2: a reranker, a rewritten prompt and a larger context (12 passages instead of 8). A golden set of 60 answerable questions with reference answers was graded by one reviewer, correct or not. The lead (simulated) reports: "v2 answers 46 of 60 correctly against 41 of 60 for v1, +8.3 points. v2 costs about 35% more per question. Ship it." All results in this scenario are simulated; the arithmetic was computed in Python.

Stage 1 of 5

Both pipelines ran once at temperature 0.7 on the same 60 questions.

What is the first thing you ask for before interpreting 46 against 41?

Which of the three changes helped, so that the unneeded ones can be removed to cut the cost per question as quickly as possible and the budget can be defended.Question-level outcomes for both systems and the run-to-run spread of one pipeline, to see how big a difference noise alone produces on 60 questions.Whether a bigger model would give an even larger improvement than +8.3 points, since the team should aim higher than this result and not settle for a modest gain.