S15 · rag eval simulation

A retrieval upgrade scores worse because the gold labels are incomplete

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A synthetic airline's agent-assist tool answers questions from 1,800 fare-rule and policy pages. Your synthetic evaluation set has 120 questions, each with one gold passage id. The gold labels were created in version 1 by looking at what the current dense retriever (configuration A) returned and marking the passage that answered the question.

A candidate configuration B (hybrid lexical plus dense search, cross-encoder rerank, top-5) is ready. Recall is defined as "the gold passage is among the top 5". Many fare rules appear in two places: a summary page and the full rule text.

Stage 1 of 4

Simulated results, gold passage in the top 5, n = 120: A 94 (0.78), B 86 (0.72). Human faithfulness review of generated answers on 60 questions: A 0.85, B 0.90. Reviewers add that many of B's answers cite a different passage than the gold one.

What do you do before deciding between A and B?

Reject B: recall is the agreed retrieval metric, 0.72 is below 0.78, the faithfulness gain rests on only 60 reviewed answers, so it is within what chance could produce, and the team agreed the metric before the test began.Take B's misses on gold and have a reviewer judge whether the passages B returned answer the question, then compare on judgments independent of the system that built the labels.Adopt B: rerankers improve the ordering of candidates in general, the faithfulness review already points in its favour, so the recall number can be set aside until the labels are next refreshed.Switch the headline metric to MRR, which is more tolerant of ordering changes, and rerun the comparison; if B is ahead on MRR the retrieval question is settled and the labels need no review.