S15 · rag eval simulation

Choose a RAG configuration for a fare-rules assistant

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

Call-centre agents at the synthetic airline ask an internal assistant about fare rules, change fees and baggage policy. The corpus is 1,800 policy pages. Answers must cite the policy section used.

You built a synthetic labelled set of 150 questions: 130 have a gold passage id; 20 are deliberately unanswerable from the corpus. A reviewer checked answers on a fixed subset of 60 answerable questions for faithfulness (every claim supported by the cited passage).

Constraints from the product owner (simulated): p95 latency at most 3.0 seconds; cost at most 2.00 USD per 1,000 queries at hypothetical prices; agents must be told when the policy does not cover a question.

Stage 1 of 4

The first prototype "feels good" in demos. A simulated engineer proposes going straight to an LLM-as-judge score of answer quality to compare configurations.

What do you measure first, and why?

The LLM-as-judge answer score only; it captures retrieval and generation together in a single end-to-end number, so there is no need for separate stage metrics.Retrieval first: gold-passage recall and MRR on the labelled set; then faithfulness on the reviewed subset and abstention on unanswerable items.Latency and cost first; quality can be tuned later, once the constraints are known.