Core day 51 of 60 · Week 11 · Part II · evaluation depth

RAG Evaluation and Optimization

Your fare-rules assistant gave a call-centre agent the wrong change fee. The team proposes a bigger model, smaller chunks or a reranker. What evidence do you collect to choose the next experiment, and which number comes first?

By the end: Given a labelled query set and ranked retrieval output, you compute recall@k, precision@k, MRR and nDCG@k, say whether a failure sits in retrieval or in the answer, and name one experiment with the metric and the paired comparison that would confirm or refute it.

At a glance

Measure each link of the RAG chain at the k the model sees, on labelled questions, and compare variants question by question.

Locate the failure
Evidence missing points to retrieval; evidence present but answer wrong points to generation.
Cutoff = context
Recall@k means nothing unless k is the number of passages in the prompt.
Pick the metric
Recall: did it arrive. MRR: how high. nDCG: graded relevance and order.
Faithful is not true
An answer can be fully supported by a stale policy passage; score correctness too.
One change per run
Same golden set, same token budget, paired wins and losses, a held-out slice.

Recall · 5 min

Without notes: list the stages a RAG answer passes through (Day 48) and say which stage a high retrieval recall does not vouch for. Then state in one sentence why the evaluation set must be frozen before you change a chunk size (Day 43).

Open due reviews

Concept · 10 min

A RAG answer is a chain: the retriever finds candidates, context assembly picks and orders them, the generator writes from what it was given. A wrong answer can start at any link, and each link has a different repair. One end-to-end score cannot say which, so you measure each link on questions whose evidence you labelled in advance.

The order of two retrieval methods can change with k, so compare them at the k the generator receives

Predict first

A reranker moves the right passage from rank 4 to rank 1 for most queries. Which metric at k = 5 shows no change?

Text description of the figure

A labelled query set over the academy's own corpus. You choose a retrieval method, a cutoff k and a chunking setting, and the lab shows recall@k, MRR and nDCG@k overall and per query, with the labelled passage marked in each ranked list so you can see which queries gained and which lost.

A RAG answer is a chain, and each link has its own metric and repair Diagram

Querylabelled evidence spans
search
Retrieverecall@k, MRR, nDCG
- candidates: to Assemble context
if it fails
Fix: candidateschunking, hybrid search, labels
Assemble contexttop k, order, budget
- top k: to Generate
if it fails
Fix: rankingreranker, filters, dedupe
Generatefaithful? correct?
- writes: to Answer
if it fails
Fix: generationprompt, abstain rule, model
Answeror abstain

A bigger model only touches the last link. It cannot answer from a passage that never reached the prompt.

The retrieval-by-answer table locates the failing link Map

Answered from memory

Fragile: fails as soon as the fact changes. Do not credit retrieval.

Working as designed

Keep these as regression cases.

Retrieval failure

Fix ingestion, chunking or ranking; a bigger model cannot help.

Generation failure

Fix context assembly, prompt or abstention; retrieval did its job.

Fill it with counts per golden question. The largest bad cell names the next experiment.

Four ranking metrics: what each one asks Comparison

Criterion	Recall@k	Precision@k	MRR	nDCG@k
Question it answers	Share of relevant passages found in the top k	Share of the top k that is relevant	How high is the first relevant passage	Graded gain, discounted by rank, against the ideal order
Uses order inside top k	No	No	Yes, first hit only	Yes, all positions
Best when	You need the evidence to reach the prompt at all	Context is expensive and noise hurts	One passage answers the question	Passages are partly relevant (graded labels)
Watch out for	Rises just because k grew; fix k to the prompt	Ceiling is R/k when a query has few relevant passages	Ignores the second and later relevant passages	Gain and ideal must be defined; values differ between tools

Toy set, k = 3: the reranker lifts the order-aware metrics most Chart

Recall@3 baseline0.67
Recall@3 reranked0.83
MRR baseline0.47
MRR reranked0.64
nDCG@3 baseline0.48
nDCG@3 reranked0.69
Precision@3 baseline0.33
Precision@3 rerankedceiling below 1: one or two relevant per query0.39

Output of the application code on six synthetic queries. At k = 8 recall is 1.00 for both runs.

The same change, question by question: nDCG@3 reranked minus baseline Chart

name change0.63
seat selection0.5
change fee0.41
infant fare0.2
refund windowtie: answer passage at rank 7 in both runs0
rule 16 codeloss: identifier lookup the baseline had first-0.5

4 wins, 1 loss, 1 tie. The average hides a regression on the slice where agents quote exact rule numbers.

Which retriever wins depends on k and on the metric Illustration

Retriever A wins at small k, B wins from k = 11. Which one is "better" depends on how many passages the prompt holds.

MRR with the same cutoff: A leads at every k, because it puts what it finds near the top. Recall and MRR can disagree about the winner.

A: precise top, misses about a quarter of queries
B: finds almost everything, ranks it lower

Simulated: 300 queries with one relevant passage each; the rank of that passage is drawn from a geometric distribution per retriever (seeded). Metrics computed with the lab's code.

A ranks what it finds near the top but misses queries; B finds almost everything lower down. Report the metric at the k the generator receives: recall@100 = 0.97 with a six-passage prompt says the evidence is in the pool, and only recall@6 says the model could see it. A larger k is reranking headroom, not quality.

Key terms: Recall@k, Precision@k, MRR, nDCG@k, Faithfulness (groundedness), Golden set and the Retrieval-by-answer table.
Every retrieval metric has a cutoff $k$ ; use the $k$ the generator actually receives.
Faithfulness is support by the context, not truth: also score correctness against a reference.
One change per run, the same context-token budget, paired wins and losses per question.

Assumptions

Evidence is labelled as document spans, so labels survive re-chunking, and two labellers agreed on a sample.
The golden set resembles real traffic and includes unanswerable questions; a slice under about ten questions gives a direction, not proof.
The cutoff equals what the generator receives, and compared variants use the same context budget.
Model-based answer scores were validated against human labels on a subset.

Application and workplace decision · 10 min

Six synthetic fare-rules queries carry graded labels (2 = answers it, 1 = helpful).
Two runs: a baseline retriever, and the same candidates after a reranker. The generator receives three passages.
Before running: which of the three proposed fixes (bigger model, smaller chunks, reranker) can this table even test?
The code prints the four metrics at k = 3 and k = 8 and the per-query change in nDCG@3. The data are tiny; the exercise is the reading.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

# Synthetic fare-rules toy set (not real data). Per query: grades of ALL relevant passages
# (2 = answers it, 1 = helpful), then the grade at ranks 1..8 for each run (0 = not relevant).
Q = {
    "change fee":     ([2, 1], [0, 1, 2, 0, 0, 0, 0, 0], [2, 1, 0, 0, 0, 0, 0, 0]),
    "name change":    ([2],    [0, 0, 0, 2, 0, 0, 0, 0], [0, 2, 0, 0, 0, 0, 0, 0]),
    "infant fare":    ([2, 1], [1, 2, 0, 0, 0, 0, 0, 0], [2, 1, 0, 0, 0, 0, 0, 0]),
    "rule 16 code":   ([2],    [2, 0, 0, 0, 0, 0, 0, 0], [0, 0, 2, 0, 0, 0, 0, 0]),
    "refund window":  ([2, 1], [0, 0, 0, 0, 0, 1, 2, 0], [0, 0, 0, 0, 0, 1, 2, 0]),
    "seat selection": ([2],    [0, 0, 2, 0, 0, 0, 0, 0], [2, 0, 0, 0, 0, 0, 0, 0]),
}

def at_k(gold, run, k):
    g = run[:k]
    hits = sum(x > 0 for x in g)
    rr = next((1 / (i + 1) for i, x in enumerate(g) if x > 0), 0.0)
    dcg = lambda xs: sum((2 ** x - 1) / np.log2(i + 2) for i, x in enumerate(xs))
    return hits / len(gold), hits / k, rr, dcg(g) / dcg(sorted(gold, reverse=True)[:k])

for k in (3, 8):
    for name, col in (("baseline", 1), ("reranked", 2)):
        m = np.mean([at_k(v[0], v[col], k) for v in Q.values()], axis=0)
        print(f"k={k} {name}: recall {m[0]:.2f} precision {m[1]:.2f} MRR {m[2]:.2f} nDCG {m[3]:.2f}")

delta = {q: float(at_k(v[0], v[2], 3)[3] - at_k(v[0], v[1], 3)[3]) for q, v in Q.items()}
print({q: round(d, 2) for q, d in delta.items()})
print("nDCG@3 reranked vs baseline: wins", sum(d > 0 for d in delta.values()),
      "losses", sum(d < 0 for d in delta.values()), "ties", sum(d == 0 for d in delta.values()))

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 27 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The metrics themselves: definitions, log2 discount, one query by hand7 minRecall, precision, MRR and nDCG defined exactly, why the discount is log2(rank + 1), and every metric computed by hand on one query and verified in Python.

Two families: set overlaps ignore order, rank metrics price it Hierarchy

Ranking metric at ktop k against labelled passages
- ignores orderSet overlap|rel ∩ top k|, two denominators
  - ÷ R (evidence)Recall@kcan only rise as k grows
  - ÷ k (context)Precision@kceiling R/k when R is small
- prices rankRank-weighted hitsa lower hit counts less
  - first hit onlyMRRweight 1/rank, mean over queries
  - every hitnDCG@kgraded gain 2^g − 1, log2 discount, ÷ IDCG

Recall and precision are the same overlap seen against R or against k. nDCG divides by the ideal ordering, so 1.0 means as good as the labels allow; its log2 discount falls more slowly than MRR's 1/rank (the discount table below).

Recall@k and precision@k, defined on sets Formula

\begin{gathered} \mathrm{recall@}k = \frac{|\,\mathrm{rel} \cap \mathrm{top}_k\,|}{|\,\mathrm{rel}\,|} \\[4pt] \mathrm{precision@}k = \frac{|\,\mathrm{rel} \cap \mathrm{top}_k\,|}{k} \end{gathered}

$\mathrm{rel}$: the set of passages labelled relevant for this query (grade > 0)
$\mathrm{top}_k$: the k passages the retriever ranked highest; order inside the set does not matter
$|\,\mathrm{rel}\,|$: denominator of recall: how much labelled evidence exists (R)
$k$: denominator of precision; with R relevant passages its ceiling is R/k

Both are set overlaps divided by a size, so neither sees order inside the top k. Averaged over the golden set, per query first.

Reciprocal rank and MRR Formula

\mathrm{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q}

$\mathrm{rank}_q$: rank of the first relevant passage for query q; the term is 0 if none is inside the cutoff
$|Q|$: number of queries in the golden set

Moving the first hit from rank 4 to rank 1 raises that query's reciprocal rank from 0.25 to 1.

nDCG@k, part by part Formula

\mathrm{nDCG@}k = \frac{\sum_{i=1}^{k} (2^{g_i}-1)/\log_2(i+1)}{\mathrm{IDCG@}k}

$g_i$: graded relevance of the passage at rank i (0, 1, 2)
$2^{g_i}-1$: exponential gain; a linear gain g_i is also common, so state which
$\log_2(i+1)$: position discount: rank 1 counts fully, rank 3 counts half
$\mathrm{IDCG@}k$: the same sum for the ideal order of ALL labelled passages, not only retrieved ones

Why log2(rank + 1): the discount by rank, next to the harsher 1/rank Table

Rank i	log2(i + 1)	nDCG discount 1/log2(i + 1)	RR weight 1/i
1	1.000	1.000	1.000
2	1.585	0.631	0.500
3	2.000	0.500	0.333
4	2.322	0.431	0.250
7	3.000	0.333	0.143
10	3.459	0.289	0.100

Rank 1 counts fully, rank 3 half, rank 7 a third: a reader scanning down still earns partial credit late in the list. Jarvelin and Kekalainen (2002) introduced a log discount for this smooth reduction; log2(i + 1) is the common modern form. Reciprocal rank's 1/i falls faster and scores only the first hit.

Slide the one answer passage down the list and watch each metric Illustration

recall@5
nDCG@5
reciprocal rank (cutoff 5)

Computed with the lab's metric code on a constructed ranking (one relevant passage, nine non-relevant). With a single relevant passage nDCG@k equals the rank discount 1 / log2(rank + 1).

One query, one labelled passage (grade 2). Move it from rank 1 to rank 10 and recompute each metric at the cutoff k = 5.

Recall only asks whether the passage is inside the cutoff; nDCG decays slowly with rank, reciprocal rank fast. Past the cutoff all three are zero.

One query, four metrics by hand (bank support KB, "card dispute deadline", illustrative) Table

Quantity	Computation	Value
Labels	grades 2 (answers it) and 1 (helpful), so R = 2	-
Run, ranks 1-5	grades 0, 2, 0, 1, 0	-
Recall@3	1 relevant in top 3, of R = 2	0.50
Precision@3	1 relevant of k = 3	0.33
Reciprocal rank	first hit at rank 2: 1/2	0.50
DCG@3	(2^2 - 1)/log2(3) = 3 x 0.631	1.893
IDCG@3	ideal order 2 then 1: 3/log2(2) + 1/log2(3) = 3 + 0.631	3.631
nDCG@3	1.893 / 3.631	0.521

Verified in Python with the application block's at_k function. The grade-1 passage at rank 4 is invisible to every @3 metric; at k = 5 recall becomes 1.00 and nDCG@5 rises to 0.64 because the discount keeps charging for the late position.

Worked example: "infant fare", baseline run, k = 3 Table

Rank	Grade retrieved	Gain 2^g - 1	Discount	Contribution
1	1 (helpful)	1	1.000	1.000
2	2 (answers it)	3	0.631	1.893
3	0	0	0.500	0.000
DCG@3	-	-	-	2.893
IDCG@3	labels 2 then 1	3 and 1	1.000, 0.631	3.631
nDCG@3	-	-	-	2.893 / 3.631 = 0.797

The reranked run puts the grade-2 passage first and scores 1.00, a gain of 0.20 (the per-query chart).

Where to look first3 minA decision path from candidate pool to ranking to generation, so each experiment changes one link.

Where to look first Diagram

Recall high at large k?
- no: to Candidate fix
yes
Recall high at prompt k?
- no: to Ranking fix
yes
Faithful and correct?
- no: to Generation fix
yes
Then check slicesstale sources, unanswerable
Candidate fixchunk boundaries, hybrid search, labels
Ranking fixreranker, lexical candidates
Generation fixcontext order, prompt, abstention

Work down the chain. Each "no" names a component to change, one at a time.

Lab sample of 70: retrieval failures are the largest bad cell Grid

Answer correct and supported →
Labelled evidence ↓	yes	no
in the top k	41	9
not in context	6	14

The hands-on lab's second synthetic sample in the retrieval-by-answer table. 41: keep as regression cases. 14: retrieval failures, the largest bad cell. 9: generation failures. 6: answered from memory, which breaks when a fee changes. Work on the largest bad cell first, one change at a time.

Answer-level checks and model judges3 minContext relevance, faithfulness, answer relevance and correctness, and how to validate a judge before trusting it.

Answer-level checks and what each one misses Comparison

Criterion	Context relevance	Faithfulness	Answer relevance	Correctness
Asks	Is the retrieved context focused on the question	Is every claim in the answer supported by the context	Does the answer address the question asked	Does it match the reference answer
Needs a reference answer	No	No	No	Yes
Misses	Whether the answer used it	Stale or wrong source passages: faithful but wrong	Correctness: a relevant answer can be false	Why it failed: pair it with the 2x2 table

The first three are the reference-free checks named by RAGAS. Computed by a model judge, they need validation against human labels (Day 45).

Before you trust a model judge Sequence

Hand-label a sample
30-50 answers, two labellers on part of it, agreement recorded.
Score the same sample
Run the judge with a fixed prompt and model version.
Measure agreement
Report agreement per class; look at the disagreements, not only the rate.
Check order effects
For pairwise comparisons, run both orders and count flips.
Keep deterministic checks
Quoted span appears in the cited passage: no judge needed.

One experiment, run honestly3 minEqual context budgets for chunking, and why longer contexts are not free.

Compare chunk sizes at the same context budget, not the same k Table

Chunk size	k	Context tokens	Fair comparison?
300 tokens	4	about 1,200	Baseline
900 tokens	4	about 3,600	No: three times the text
300 tokens	12	about 3,600	Yes, against 900 x 4

Label evidence as spans of the source document so one golden set scores every chunking.

Accuracy can depend on where the evidence sits in a long context Chart

Illustrative curves: they show the pattern, not measured results.

Shape reported by Liu et al. (2024) for some models and tasks; values here are illustrative. More passages are not a free gain.

Small golden sets, bootstrap mechanics and selection6 minHow wide the interval is at 50 or 200 questions, the paired bootstrap built step by step and by hand, and why the winner of many variants is optimistic.

How wide is a 95% interval on a rate near 0.8? Chart

Normal approximation for one rate. A paired difference is tighter, but it is driven by the few questions that change outcome.

The paired bootstrap, step by step Sequence

Score per question
Both variants' outcomes and the difference d_i = B minus A for each of the n questions; the headline is mean(d).
Resample questions
Draw n indices WITH replacement (some twice, some never): another plausible sample of the same traffic.
Keep the pairing
A drawn question brings both outcomes, so d_i moves as one unit. Resampling A and B apart overstates the noise.
Recompute and repeat
Mean of the resampled d's; repeat about 5,000 times for a distribution of plausible headlines.
Read the percentiles
2.5th and 97.5th percentiles bound a 95% interval. Zero inside: consistent with no real difference.

Only the discordant questions (d_i != 0) ever move a resampled mean, so the interval's width tracks the fixed-plus-broken count, not n and not the net gain.

Bootstrap by hand: 5 paired differences, 3 resamples (seed 42) Table

Draw	Question indices drawn	Resampled d values	Mean
Original	1 2 3 4 5	+1, 0, 0, -1, +1	+0.20
Resample 1	1 4 4 3 3	+1, -1, -1, 0, 0	-0.20
Resample 2	5 1 4 2 1	+1, +1, -1, 0, +1	+0.40
Resample 3	3 5 4 4 4	0, +1, -1, -1, -1	-0.40

Verified in Python (numpy default_rng(42), 5,000 resamples): the 95% interval is -0.40 to +0.80. A +0.20 headline from five questions is consistent with anything from "B hurts" to "B helps a lot", which is the honest reading of tiny golden sets.

The interval is set by the questions that changed outcome, not by the headline Illustration

5 questions fixed, 0 broken. Only 5 questions differ, all in B's favour: the interval stays above zero.

Same +5 headline, but 15 fixed and 10 broken. 25 questions changed outcome, so the interval is wide and includes zero.

The second case resampled as two independent groups: ignoring the pairing adds the spread of A and B separately and widens the interval further.

Synthetic outcomes on 100 questions (A correct on 70). Percentile bootstrap over questions, 2,000 seeded replicates per panel; the same procedure as the lab's paired interval. Red line: no difference.

Three bootstrap distributions of the same +5-point headline. Left two: more discordant questions (fixed plus broken), wider interval. Right: the same data resampled as two independent groups, which throws away the pairing and overstates the noise.

Report paired outcomes, not two averages (hands-on lab numbers) Table

Quantity	Value	Reading
A correct	60 of 80 (0.750)	Baseline variant
B correct	62 of 80 (0.775)	Headline: +2.5 points
Fixed by B / broken by B	6 / 4	Net gain of 2 questions
Paired bootstrap 95% interval	-0.050 to +0.100	Includes zero
Sign test p-value	0.75	No evidence of a real difference

The best of many equal variants looks better than it is Illustration

All variants are equally good (true rate 0.695). The best observed score still climbs with every variant you try: about +0.059 after 30.

Each dot is one search over 30 variants: the winner's score on the tuning set (x) and on 150 fresh questions (y). Average 0.755 when picked, 0.692 on fresh questions.

Simulated: 150 questions (half easy, 30% medium, 20% hard), every variant with the same per-question chances; 300 seeded searches. No variant is truly better, so every gain on the tuning set is selection.

A simulation with no real winner. Every variant tried on the same golden set is another draw, so the selected score rises with the number of variants and the winner's margin includes the luck that picked it; on fresh questions it falls back.

After launch, and what reviewers misreadadvanced3 minHow a golden set keeps up with real traffic, and the six readings of RAG metrics that a staff-level reviewer pushes back on.

After launch, the golden set is fed by the traffic it judges Loop

Live trafficlogged query, passages, answersample weekly ↓
Stratified sampleper slice, so rare types show uplabel ↓
Label evidencespans; two labellers on partadd cases ↓
New golden versionnew slices; each version frozenscore ↓
Score at prompt krecall, nDCG, correctness per sliceif regressed ↓
One-change testrefutation rule written firstship or drop ↓
↺ back to 1 · Live traffic

Practitioner pattern, not a standard. Versioning matters: a variant is compared on one frozen version, while the next version absorbs the slices users started asking about.

Misreadings a staff-level reviewer catches Overview

Recall@100 as quality
It is pool headroom. Quality is what reaches the prompt.
Faithful = correct
A faithful answer can restate a stale passage. Score correctness too.
nDCG across tools
Gain scale, ideal ordering and k differ. Recompute with one script.
Average beats baseline
Ask for fixed vs broken counts and the slice that carries the gain.
Same k, fair test
Different chunk sizes at the same k compare different token budgets.
Online thumbs as recall
Feedback is sparse and self-selected; use it to pick cases to label, not as a score.

Definitions that differ between toolsadvanced2 minRecall versus hit rate, linear versus exponential gain, and the ideal ordering that makes nDCG honest.

Definitions that differ between tools Table

Term	Variant A	Variant B	What to do
Recall@k	Share of relevant passages found	Hit rate: any relevant passage found	Same only when R = 1; state which
Multi-passage answers	Any one passage counts	All required passages in top k	Use the strict form when an answer needs several
nDCG gain	Linear: g	Exponential: 2^g - 1	Never compare values across gain choices
nDCG ideal	From all labelled passages	From retrieved passages only	Only the first is correct

Memory anchor

Metric at the model's k, one change per run, wins and losses per question.

Useful for

Choosing between a reranker, a different chunk size, lexical candidates or a model change from evidence instead of from an article, whether the corpus is a bank's support knowledge base or a product catalogue.
Writing a regression gate for retrieval changes that a reviewer can re-run.
Explaining to a product owner why one higher average does not settle a release.

Not suitable for

Judging open-ended generation quality (tone, helpfulness) where no evidence passage exists to label.
Corpora where nobody can label evidence at all; there, start with a small human review before building metrics.

Traps

Changing k between experiments

Recall rises because more passages were returned, and the gain is credited to the retriever.

How to avoid it

Fix the cutoff to what the generator receives and report a larger k only as headroom.

Labelling by chunk id

Re-chunking invalidates the labels, so every chunking experiment needs relabelling or silently scores nothing. A hospital team re-chunking triage guidelines from sections to paragraphs loses its whole golden set this way.

How to avoid it

Label evidence as spans of the source document and map spans onto whichever chunking is under test.

Reading faithfulness as truth

An answer that faithfully restates an outdated passage scores perfectly and is wrong, such as an insurance assistant quoting last year's claim-filing window from a stale policy document.

How to avoid it

Track correctness against a reference alongside faithfulness, and investigate faithful-but-wrong cases as corpus problems.

Senior reasoning

A senior does not ask which retriever is best. They ask which stage fails, for which query type, and what single change would move that stage. They compute the metric at the production cutoff, split by slice, compare variants on paired per-question outcomes, and write down the result that would make them abandon the idea before they run it.

Interview question

An e-commerce site's product-help RAG assistant shows offline recall@5 of 0.9, but shoppers keep reporting wrong answers about returns and warranties. How do you find out why?

Outline your answer first.

Recall cards scheduled after completion

Why report recall@k at the k you pass to the model, and which metric ignores order inside the top k?
Faithfulness is high but users get wrong answers. What does that tell you?
What must you hold fixed when comparing chunk sizes, and how do you report the result?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Cumulated gain-based evaluation of IR techniques — Kalervo Järvelin, Jaana Kekäläinen, 2002core
Defines cumulated gain, discounted cumulated gain and the normalised form for graded relevance; the basis of nDCG@k. (Definitions of CG, DCG and nDCG)
Introduction to Information Retrieval — Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze, 2008core
Precision at k, MAP and NDCG as measures for ranked retrieval results. (Section 8.4, evaluation of ranked retrieval results)
The TREC-8 Question Answering Track — Ellen M. Voorhees, Dawn M. Tice, 2000optional
The TREC-8 question answering track scored each question by the reciprocal rank of the first correct answer, the origin of MRR.
RAGAs: Automated Evaluation of Retrieval Augmented Generation — Shahul Es, Jithin James, Luis Espinosa Anke, Steven Schockaert, 2024core
Names the reference-free RAG metrics (faithfulness, answer relevance, context relevance); computing them with a model judge needs its own validation (Day 45).
Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2024optional
Evidence that accuracy depends on where in a long context the relevant passage sits, which is why more context is not a free gain.
Passage Re-ranking with BERT — Rodrigo Nogueira, Kyunghyun Cho, 2019optional
Cross-encoder re-ranking of first-stage candidates improved passage ranking, the mechanism the toy reranker stands in for.

Optional sections

Deep dive · 10 min

Definitions you must write down. Recall@k has two common forms: the fraction of labelled-relevant passages found in the top $k$ (used here) and "hit rate", whether any relevant passage appears. They coincide only when $R=1$ . For questions that need several passages, a stricter "all required passages in the top $k$ " variant is the honest one.

nDCG variants. Gains can be linear ( $g_i$ ) or exponential ( $2^{g_i}-1$ ); the discount is usually $\log_2(i+1)$ ; the ideal ordering must use all labelled passages, not only the retrieved ones, or a run that retrieves little looks perfect. Two nDCG values computed with different gain scales or cutoffs are not comparable. When every question has exactly one relevant passage, nDCG adds little over MRR.

Answer-level scores. Context relevance, faithfulness and answer relevance can be computed by a model judge without a reference answer, which makes them cheap and also makes them only as good as the judge. Validate the judge on a human-labelled subset (Day 45), run pairwise comparisons in both orders, and keep a deterministic check wherever one exists, such as "every quoted span appears in the cited passage".

Small golden sets. With 50 to 150 questions, a gap of two or three points is a difference of two or three questions. Compare variants on the same questions, count questions fixed and broken, and put a bootstrap interval over questions on the paired difference. Each variant tried and each look at the set adds a comparison, so keep a held-out slice and report how many variants you tried.

Hands-on lab: Paired comparison of two RAG variants on a synthetic golden set · 15 min

The code simulates per-question outcomes for variant A and variant B on 80 synthetic questions (answer correct and supported = 1). B fixes 6 questions A got wrong and breaks 4 that A got right. Run it, then (1) build the retrieval-by-answer table for the second block and say which repair each cell suggests, (2) explain why the interval and the sign-test p-value disagree with a headline of "B is 2.5 points better", and (3) change the numbers so B fixes 14 and breaks 4, and report when you would ship.

import numpy as np
from scipy.stats import binomtest

rng = np.random.default_rng(0)
n = 80
a = np.zeros(n, dtype=int)
a[:60] = 1                                  # A answers 60 of 80 correctly (synthetic)
b = a.copy()
b[60:66] = 1                                # B fixes 6 questions A got wrong
b[:4] = 0                                   # B breaks 4 questions A got right
fixed, broken = int(((a == 0) & (b == 1)).sum()), int(((a == 1) & (b == 0)).sum())
print("A", a.mean(), "B", b.mean(), "fixed", fixed, "broken", broken, "net gain", fixed - broken)

diffs = b - a
boot = [diffs[rng.integers(0, n, n)].mean() for _ in range(5000)]
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"bootstrap 95% interval for the paired difference: [{lo:+.3f}, {hi:+.3f}]")
print("sign test p-value:", round(binomtest(fixed, fixed + broken, 0.5).pvalue, 2))

# retrieval-by-answer table on a second synthetic sample: (evidence in context, answer supported)
pairs = [(1, 1)] * 41 + [(1, 0)] * 9 + [(0, 1)] * 6 + [(0, 0)] * 14
for ev in (1, 0):
    for ans in (1, 0):
        c = sum(p == (ev, ans) for p in pairs)
        print(f"cell evidence={ev} answer_supported={ans}: {c}")

Interview practice · 10 min

On an insurance claims-support assistant, a teammate shows that variant B beats variant A by three points on a 90-question golden set and wants to ship B. What do you ask for before agreeing?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-13 · depends on numpy, scipy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 27 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min