Core day 57 of 60 · Week 12 · Part II · evaluation depth

LLM Project

Your retrieval report says "recall@5 is 0.80, so the assistant is ready for a pilot". What does that sentence prove, what does it leave open, and which evidence labels must the report carry so that retrieval evidence is never read as answer quality?

By the end: From your own retrieval-project artefacts you label every reported number as retrieval-only, generated-answer, assumed or not assessed, check the unsupported-answer behaviour on both answerable and unanswerable queries, and write the strongest sentence the evidence supports.

At a glance

Retrieval numbers show the evidence was available, not that an answer used it; label every number and score abstention from both sides.

Two claims
Retrieval: passage in the top k. Generation: answer supported by its cited passages.
Four labels
Retrieval-only, generated answers, assumed, not assessed. Every number carries one.
Abstain both ways
Score refusals on unanswerable queries and false refusals on answerable ones.
Blocking gates
One wrong-jurisdiction citation blocks a pilot; it is a count, not a rate.
Not assessed is fine
Saying exactly where the evidence stops is a strength in a final report.

Recall · 5 min

From memory, one line each: (1) What is the difference between recall@5 and MRR@10? (2) Why must the held-out part of your query set stay closed during development? (3) Which control stopped the injected passage in your Week 11 incident? Then clear your due review cards.

Open due reviews

Concept · 10 min

"The document Q&A works" hides two claims. Retrieval: the governing passage is in the top k. Generation: the answer says only what the cited passages support. Your Weeks 9-11 thread always measures the first, and the second only if a real model ran and you read its answers. A third question sits beside them: what happens when the corpus cannot answer?

Recall, rank-aware metrics and chunking move differently - and none of them says whether an answer is grounded

Predict first

A retriever reaches recall@5 of 0.80 on answerable queries and a generator reads the five passages. What is the share of grounded answers?

Text description of the figure

A retrieval evaluation lab with labelled queries where you choose the retrieval method, the cutoff k and the chunking. Recall@k, MRR and nDCG are shown per configuration, so you see that a change can raise one metric and leave another unchanged.

One pipeline, two claims, and the question beside them Diagram

Answerable query
- rank passages: to Retrieve top k
Unanswerable query
- top-1 score: to Below the threshold?
Retrieve top kclaim 1: recall@k, MRR
- k passages: to Generate
Below the threshold?
- no: to Unsupported answer
yes
Abstainthe right outcome
Generateclaim 2: answers read
- writes: to Cited answer
Unsupported answerthe corpus cannot back it
Cited answersupported or not?

A retriever returns k passages for any query, including one no document governs. The abstention rule decides what happens next.

Four evidence labels Overview

Measured, retrieval-only
Recall@k, MRR, nDCG from your run files. Says nothing about answers.
Measured, generated answers
Model id, date and how many answers you read against their citations.
Assumed
A cost, latency or volume you chose; say where it came from.
Not assessed
Stated plainly. It tells the reader where the evidence stops.

Score the abstention rule from both sides Map

Correct abstention

Count it against the unanswerable queries (gate 0.80 in the toy run).

False abstention

Eager refusal fails users; gate 0.10 in the toy run.

Unsupported answer

An answer the corpus cannot back. The costly cell.

Served answer

Now check the citation: right passage, right jurisdiction, current version.

Reporting only the left column, or only the right, hides half of the rule's behaviour.

One threshold, two errors, and why tuning cannot pass both gates Illustration

At 0.50: 86% of unanswerable queries refused, 25% of answerable ones refused too.

correct abstention
false abstention
unsupported share of served (dashed)

Raising the threshold buys correct abstention with false abstention. With these scores no threshold passes both gates.

current top-1 score
better-separated score

A score that separates the two groups better passes both gates for thresholds 0.43 to 0.57 on this synthetic set.

Seeded synthetic scores: 200 answerable and 120 unanswerable queries per score type. Gates as in the application (0.80 and 0.10). Choose the threshold on development queries, then confirm once on the held-out part.

The threshold only chooses a point on the trade-off. A score that separates answerable from unanswerable queries moves the whole curve.

Toy run against gates fixed before the run Table

Gate	Value	Threshold	Result
Recall@5 on answerable queries	0.80 (n = 10)	at least 0.80	Pass
Abstention on unanswerable queries	0.67 (4 of 6)	at least 0.80	Fail
False abstention on answerable queries	0.20 (2 of 10)	at most 0.10	Fail
Wrong-jurisdiction top-1 served	1	0	Fail: blocks the pilot
Generated-answer competence	not assessed	-	Label it

The same 80% recall, four query-set sizes Illustration

95% interval 0.50 to 1.00: width 0.50 (normal approximation 0.50).

95% interval 0.68 to 0.93: width 0.25 (normal approximation 0.25).

95% interval 0.72 to 0.87: width 0.15 (normal approximation 0.16).

95% interval 0.76 to 0.84: width 0.08 (normal approximation 0.08).

Synthetic: each set has exactly 80% hits; 2,000 seeded bootstrap resamples per panel. The shaded band is the 95% percentile interval. The 10-query panel is the size of the application's toy run.

Bootstrap distributions computed on synthetic hit lists. The width shrinks with the square root of the number of queries.

Give every number one Evidence label: measured retrieval-only; measured generated answers (model, date, number read); assumed; not assessed.
A retrieval hit shows evidence was available, never that it was used: retrieval success is not a groundedness result.
Four checks: the Golden set (labels before runs, frozen held-out file, overlap check, audit rate); the paired Recall@k result against the Baseline with an interval; the Abstention rule scored on both unanswerable and answerable queries, plus safety gates averages cannot hide; provenance and generation mode.
A public regulation may be memorised by the model, so an uncited correct answer is not evidence for retrieval.

Assumptions

Labels were written before retrieval was run and the held-out part was not used for tuning; otherwise it is a development set.
The queries are in one author's register of English; results describe that phrasing, not agents in general.
The unanswerable queries were checked against the corpus snapshot; an "unanswerable" query that the corpus does answer inflates abstention scores.

Application and workplace decision · 10 min

(a) List every number in your results table and attach one of the four labels; no run file behind it means not shown.
(b) From your development and held-out run files, compute abstention on unanswerable and on answerable queries, and read five failures.
(c) Write the claim as one sentence that carries its label.
Then run the code: a toy run file (synthetic passages, ten answerable and six unanswerable queries) scored against four gates fixed before the run, in the style of project 036. Substitute your own run file.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np

THRESH = 0.45  # abstain when the top-1 score is below this (toy value; tune on dev in a real run)
# Toy run file, synthetic: answerable queries as (jurisdiction, relevant id, ranked top-5 ids, top-1 score)
ans = [
    ("eu", "eu7", ["eu7", "eu5", "us3", "eu8", "eu2"], 0.71),
    ("eu", "eu9", ["eu7", "eu9", "eu5", "eu1", "eu2"], 0.66),
    ("us", "us3", ["us3", "us1", "eu5", "us4", "us2"], 0.74),
    ("us", "us5", ["us1", "us2", "us3", "us5", "eu7"], 0.58),
    ("eu", "eu12", ["eu5", "eu7", "eu1", "eu2", "eu9"], 0.52),
    ("eu", "eu4", ["eu4", "eu5", "eu7", "us1", "eu2"], 0.49),
    ("us", "us8", ["us2", "us8", "us1", "eu5", "us3"], 0.47),
    ("eu", "eu6", ["eu6", "eu5", "eu9", "eu1", "eu2"], 0.41),
    ("us", "us9", ["eu5", "us1", "us2", "us3", "us4"], 0.39),
    ("eu", "eu3", ["us3", "eu3", "eu5", "eu7", "eu1"], 0.55),
]
unans = [("eu", .28), ("us", .33), ("eu", .36), ("us", .40), ("eu", .48), ("us", .52)]  # (jurisdiction, top-1 score)

hit = np.array([rel in top5 for _, rel, top5, _ in ans])
rr = np.array([1 / (top5.index(rel) + 1) if rel in top5 else 0 for _, rel, top5, _ in ans])
rng = np.random.default_rng(57)
lo, hi = np.percentile([rng.choice(hit, len(hit)).mean() for _ in range(2000)], [2.5, 97.5])
abst_un = np.mean([s < THRESH for _, s in unans])  # correct abstentions
abst_an = np.mean([s < THRESH for *_, s in ans])  # false abstentions
wrong = sum(top5[0][:2] != jur and s >= THRESH for jur, _, top5, s in ans)  # served, wrong jurisdiction

print(f"recall@5 {hit.mean():.2f} (95% bootstrap {lo:.2f} to {hi:.2f}, n={len(hit)}), MRR@5 {rr.mean():.3f}")
print(f"abstained on unanswerable {abst_un:.2f}; falsely abstained on answerable {abst_an:.2f}")
print(f"wrong-jurisdiction top-1 served: {wrong}")
gates = {"recall@5 >= 0.80": hit.mean() >= 0.80, "abstain unanswerable >= 0.80": abst_un >= 0.80,
         "false abstention <= 0.10": abst_an <= 0.10, "zero wrong-jurisdiction": wrong == 0}
for name, ok in gates.items():
    print(f"  {'pass' if ok else 'FAIL'}  {name}")
print("pilot go:", all(gates.values()), "| label: retrieval-evaluation mode, generated-answer competence not assessed")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 39 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The method: selective answering as a decision ruleadvanced6 minCoverage and selective risk defined, and the minimum-expected-cost rule derived, showing why costs and the traffic mix both set the threshold.

Selective answering: coverage and risk as functions of one threshold Formula

\phi(\tau) = P(s \ge \tau), \qquad R(\tau) = P(\text{unsupported or wrong} \mid s \ge \tau)

$s$: a confidence score per query: here the top-1 retrieval score; a reranker or an entailment score works the same way
$\tau$: the threshold: answer when s is at least tau, abstain otherwise
$\phi(\tau)$: coverage: the share of all queries that receive an answer; it can only fall as tau rises
$R(\tau)$: selective risk: the error rate among served answers only; a refusal is lost coverage, not an error
$\text{unsupported or wrong}$: unsupported: no passage governs the query; wrong: a passage exists and the answer still fails it

An abstention rule is a selective classifier: one score, one threshold, answer or refuse. Its two errors are an unsupported answer and a false abstention. Every threshold gives one (coverage, risk) pair; together they form the risk-coverage curve, and choosing a threshold means choosing a point on it.

The threshold rule, derived in four lines Sequence

Cost of answering
With q = P(answerable | s), answering risks an unsupported answer: expected cost (1 - q) x c_u.
Cost of refusing
Refusing risks a false abstention: expected cost q x c_a.
Answer when cheaper
(1 - q) x c_u < q x c_a, so answer when q > c_u / (c_u + c_a).
Bayes brings in the mix
q / (1 - q) = (1 - pi) / pi x f_A(s) / f_U(s): the traffic mix pi enters the rule exactly like a cost.

The threshold as a decision rule: answer when it is cheaper than refusing Formula

\text{answer} \iff (1 - q)\,c_u < q\,c_a \iff q(s) > \frac{c_u}{c_u + c_a} \iff \frac{f_A(s)}{f_U(s)} > \frac{\pi}{1 - \pi}\cdot\frac{c_u}{c_a}

$q(s)$: P(the corpus can answer | score s), estimated on development queries; it must be calibrated to be used this way
$c_u$: cost of serving an unsupported answer: a wrong coverage promise to a policyholder, a disputed refund
$c_a$: cost of refusing an answerable query: a hand-off to a human agent, a lost minute
$\pi$: share of unanswerable queries in the traffic the system will serve - not in the test file
$f_A / f_U$: likelihood ratio of the score under answerable vs unanswerable queries; if it rises with s, the rule is one threshold on s

Bayes' rule turns q into prior odds times the likelihood ratio, which is why the traffic mix enters. With c_u = 3 c_a the rule answers only when q exceeds 0.75. Costs and mix choose the point; how well the score separates the two groups decides which points exist.

Gates or a cost rule: two legitimate ways to pick the point Comparison

Criterion	Gates	Cost rule
What it fixes	Constraints on the two error rates	One expected cost to minimise
Toy example	Correct abstention at least 0.80, false abstention at most 0.10	Unsupported answer = 3 refusals, at the expected traffic mix
Written	Before the run	Before the run
The illegitimate move	Loosening a gate after seeing the results	Re-pricing the costs after seeing the results

Gates can be infeasible, as on the toy run; a cost rule always returns a cheapest point.

Same algebra, different point: where the cost ratio puts the threshold Scale

SaaS help centrea refusal is a support ticket: answer more often
Insurance policy assistanttoy costs: an unsupported answer = 3 refusals
Clinical-guideline lookupan unsupported dosage answer: refuse more often

Qualitative placement, not measured costs. Moving right raises the threshold; the costs differ by orders of magnitude across sectors.

Sweeping the threshold by hand on the toy run6 minThe sweep algorithm, the toy run file pushed through it row by row, and the AUC computed by counting pairs.

Sweeping the threshold: the algorithm Sequence

Score development queries
One score per query, with its label: answerable or unanswerable. Never the held-out part.
Sort once
Sort all scores; only the distinct scores are candidate thresholds, since nothing changes between two neighbours. O(n log n).
Cumulative counts
Walking up the sorted list, count answerable and unanswerable queries below each candidate: false and correct abstentions.
Rates per candidate
Coverage, false abstention, correct abstention, unsupported share among served and expected cost, all from the two counts.
Choose by the rule written first
The cheapest candidate under the stated costs and mix, or the candidates that pass every gate - possibly none.

Worked by hand: the toy run file swept (10 answerable, 6 unanswerable) Table

Answer if score at least	Served (of 16)	False abstention	Correct abstention	Unsupported share of served	Cost per 1,000 (test mix; 10% mix)
0.30	15	0 of 10	1 of 6	5 / 15 = 0.33	938; 250
0.37	13	0 of 10	3 of 6	3 / 13 = 0.23	563; 150
0.45 (application)	10	2 of 10	4 of 6	2 / 10 = 0.20	500; 280
0.50	7	4 of 10	5 of 6	1 / 7 = 0.14	438; 410
0.53	5	5 of 10	6 of 6	0 / 5 = 0	313; 450
0.60	3	7 of 10	6 of 6	0 / 3 = 0	438; 630

Cost = 1,000 x [pi x 3 x (1 - correct abstention) + (1 - pi) x 1 x false abstention], refusal = 1 unit, unsupported answer = 3 (hypothetical). No row passes both gates (correct abstention 0.80 needs at least 0.49, which refuses 3 of 10 answerable). Recomputed in Python.

Why no threshold passes both gates on the toy run, and what a better score would change Illustration

Answerable scores start at 0.39; unanswerable ones reach 0.52. The correct-abstention gate needs a cut above 0.48, which already refuses 3 of 10 answerable queries. AUC = 51.5 / 60 = 0.86.

Each dot is one candidate threshold. Coverage falls from 16 to 1 served answers while the unsupported share falls from 0.38 to 0.00. A threshold only picks a dot; a better-separated score moves the whole curve down.

The application's synthetic run file, unchanged. Answer when the score is at least τ; a tie (0.52 against 0.52) counts as half a pair in the AUC, as sklearn's roc_auc_score does.

Left, the sixteen toy scores on one axis; any cut inside the overlap errs on both sides. Right, the same sweep as a risk-coverage curve. Checked against sklearn roc_auc_score.

Served-error algebra and the traffic mixadvanced5 minWhy the unsupported share of served answers is a property of the rule and the traffic together, and how to reweight a test file to production.

Served-error algebra: the unsupported share is a floor on selective risk Formula

R(\tau) = \frac{U_{\text{served}} + e\,A_{\text{served}}}{U_{\text{served}} + A_{\text{served}}} \;\ge\; \frac{U_{\text{served}}}{U_{\text{served}} + A_{\text{served}}}

$U_{\text{served}}$: unanswerable queries the system answered anyway (toy run: 6 - 4 = 2)
$A_{\text{served}}$: answerable queries it did answer (toy run: 10 - 2 = 8)
$e$: generation error rate on served answerable queries, known only from reading answers; a retrieval-only run sees just the floor

Toy run: 2 / (2 + 8) = 20% of served answers have no governing passage at all - on the test file's mix of queries, which is not the traffic's. Wrong answers to answerable queries come on top.

The same rates, a different traffic mix Formula

u = \frac{\pi\,(1 - a_U)}{\pi\,(1 - a_U) + (1 - \pi)\,(1 - f_A)}

$u$: unsupported share of served answers
$\pi$: share of unanswerable queries in the traffic: 6 / 16 = 0.375 in the toy file
$a_U$: correct-abstention rate on unanswerable queries (toy run at 0.45: 4 of 6)
$f_A$: false-abstention rate on answerable queries (toy run at 0.45: 2 of 10)

The served-error count formula rewritten with rates, exactly like precision from recall and a base rate. The two rates are properties of the rule; the share among served answers is a property of the rule and the traffic together.

Unsupported share of served answers at threshold 0.45, by traffic mix Table

Unanswerable share of traffic	Unsupported part	Answerable part	Unsupported share of served
5% (a SaaS help centre, hypothetical)	0.05 x 0.333 = 0.017	0.95 x 0.8 = 0.760	2.1%
10% (a bank's internal-policy assistant, hypothetical)	0.033	0.720	4.4%
20%	0.067	0.640	9.4%
37.5% (the toy test file)	0.125	0.500	20.0%
50% (an open public chatbot, hypothetical)	0.167	0.400	29.4%

Same threshold, same two error rates, a ten-fold range of served risk. A test file deliberately rich in unanswerable queries measures the rates well; reweight to the expected traffic before quoting a risk. Computed in Python.

The served-error algebra and the cost-optimal threshold, computed on the toy run Illustration

answer if score ≥ 0.37
answer if score ≥ 0.45
answer if score ≥ 0.53

Dots mark the test mix. At 0.45 the same rates give 20% unsupported among served answers on the test mix and 4% at a 10% production mix. Report the share for the traffic you expect, not the test file's.

test mix (6 of 16 unanswerable)
production mix 10% (hypothetical)

With an unsupported answer costing 3 times a refusal, the cheapest rule on the test mix answers only scores of at least 0.53; at 10% unanswerable traffic it answers from 0.37. The rule depends on both costs and the mix.

Computed from the application's toy run file: ten answerable and six unanswerable top-1 scores. The 3:1 cost ratio and the 10% production mix are hypothetical inputs. Sixteen queries make every rate move in steps of a tenth or a sixth.

Left, the unsupported share of served answers as the traffic mix changes, for three thresholds. Right, expected cost against the threshold at the test mix and at a 10% mix. Cost is a step function: any cut in (0.52, 0.55] costs the same at the test mix, any cut in (0.36, 0.39] at a 10% mix.

Check 1: the evaluation set3 minFive properties that make a golden set worth trusting, and how to write unanswerable queries that test something.

Check 1: is the evaluation set trustworthy? Sequence

Labels before runs
Governing passages written down before any retrieval run.
Frozen held-out file
Hashed, and opened only for the final claim.
Overlap check
Query wording copied from the evidence text inflates lexical scores.
Label audit
A second reading of a sample; record the disagreement rate.
Counts per type
Answerable and unanswerable, per jurisdiction and query type.

Writing unanswerable queries that test something Overview

Near the corpus
Same topic and vocabulary, but not governed by it: a dental claim against a home-insurance policy corpus.
Not off-topic
A question about cooking is rejected trivially and overstates abstention quality.
Checked against the snapshot
An unanswerable query the corpus does answer inflates the score.

Check 2: the result and its interval4 minRepeated looks, paired comparisons and why equal gains can be clear or inconclusive.

Check 2: what kind of number is the held-out result? Diagram

Held-out used to tune?
- yes: to Development-grade
no
Paired CI excludes 0?
- no: to Inconclusive
yes
A measured gainstill retrieval-only
Development-gradereport every run; need a fresh set
Inconclusivenot no difference

A paired interval is driven by the few queries that change outcome between the two retrievers.

A paired comparison is decided by the queries that flip Illustration

Synthetic outcomes: 60 queries, the old retriever hits 42; 2,000 seeded resamples of the per-query difference. Green: interval excludes zero.

Same net gain of 4 hits in every row (recall +0.067). The more queries flip in both directions, the wider the paired interval; analysing the runs as unpaired throws the pairing away and widens it too.

Equal net gains can be clear or inconclusive, depending on how many queries changed outcome in each direction.

Standard error of a paired recall difference Formula

SE(\hat\Delta) = \frac{1}{\sqrt{n}}\sqrt{\frac{b + c}{n} - \left(\frac{b - c}{n}\right)^{2}}

$b$: queries only the new retriever hits
$c$: queries only the old retriever hits
$n$: queries scored by both; agreements (hit-hit, miss-miss) add nothing but n
$\hat\Delta = (b - c)/n$: the recall difference; the same b - c is tighter when b + c is small

The per-query difference takes values +1, -1 or 0; this is its sample standard error. McNemar's test uses the same two counts.

How wide is a 95% interval on recall near 0.80? (interval width) Chart

10 queries (toy bootstrap)0.50 to 1.000.5
40 queriesabout 0.68 to 0.920.25
100 queriesabout 0.72 to 0.880.16
400 queriesabout 0.76 to 0.840.08

The 10-query row is the application's bootstrap; the others use the normal approximation 2 x 1.96 x sqrt(0.8 x 0.2 / n). The width is a finding, not a defect to hide.

How many queries are enoughadvanced3 minPlan the size of the query set from the precision a decision needs, before labelling.

How many queries for a given precision? (p = 0.80, 95%) Table

Target half-width	Queries needed	A measured 0.80 then rules out
0.10	62	True recall below about 0.70
0.05	246	True recall below about 0.75
0.025	984	True recall below about 0.775

n = 1.96^2 x p(1 - p) / h^2. A paired comparison needs fewer queries when the retrievers mostly agree. Size the unanswerable slice too: on 6 queries each one moves the rate by a sixth, and 4 of 6 has a 95% Wilson interval of 0.30 to 0.90. Plan both before labelling (Miller, 2024).

Check 3: setting and confirming the threshold3 minThe order of work that keeps the held-out part unspent while the threshold is chosen, and the loop that re-tunes it after a pipeline change.

Setting the abstention threshold without spending the held-out part Sequence

Write the gates first
Correct abstention, false abstention and the blocking counts, before any run.
Tune on development queries
Sweep the threshold on development answerable and unanswerable queries only.
No feasible threshold?
Change the score (reranker, metadata filter, better unanswerables), not the gates.
Confirm once on held-out
One run, every gate reported, including the ones that fail.

A threshold belongs to one score scale: re-tune when the scale moves Loop

Threshold seton dev queries, confirmed once on held-outserve ↓
Daily abstentionthe live diagnostic to watcha pipeline change ↓
Pipeline changenew embedding, chunking or corpusscale shifts ↓
Rate jumpsthe old threshold fails silentlyre-tune ↓
Re-run dev sweepnever on the held-out partconfirm once ↓
↺ back to 1 · Threshold set

A threshold tuned on one score scale fails silently when the embedding model, the chunking or the corpus changes, because the same score no longer means the same posterior.

Check 4: provenance, generation mode and reading answersadvanced4 minThe manifest, the model record, and a five-way classification of read answers with the repair each outcome points to.

Check 4: provenance and generation mode Overview

Corpus manifest
Source, retrieval date, checksum and licence per document.
Generation mode
Model identifier and date, or the sentence: generated-answer competence not assessed.
Answers read
At least 30 sampled answers read against their citations, with counts per failure type.
Memorisation check
Correct answers without the governing passage in context are not credited to retrieval.

Reading generated answers: five outcomes, and the repair each points to Hierarchy

One read answeragainst its cited passages
- correctCorrect answer
  - passage in contextCredit the pipelinekeep as a regression case
  - passage not in contextPossibly memorisedno credit; fails when the rule changes
- wrong or unsupportedFailed answer
  - passage was in contextGeneration failureprompt, context order, citation check
  - wrong jurisdiction/staleBlocking failuremetadata filter, corpus versioning
  - adds unsupported claimsUnsupported claimquote-span check, tighter format

Count each leaf on a sample of at least 30 read answers. Report counts with the sample size, not a percentage alone.

Choosing the retrieval metricadvanced3 minWhat recall, MRR and nDCG reward, and the two knobs of the lexical baseline.

Recall@k, MRR, nDCG: what each rewards Comparison

Criterion	Recall@k	MRR	nDCG@k
Counts	Share of relevant passages in the top k	1 / rank of the first relevant passage	Discounted gain of graded relevance
Rewards	Getting the evidence into the context at all	Putting one right passage first	Good order of several relevant passages
Blind to	Position inside the top k	Every relevant passage after the first	Nothing, if the grades are wrong
Use it when	The generator reads all k passages	One governing passage; the top-1 is shown or cited	Graded labels exist and order matters

BM25 knobs for the lexical baseline: k1 (term saturation) and b (length normalisation). Course code uses 1.5 and 0.75; Elasticsearch defaults to 1.2 and 0.75.

What a staff-level reviewer asks2 minFive readings of a retrieval report and the question that follows each.

Five readings of a retrieval report, and the question that follows each Overview

Recall rose after a chunking change
Did k passages now cover more or less text? Compare at equal context tokens.
The held-out set is frozen
Is the corpus? A changed snapshot can make labelled passages stale.
Abstention passes its gate
On unanswerable queries near the corpus, or on easy off-topic ones?
An LLM judge scored the answers
Against how many human-read answers, with what agreement?
The interval excludes zero
Paired on the same queries, and how many queries flipped?

Memory anchor

Retrieval shows the evidence was there; only read answers show it was used - label every number.

Useful for

Reviewing a RAG or document-search report before it reaches a product owner, from an insurer's policy assistant to a SaaS support-docs bot.
Deciding which numbers in a results table may be called answer quality and which are retrieval-only.
Writing a limitations paragraph that tells the reader what was not assessed.

Not suitable for

Establishing that generated answers are correct; that needs a configured model and read, cited answers.
Legal, medical or policy conclusions; the system ranks paragraphs for a trained reader (a claims handler, a clinician, an agent) and does not determine entitlement.

Traps

Averaging away a blocking failure

A recall of 0.85 sits next to one answer that promises compensation under the wrong jurisdiction. The average is reported and the single failure is not, although it is the one that costs money.

How to avoid it

Define safety gates before the run and report their counts separately from averages.

Scoring only answerable queries

The report shows recall and MRR on answerable queries and never tests what happens when the corpus has no answer, so the assistant's confidence on unanswerable questions is unknown.

How to avoid it

Keep a labelled unanswerable slice and report abstention on both answerable and unanswerable queries.

Senior reasoning

A senior reads a retrieval report by asking what each number is allowed to mean. They attach the evidence label, ask how many times the held-out part was opened, demand the unanswerable slice and the gates that averages cannot hide, and write the generation claim last and only if it was measured. They are comfortable saying "not assessed" in a final report, because it tells the reader exactly where the evidence stops.

Interview question

"Your retrieval recall looks good. Why is that not enough to launch an answer-generating assistant?" How would you structure the answer?

Outline your answer first.

Recall cards scheduled after completion

What are the four evidence labels for a number in a retrieval report?
How is unsupported-answer behaviour scored, and why on both query groups?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis et al. (12 authors), 2020core
Retrieval-augmented generation as a retriever plus a generator, which is why the two components need separate measurements.
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, Iryna Gurevych, 2021core
Across 18 datasets BM25 is a robust zero-shot baseline, so a retrieval improvement has to be shown against it on your own labels.
Survey of Hallucination in Natural Language Generation — Ziwei Ji et al., 2023optional
Taxonomy of unsupported (hallucinated) generated content and why fluent output needs verification.
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations — Evan Miller, 2024optional
Report standard errors, compare models with paired question-level differences and use power analysis for the number of questions.
Similarity settings (Elasticsearch Reference) — Elasticoptional
BM25 similarity settings (k1, b) of a production search engine, the knobs of the lexical baseline in the metric chapter.
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets — Patrick Lewis, Pontus Stenetorp, Sebastian Riedel, 2021optional
Near-duplicate overlap between test and training questions inflates scores, which motivates the overlap check on your own queries.

Optional sections

Deep dive · 10 min

Why the interval is wide. For a proportion estimated on $n$ queries, the standard error is about $\sqrt{p(1-p)/n}$ . At $p = 0.8$ and $n = 40$ that is about 0.063, so a 95% interval is roughly 0.68 to 0.92. A paired difference between two retrievers can be tighter than two separate intervals because most queries behave the same under both, but it is still driven by the few queries that change outcome (three in the example above).

Repeated looks at the held-out part. Every run that informs a decision adds a selection step. If you run the held-out set four times and choose the configuration with the best result, the reported number is optimistic in the same way as a tuned test score. Report every run, or keep a second, untouched part for the final claim.

Why unanswerable queries are expensive to write. A good unanswerable query is near the corpus in topic and vocabulary but not governed by it, such as a dental claim put to a home-insurance policy corpus. Random off-topic questions are trivially rejected and overstate abstention quality.

Interview practice · 10 min

Your assistant cites the correct passage in the top five for 85% of queries, yet agents distrust it. How would you find out what is going wrong?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-18 · depends on numpy

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 39 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min