Core day 48 of 60 · Week 10 · Part II · application depth

Retrieval-Augmented Generation

The airline's fare-rule assistant answers baggage questions from a folder of policy documents. On one test question the right passage is in the top three results, yet the answer quotes last year's 20 kg allowance. Which stage failed, and how do you find out without reading answers by hand?

By the end: For a labelled query you trace ingestion, chunking, retrieval, context assembly and generation, compute recall@k for a lexical baseline, and say whether a failure is a retrieval miss, a context problem or an ungrounded answer.

At a glance

RAG retrieves evidence at request time; measure retrieval and grounding separately, because a retrieved passage is not yet a supported answer.

Evidence on demand
Search the corpus, pass a few chunks, answer from them with citations.
Two failure sites
Retrieval can miss; generation can ignore, blend or invent with evidence present.
Metadata first
Status and validity filters keep superseded rules out of the context.
Lexical plus vector
BM25 wins on codes, vectors on paraphrase; fuse by rank.
Measure each stage
Recall@k at the production k, then a retrieval-by-answer table.

Recall · 5 min

If no cards are due, answer from memory: why must text inside an email or a retrieved passage be treated as data rather than as instructions, and which three checks in code enforce that?

Open due reviews

Concept · 10 min

Weights hold language ability and a frozen snapshot of facts, not your policy folder or last week's change. Retrieval-augmented generation (RAG) puts evidence in front of the model at request time, so updating a document updates the assistant. Two things can fail: retrieval can miss the passage, and the model can mis-use a passage it was given. A hit is not a Grounded answer.

Chunk size, retrieval method and k change which passages reach the context

Predict first

A query shares no words with the passage that answers it. Which retrieval method has no chance of finding it?

Text description of the figure

A picker of 40 labelled queries, or a typed query, over the academy's own glossary, insight and lesson text, re-chunked at the chosen size. Controls pick chunk size, retrieval method (BM25, TF-IDF, LSA or hybrid RRF) and k, then list the ranked passages with scores, source records and the words shared with the query, and mark which ones are labelled relevant. The lab stops at the context a generator would receive; no answer is generated.

The retrieval half: each stage sees only what the last one passed on Diagram

Ingest and parsetext, source, date
by section
Chunk + metadataid, validity, audience
IndexBM25 and/or vectors
query
Filter + top-kstatus = current; RRF if hybrid
100 candidates
Rerankwider list, slower model
top k (3-5)
Assemble contextdedupe, cap tokens, label ids

Lewis et al. 2020 introduced the retriever-plus-generator pattern. A filter applied after ranking, or a k cut too early, removes evidence no later stage can recover.

The answer half: two checks before an answer is shown Diagram

Generateanswer only from the labelled chunks
Context supports it?
- yes: to Cited ids in context?
no
Abstainsay what is missing
Cited ids in context?
- yes: to Show answer
no
Reject or review
Show answer

On yes the model answers with chunk ids. The id check is code; whether the chunk really supports the sentence still needs sampled review. An audit of generative search engines found many citations that did not support their sentences (Liu et al. 2023).

The retrieval-by-answer table Map

Generation or context failure

Evidence present but ignored, blended with a stale chunk, or embellished.

Working

Retrieved and used; keep it in the regression suite.

Retrieval miss

Fix chunking, metadata, index or k before touching the prompt.

Right without evidence

From weights or luck; not grounded, will not survive a policy change.

Count your labelled queries into the four cells; the largest wrong cell names the stage to fix.

Lexical, vector and hybrid retrieval Comparison

Criterion	Lexical (BM25)	Vector	Hybrid (RRF)
Strong on	Codes: error codes, SKUs, fare bases, exact terms	Paraphrase and synonyms	Both kinds of query, if each list contains the gold
Weak on	Paraphrase with no shared word	Rare identifiers with no semantic neighbours	Cannot recover a chunk neither list returns
What you maintain	An inverted index	An embedding model version and a vector index	Two indexes and a fusion step

BM25 was a robust zero-shot baseline across BEIR's 18 datasets, where dense retrievers often fell behind out of domain (Thakur 2021).

Recall@k for BM25, LSA and their fusion on the academy's own corpus Illustration

Queries that reuse the corpus wording (n = 28): the methods are close; at k = 5, BM25 0.73, LSA 0.76, hybrid 0.79.

Paraphrases (n = 12): BM25 flattens; at k = 5, BM25 0.44, LSA 0.58, hybrid 0.57. LSA is weak at the very top (k = 1: 0.09), so fusion borrows BM25's head and LSA's tail.

BM25
LSA (rank 50)
Hybrid: BM25 + LSA, RRF

The academy's own text (glossary, insights, lesson concepts; 567 documents in 120-word chunks) and its 40 labelled queries, scored with the retrieval widgets' code. Recall = share of a query's relevant documents in the top k, averaged. Twenty queries per style: the shapes are informative, small differences are not.

On queries that reuse the corpus wording the three methods are close. On paraphrases BM25 flattens, LSA is weak at the very top, and fusion takes some of each.

Stages: ingest -> Chunking with metadata -> index (BM25, vectors or both) -> filter and retrieve top-k -> rerank -> assemble context -> answer with citations -> validate.
Each stage has a symptom; measure each one. Recall@k uses the k you actually pass to the model.
Hybrid search merges rankings by position (Reciprocal rank fusion), not by raw scores.
Retrieved text is untrusted data (Day 47); cross "gold chunk in context" with "answer correct and supported" to locate failures.

Assumptions

Each query has a gold chunk label written independently of the chunk's wording; queries copied from the chunk inflate lexical scores.
Retrieved text is untrusted data. A passage that says "ignore your rules" is content and gets no authority (Day 47).
The corpus here is synthetic and tiny; five queries show mechanisms, not method rankings.

Application and workplace decision · 10 min

Six synthetic fare-rule chunks (one is last year's superseded baggage rule) and five labelled queries written without looking at the chunks.
The code ranks chunks with TF-IDF cosine and BM25, prints the gold chunk's rank (99 = not retrieved), recall@1 and recall@3, then inspects the baggage query's context at k = 2 before and after a status filter.
Predict first: which query fails, and why?

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
CHUNKS = {  # synthetic fare-rule chunks (not real policy): id -> (status, text)
    "c1": ("current", "Baggage: international economy fares include one checked bag up to 23 kg; domestic economy 15 kg."),
    "c2": ("superseded", "Baggage: economy fares include one checked bag up to 20 kg on all routes (valid until 2023-12-31)."),
    "c3": ("current", "Changes: flex fares can be changed free of charge until 24 hours before departure; basic fares pay 60 EUR."),
    "c4": ("current", "Refunds: unused tickets are refunded within 14 days to the original form of payment."),
    "c5": ("current", "Pets: small pets up to 8 kg in an approved carrier may travel in the cabin on domestic routes."),
    "c6": ("current", "Seats: seat selection is free on flex fares; basic fares pay 12 EUR per seat."),
}
QUERIES = [("how many kg of checked baggage are included on economy", "c1"), ("can I change my flex ticket for free", "c3"),
           ("how do I get my money back", "c4"), ("can my dog travel with me", "c5"), ("is there a charge to pick where I sit", "c6")]
ids, texts = list(CHUNKS), [v[1] for v in CHUNKS.values()]
cv = CountVectorizer().fit(texts)
TF = cv.transform(texts).toarray()
idf = np.log(1 + (len(texts) - (TF > 0).sum(0) + 0.5) / ((TF > 0).sum(0) + 0.5))
dl = TF.sum(1, keepdims=True)
def bm25(q, k1=1.5, b=0.75):
    w = TF * (k1 + 1) / (TF + k1 * (1 - b + b * dl / dl.mean()))
    return (w * idf * (cv.transform([q]).toarray()[0] > 0)).sum(1)
tv = TfidfVectorizer().fit(texts)
M = tv.transform(texts)
def tfidf_cosine(q):
    return (M @ tv.transform([q]).T).toarray().ravel()
def rank(scores, gold):
    order = [j for j in np.argsort(-scores) if scores[j] > 0]  # chunks with no term match are not retrieved
    return order.index(gold) + 1 if gold in order else 99
for name, fn in [("TF-IDF cosine", tfidf_cosine), ("BM25", bm25)]:
    r = [rank(fn(q), ids.index(g)) for q, g in QUERIES]
    print(f"{name:<14} gold ranks {r}  recall@1 {np.mean([x <= 1 for x in r]):.2f}  recall@3 {np.mean([x <= 3 for x in r]):.2f}")
s = bm25(QUERIES[0][0])
top = [ids[j] for j in np.argsort(-s)[:2]]
print("k=2 context:", top, [CHUNKS[i][0] for i in top])
print("after status filter:", [ids[j] for j in np.argsort(-s) if CHUNKS[ids[j]][0] == "current"][:2])

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 37 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Retrieval as ranking: the objects and the metrics7 minWhat a retriever computes, the three standard ranking metrics defined exactly, and all three worked by hand on a toy ranking.

What a retriever computes, and what a metric compares it with Diagram

Query q
- to Score s(q, d)
Chunks Dthe whole corpus
Score s(q, d)for every chunk
sort, cut at k
Top k by sall later stages see only these
- ranking: to Metric
Relevance labelsgold: binary or 2/1/0
relevance
Metricranking compared with labels

Relevance is a label, not a score: a metric is only as good as its labels and is blind to chunks nobody labelled.

Three questions about one ranking, three metrics Hierarchy

A ranked list and its labels
- did the gold arrive?recall@kread it at the production k
- how high is it?MRRtop position dominates; ignores later relevant chunks
- is the top k ordered?nDCG@kneeds graded labels; depends on the gain convention

Each is a mean over labelled queries; the equations and two hand computations follow.

Recall at k Formula

\mathrm{recall@}k = \frac{1}{|Q|} \sum_{q \in Q} \mathbb{1}[\mathrm{rank}(g_q) \le k]

$g_q$: gold chunk labelled for query q
$k$: the number of chunks you actually pass to the model; recall@k never falls as k grows
$|Q|$: labelled queries; the standard error of the mean shrinks like 1 / sqrt(|Q|), so five queries give a hint, not a ranking (Day 50)

Recall at a larger k than production flatters the retriever; recall at the reranker's candidate depth tells you what reranking could recover.

Mean reciprocal rank Formula

\mathrm{MRR} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q}

$\mathrm{rank}_q$: position of the first relevant chunk for query q
$1/\mathrm{rank}_q$: 1 at the top, 0.5 at position 2, and 0 when nothing relevant is retrieved

Recall@k asks whether the gold arrived; MRR also rewards how high it sits, which matters when the model reads the first chunk most.

Normalised discounted cumulative gain at k Formula

\mathrm{nDCG@}k = \frac{1}{\mathrm{IDCG@}k} \sum_{i=1}^{k} \frac{rel_i}{\log_2(i+1)}

$rel_i$: graded label of the chunk at position i (here 2 = answers it, 1 = partly relevant, 0 = not)
$\log_2(i+1)$: position discount: 1 at the top, 0.63 at position 2, 0.5 at position 3
$\mathrm{IDCG@}k$: the same sum for the ideal ordering of the same labels, so a perfect ranking scores 1

Linear gain, as scikit-learn's ndcg_score uses; many papers use 2^rel - 1, which weights the top grade more. State which one you report. Day 51 derives the discount.

Recall@k and MRR by hand: five queries on a hospital's guideline search (synthetic) Table

Query	Rank of first relevant chunk	In top 3?	Reciprocal rank
Sepsis antibiotic timing	1	yes	1.00
Paediatric paracetamol dose	2	yes	0.50
Anticoagulant before surgery	4	no	0.25
Hand-hygiene audit form	1	yes	1.00
"Blood thinner, tooth out"	not retrieved	no	0
Mean over 5 queries	-	recall@3 = 0.60 (recall@5 = 0.80)	MRR = 2.75 / 5 = 0.55

Verified in Python. The last query is a lay paraphrase of the third: different words, no shared term. Recall@5 rises to 0.80 only if five chunks really reach the model.

nDCG@4 by hand for one query with graded labels Table

Position i	Label	Discount 1/log2(i+1)	Contribution
1	1 (partly relevant)	1.000	1.000
2	2 (answers it)	0.631	1.262
3	0	0.500	0
4	2 (answers it)	0.431	0.861
DCG@4	-	-	3.123
Ideal order 2, 2, 1, 0	-	1.000, 0.631, 0.500, 0.431	2.000 + 1.262 + 0.500 = 3.762
nDCG@4	-	-	3.123 / 3.762 = 0.830

Same result as scikit-learn's ndcg_score (checked in 1.9.1). The ranking found both answering chunks but put a weaker one first and an answer last; nDCG prices exactly that.

BM25, derived term by term and computed on two documentsadvanced8 minWhere each factor of the BM25 formula comes from, one score computed by hand, the two knobs drawn, and where the model breaks.

BM25 score of chunk d for query q Formula

\sum_{t \in q} \mathrm{idf}(t) \cdot \frac{f_{t,d}\,(k_1+1)}{f_{t,d} + k_1 \left(1 - b + b\,\frac{|d|}{\overline{|d|}}\right)}

$\sum_{t \in q}$: probability ranking principle + binary independence: the log-odds of relevance splits into one weight per matching term
$\mathrm{idf}(t)$: rarity. With no labels the weight is log((N - n + 0.5)/(n + 0.5)), negative for terms in over half the docs; the code, like Lucene, uses ln(1 + that ratio)
$f_{t,d}$: count of term t in chunk d
$\frac{f(k_1+1)}{f+k_1}$: replaces the 2-Poisson eliteness model: evidence rises with the count but levels off; exactly 1 at f = 1, never above k1 + 1
$k_1$: saturation; the 10th repeat adds little (1.5 in the code)
$1 - b + b\,|d|/\overline{|d|}$: length: long chunks repeat words by verbosity, not topic. b = 0 ignores length, b = 1 normalises fully (0.75 in the code)

Robertson and Zaragoza 2009. Guarantee: each term contributes at most (k1 + 1) x idf(t), so repeating one word cannot dominate a score.

BM25 by hand: two telecom support articles, query "reset router password" Table

Term	idf (N = 2)	Doc 1, 8 words	Doc 2, 12 words
Length factor K = k1(1 - b + b\|d\|/avg)	-	1.5 x (0.25 + 0.75 x 0.8) = 1.275	1.5 x (0.25 + 0.75 x 1.2) = 1.725
reset	ln(1 + 1.5/1.5) = 0.693	f = 2: 2 x 2.5 / 3.275 = 1.527; x 0.693 = 1.058	f = 0: 0
router	ln(1 + 0.5/2.5) = 0.182	f = 1: 2.5 / 2.275 = 1.099; x 0.182 = 0.200	f = 1: 2.5 / 2.725 = 0.917; x 0.182 = 0.167
password	0.693	f = 0: 0	f = 1: 0.917 x 0.693 = 0.636
Score	-	1.259 (rank 1)	0.803 (rank 2)
What it shows	router is in both: worth almost nothing (rarity)	the 2nd reset adds 0.43 to the first's 1.10 (saturation)	single matches earn 0.917, not 1.099: longer (length)

Doc 1: "Reset the router: hold reset for ten seconds". Doc 2: "Change the wifi password in the app; the router keeps its settings". Average length 10, k1 = 1.5, b = 0.75 and the idf of the application code; verified in Python with CountVectorizer tokens.

Why Doc 1 outranks Doc 2, term by term (the hand computation above) Bridge

Doc 2 score0.8032
reset: twice in Doc 1, absent in 2rare term; second occurrence saturates+1.0582
router: in bothlow idf; Doc 1 shorter, so slightly more+0.0331
password: only in Doc 2a rare term Doc 1 lacks−0.6359
Doc 1 score (1.259 rounded)1.2586

Each step is Doc 1's contribution minus Doc 2's for one query term, to four decimals. Rarity decides most of the gap: the rare terms move the score by about 0.6 to 1.1, the common one by 0.03.

The two knobs inside BM25 Illustration

With k1 = 1.5, the 2nd occurrence adds 0.43, the 10th adds 0.031; ten occurrences weigh 2.2x one, never more than k1 + 1 = 2.5. A chunk that repeats a word cannot buy its way to the top.

b decides how much a long chunk is penalised. With b = 0.75, one occurrence in a chunk twice the average length weighs 0.69 instead of 1.00; with b = 0 length is ignored, which favours long chunks that mention everything.

Computed from the BM25 term weight with the constants of the academy's retrieval code (k1 = 1.5, b = 0.75); the idf factor multiplies every curve and is left out. Robertson and Zaragoza (2009) describe the function.

k1 caps what repeating a term can earn; b decides how strongly a long chunk is discounted. Both curves use the constants of the academy's retrieval code.

Where BM25 breaks, and how you would see it Overview

Vocabulary mismatch
A query term absent from the chunk adds exactly zero, so synonyms score nothing. Watch paraphrases and zero-hit queries.
Bag of words
Word order is ignored; only which terms match, and how often, counts.
Scores drift with the corpus
idf changes whenever documents change, so raw scores are not comparable across index builds.

Fusion by rank, computed by handadvanced5 minWhy ranks fuse and scores do not, one fusion computed by hand, a win and a loss of fusion on real queries, and what the slice table can show.

Reciprocal rank fusion Formula

\mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{c + \mathrm{rank}_r(d)}

$R$: the rankings being fused (BM25, vector)
$c$: constant; 60 in the lab, a common default
$\mathrm{rank}_r(d)$: position of d in ranking r: one scale for every retriever

A BM25 score is unbounded and corpus-dependent, a cosine lies in [-1, 1]; adding them weights whichever has the larger spread. The price of using positions: confidence is discarded, so a crushing hit and a marginal one at the same position count the same.

Reciprocal rank fusion by hand: an online shop's search for "waterproof hiking boots" Table

Product	BM25 rank	Vector rank	Terms, c = 60	RRF score	Fused rank
A	1	2	1/61 + 1/62	0.03252	1
C	3	1	1/63 + 1/61	0.03227	2
B	2	4	1/62 + 1/64	0.03175	3
E	not listed	3	1/63	0.01587	4
D	4	not listed	1/64	0.01563	5

Hypothetical lists, verified in Python. With c = 60 the step from rank 1 to rank 4 inside one list (0.0164 to 0.0156) is tiny next to appearing in a second list at all, so fusion rewards agreement. Raw BM25 and cosine scores never enter.

Fusion by rank, on two real queries Illustration

Fusion beats both lists: a passage that both rank moderately collects two shares of 1 / (60 + rank) and overtakes passages that only one list ranks highly. First relevant passage: BM25 rank 8, LSA rank 15, fused rank 4.

Fusion can also lose to the better list: when one retriever is right and the other is confidently wrong, the wrong list's passages dilute the right one's. First relevant passage: BM25 rank 48, LSA rank 1, fused rank 11.

passage from a document labelled relevant
line: where a fused passage came from

Two of the 40 labelled queries of the academy corpus, chosen by code as the query where fusion gains most over the better single list and the one where it loses most; on most queries all three put a relevant passage first. 120-word chunks, the retrieval widgets' code.

Reciprocal rank fusion rewards agreement between lists. It can beat both inputs, and it can also fall behind the better one when the other list is confidently wrong.

Recall@3 by query type (optional lab, 14 chunks, 12 queries) Grid

Query type →
Method ↓	Identifier	Keyword	Paraphrase	All
BM25	1.00	1.00	0.17	0.58
LSA	1.00	1.00	0.33	0.67
RRF(BM25+LSA)	1.00	1.00	0.33	0.67

Six paraphrase queries cannot establish that one method is better; they show where to collect more labels.

Stage by stage, symptom by symptom4 minWhat each stage looks like when it fails, the check that exposes it, and which stage binds in different sectors.

Stage, symptom, check Table

Stage	Symptom when it fails	Check
Ingest and parse	Evidence absent or garbled (flattened table)	Spot-check parsed text against the source
Chunk	Rule found without its exception	Chunk along headings; keep section ids
Index and retrieve	Gold chunk missing from top-k	Recall@k per query type
Rerank	Gold at rank 12, never in context	Recall at the candidate depth vs final k
Assemble	Stale and current versions both present	Status filter; duplicate count in context
Generate and validate	Claim not in any cited chunk	Id check in code; sampled support review

One pipeline, a different binding constraint per sector (hypothetical) Overview

Telecom support articles
Error codes and model numbers favour BM25; "my internet keeps dropping" needs a vector or hybrid index.
Hospital guidelines
Versions are a safety issue: the validity filter is a clinical control; audience splits clinician and patient text.
Bank procedures
Entitlements filter inside the retriever: staff-only procedures never reach a customer answer.
Retail catalogue
Price and stock change hourly: filter on metadata at query time, and send "how many under 50 EUR" to SQL.
HR policies
One rule per country: without jurisdiction metadata the answer cites the wrong country's leave policy.

The stages and the metrics do not change between sectors; which failure costs most, and therefore which metadata is mandatory, does.

The application's run, and grounding2 minThe five-query result of the application code, as a grid of method by cutoff.

Five labelled queries, two lexical retrievers (application output) Grid

Cutoff →
Retriever ↓	recall@1	recall@3
TF-IDF cosine	0.60	0.80
BM25	0.80	0.80

The miss at k = 3 is the same query for both; five queries cannot rank methods.

Chunking and metadata4 minThe fields every chunk needs, and chunk size measured as a recall-for-words trade on real labelled queries.

Metadata every chunk should carry Overview

Document and section id
Lets you cite, group and score by source.
Validity dates and status
Current versus superseded; filter before ranking.
Audience
Staff-only procedures never reach a customer answer.
Parent context
Heading path so a rule keeps its exception nearby.

Chunk size trades recall against the words you pay for Illustration

60-word chunks
120-word chunks
Whole documents

Each dot is one more passage (k = 1, 2, 3 ...). Big chunks reach a given recall with fewer passages but many more words; the left edge of the chart is what a tight context budget can afford.

The same chunkings read two ways. At a fixed k the biggest chunks look best; at a fixed word budget the ranking changes.

BM25 on the academy corpus (567 documents) with the widgets' chunker (25% overlap) and the 40 labelled queries. Relevance is labelled per document, so a passage counts when its document is relevant; real chunking choices also depend on whether a passage alone answers the question.

Compared at the same k, whole documents look best because each passage carries more text. Compared at the same word budget, small chunks reach more relevant documents.

Prefixing chunks with document context halved retrieval failures (vendor-reported) Chart

Standard chunks5.7 %
Contextual chunks + contextual BM252.9 %

Anthropic 2024, on the vendor's own tests: each chunk prefixed with a short generated description of its document before indexing. A hypothesis to test on your golden set, not a result.

Reranking and its ceilingadvanced2 minWhat a second-stage reranker can and cannot recover.

Two-stage retrieval, bottom to top Stack

First stageBM25, vectors or both return a wide candidate list (tens to a hundred) cheaply.
RerankerA cross-encoder reads query and passage together and rescores only the candidates (Nogueira and Cho 2019).
Final kThe top few reranked passages go into the context.
CeilingRecall at the candidate depth is the most a reranker can reach; it cannot add a passage the first stage missed.

Report recall at the candidate depth and at the final k side by side; the gap is what reranking can win, at the cost of its latency.

Assembling the context2 minFive steps between the ranked list and the prompt.

Assembling the context Sequence

Filter
Status, validity and audience, before ranking.
Deduplicate
Near-copies crowd out distinct evidence.
Cut weak chunks
A relevance threshold drops padding.
Order and cap
Fit the token budget; long contexts bury the middle (Liu 2024).
Label
Each chunk gets an id and is marked as untrusted data.

Knobs and monitoring3 minEvery retrieval knob with its value in the academy code and its trade-off, and the signals to watch once it runs.

Retrieval knobs, their values in the academy code, and what each trades Table

Knob	Value here	Trade-off
Chunk size and overlap	120 words, 25% overlap	Smaller: precise passages, more of them; larger: context per hit, fewer distinct sources
k passed to the model	3 to 5	Recall against tokens, cost and distraction from weak passages
Candidate depth for fusion or rerank	100	Ceiling for later stages against compute
BM25 k1 and b	1.5 and 0.75	Term saturation and length discount
RRF constant c	60	Smaller c: top ranks dominate; larger: flatter, more agreement-driven
LSA rank	50	Higher: finer topics, closer to exact words; lower: broader matches
TfidfVectorizer token pattern	2+ word characters	scikit-learn drops one-character tokens by default, such as a lone digit

Values from src/lib/stats/w6-ir.ts and the retrieval widgets; the scikit-learn default checked in version 1.9.1.

What to watch in production (practitioner heuristic) Overview

Golden-set recall per index build
Re-score recall@k at the production k whenever documents, chunking or the embedding model change.
Stale evidence in context
Share of contexts holding a superseded or expired chunk; it should be zero.
Citation validity
Share of answers whose cited ids were in the context, plus a sampled support review.
Abstentions and empty results
Rising abstention or zero-hit queries point to a content gap or a broken index.
Latency per stage
Retrieval, rerank and generation timed separately so a slow stage is visible.

Memory anchor

Retrieval success is not a grounded answer: score the stages separately, at the k you actually use.

Useful for

Giving an LLM access to documents that change, are private, or are too large for the prompt, without retraining: clinical guidelines (healthcare), product manuals and error codes (telecom, manufacturing), internal procedures (banking, HR).
Debugging an assistant by stage (ingestion, chunking, retrieval, context, generation) with labelled queries instead of opinions.
Deciding whether a lexical baseline is already enough before adding a vector index.

Not suitable for

Questions that need counting or aggregation over many records; route those to SQL (insight 0628).
Teaching a model a new behaviour or format; that is a prompting or fine-tuning question (Day 43).
Concluding which retriever is best from five queries; the run shows mechanisms and needs a much larger labelled set.

Traps

Counting a retrieval hit as a correct answer

The gold chunk is in the context, so the dashboard shows success, but a stale or conflicting chunk sits next to it and the answer quotes the wrong one. Complaints continue while retrieval metrics look healthy.

How to avoid it

Label evidence and answers separately, count the four cells of the two-way table, filter by status and validity dates, and check cited ids and support.

Writing queries from the chunks

The person labelling reads a chunk and writes a question using its words. Lexical retrieval looks excellent and the first paraphrased real query fails: a patient asks about a "blood thinner" and the guideline says "anticoagulant", or a customer writes "internet keeps dropping" for an article titled "intermittent WAN loss".

How to avoid it

Collect queries from real traffic or from people who have not seen the chunks, tag each by type (identifier, keyword, paraphrase) and report recall by slice.

Senior reasoning

A senior treats RAG as a set of separately measurable components, not as one prompt. They start with a lexical baseline because it is cheap, strong on identifiers and a yardstick, add vectors where a labelled paraphrase slice fails, filter on metadata in the retriever, and refuse to read an answer-accuracy number before they know which cell of the retrieval-by-answer table it came from. They ask which failure is most expensive in this sector (a superseded clinical guideline, a staff procedure shown to a customer, yesterday's price) and make the metadata that prevents it mandatory.

Interview question

"Your RAG assistant gives wrong answers on 20% of questions. Walk me through finding out why."

Outline your answer first.

Recall cards scheduled after completion

List the RAG pipeline stages in order and give one failure symptom for each.
Why can retrieval succeed and the answer still be ungrounded, and how do you detect which happened?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis et al. (12 authors), 2020core
Introduces retrieval-augmented generation as a retriever feeding a generator over non-parametric memory, the pattern used throughout today.
The Probabilistic Relevance Framework: BM25 and Beyond — Stephen Robertson, Hugo Zaragoza, 2009core
Authoritative account of BM25 (term-frequency saturation, rarity weighting, length normalisation) behind the lexical baseline.
Introduction to Information Retrieval — Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze, 2008optional
Defines precision at k and ranked-retrieval evaluation used for recall@k. (Chapter 8 (Evaluation in information retrieval))
BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, Iryna Gurevych, 2021optional
Reports BM25 as a robust zero-shot baseline across 18 datasets where dense retrievers often underperformed out of domain.
Reciprocal rank fusion outperforms condorcet and individual rank learning methods — Gordon V. Cormack, Charles L. A. Clarke, Stefan Buettcher, 2009optional
Reciprocal rank fusion merges rankings by position rather than by raw score, the fusion used in the hybrid lab.
Passage Re-ranking with BERT — Rodrigo Nogueira, Kyunghyun Cho, 2019optional
Cross-encoder reranking of first-stage candidates, the second stage in the two-stage retrieval figure.
Introducing Contextual Retrieval — Anthropic, 2024optional
Vendor-reported effect of prefixing chunks with generated context before indexing; cited as a hypothesis to test, not a result.
Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, 2024optional
Accuracy drops when the relevant passage sits in the middle of a long context, which is why order and count of chunks matter.
Evaluating Verifiability in Generative Search Engines — Nelson F. Liu, Tianyi Zhang, Percy Liang, 2023optional
Audit of generative search engines found citations that did not support their sentences; supports checking support beyond citation ids.
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, Mario Fritz, 2023optional
Instructions planted in retrieved content can hijack an LLM application; retrieved text must stay data.

Optional sections

Deep dive · 10 min

BM25 in one line. For a query term $t$ in document $d$ of length $|d|$ , with average length $\bar d$ : $\text{score} = \sum_t \text{idf}(t)\,\dfrac{\text{tf}\,(k_1+1)}{\text{tf} + k_1\,(1 - b + b\,|d|/\bar d)}$ . The fraction saturates in term frequency (a tenth occurrence adds little), $b$ controls length normalisation, and $\text{idf}$ weights rare terms. TF-IDF cosine instead normalises vectors and has no saturation; on short chunks the two often agree, which is why a five-query difference is not a finding.

Reciprocal rank fusion. Each retriever returns a ranking; a document's fused score is $\sum_r 1/(c + \text{rank}_r)$ with a constant $c$ (60 is a common default). It ignores score scales, so BM25 and cosine scores need no calibration. It also discards confidence: a very strong lexical hit and a marginal one at the same rank count the same.

Chunking. Fixed windows with overlap are a default for unstructured text. Structured documents (clauses, headings, tables) are better split on their structure with the heading path kept, and each chunk can be prefixed with a short context header before indexing (a vendor reported fewer retrieval failures on its own tests with this; treat it as a hypothesis for your corpus).

Two-way table. Rows: gold evidence in context (yes, no). Columns: answer correct and supported (yes, no). The cell "no evidence, answer correct" means the model answered from memory; it will fail on facts that changed after training.

Security. Retrieved text is attacker-controllable whenever someone else can write to the corpus. Apply permissions inside the retriever, label chunks as untrusted, and keep tool allow-lists independent of what passages say (Days 47 and 53).

Hands-on lab: Lexical, latent-semantic and fused retrieval by query type · 15 min

Fourteen synthetic chunks and twelve queries tagged identifier, keyword or paraphrase. The code compares BM25, a latent semantic (TruncatedSVD of TF-IDF) retriever and a reciprocal-rank fusion of the two, reporting recall@3 per slice. Before running, predict which slice a lexical method fails on and whether fusion can recover it. After running, write two sentences: what the slice table shows, and why six paraphrase queries and fourteen chunks cannot establish that latent semantic retrieval is better.

import numpy as np
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

TEXT = {  # synthetic fare-rule chunks (not real policy)
    "c01": "Baggage: international economy fares include one checked bag up to 23 kg; domestic economy 15 kg.",
    "c02": "Changes: flex fares can be changed free of charge until 24 hours before departure; basic fares pay 60 EUR.",
    "c03": "Refunds: unused tickets are refunded within 14 days to the original form of payment.",
    "c04": "Pets: small pets up to 8 kg in an approved carrier may travel in the cabin on domestic routes.",
    "c05": "Seats: seat selection is free on flex fares; basic fares pay 12 EUR per seat.",
    "c06": "Check-in: online check-in opens 48 hours before departure and closes 90 minutes before departure.",
    "c07": "No-show: a passenger who misses the outbound flight forfeits the return flight unless changed before departure.",
    "c08": "Fare basis YLOW26: lowest economy fare, no changes allowed, no refund.",
    "c09": "Fare basis YFLX26: flexible economy fare, free changes, refundable.",
    "c10": "Lounge: lounge access is included with business fares and for gold members.",
    "c11": "Delays: when a flight is delayed by more than three hours the airline offers meal vouchers at the airport.",
    "c12": "Assistance: passengers who need wheelchair help should request it at least 48 hours before departure.",
    "c13": "Infants: children under two travel on an adult's lap and pay 10 percent of the adult fare.",
    "c14": "Miles: gold members earn 50 percent bonus miles on every paid flight.",
}
QUERIES = [  # (query, gold id, type)
    ("what does fare basis YFLX26 allow", "c09", "identifier"), ("is YLOW26 refundable", "c08", "identifier"),
    ("how many kg of checked baggage are included on economy", "c01", "keyword"), ("what time does check-in close", "c06", "keyword"),
    ("can I change my flex ticket for free", "c02", "keyword"), ("how do I get my money back", "c03", "paraphrase"),
    ("can my dog travel with me", "c04", "paraphrase"), ("is there a charge to pick where I sit", "c05", "paraphrase"),
    ("my plane is late, will I get something to eat", "c11", "paraphrase"), ("do toddlers need their own ticket", "c13", "paraphrase"),
    ("who gets airport lounge access", "c10", "keyword"), ("what if I do not turn up for my outbound flight", "c07", "paraphrase"),
]
ids, docs = list(TEXT), list(TEXT.values())
cv = CountVectorizer().fit(docs)
TF = cv.transform(docs).toarray()
df = (TF > 0).sum(0)
idf = np.log(1 + (len(docs) - df + 0.5) / (df + 0.5))
dl = TF.sum(1, keepdims=True)
tv = TfidfVectorizer().fit(docs)
M = tv.transform(docs)
svd = TruncatedSVD(n_components=6, random_state=0).fit(M)
Z = svd.transform(M)
Z /= np.linalg.norm(Z, axis=1, keepdims=True) + 1e-9

def bm25(q, k1=1.5, b=0.75):
    return ((TF * (k1 + 1) / (TF + k1 * (1 - b + b * dl / dl.mean()))) * idf * (cv.transform([q]).toarray()[0] > 0)).sum(1)

def lsa(q):
    v = svd.transform(tv.transform([q]))[0]
    return Z @ v / (np.linalg.norm(v) + 1e-9)

def ranking(scores):
    return [j for j in np.argsort(-scores) if scores[j] > 1e-9]

def rrf(q, c=60):
    fused = np.zeros(len(ids))
    for fn in (bm25, lsa):
        for pos, j in enumerate(ranking(fn(q)), 1):
            fused[j] += 1 / (c + pos)
    return fused

methods = {"BM25": bm25, "LSA": lsa, "RRF(BM25+LSA)": rrf}
print(f"{'method':<15}" + "".join(f"{t:>12}" for t in ["identifier", "keyword", "paraphrase", "all"]) + "   (recall@3)")
for name, fn in methods.items():
    row = {"identifier": [], "keyword": [], "paraphrase": []}
    for q, gold, kind in QUERIES:
        row[kind].append(ids.index(gold) in ranking(fn(q))[:3])
    allhits = sum(row.values(), [])
    print(f"{name:<15}" + "".join(f"{np.mean(row[k]):12.2f}" for k in row) + f"{np.mean(allhits):12.2f}")

Interview practice · 10 min

A hospital's digital team wants to "put all our clinical guidelines into a vector database and let the model answer staff questions". What do you ask and what do you build first?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-12 · depends on scikit-learn, numpy