Skip to content
ML Academy

project-037 · LLM and GenAI · cross-industry

Hybrid lexical/latent retrieval and lightweight reranking ablation over the academy's own knowledge base

The academy wants a search box across its glossary, insights and lessons so a learner who types "why did my validation score drop after I scaled the whole dataset" lands on the right explanation. Which retrieval configuration should ship - BM25, TF-IDF, LSA, a hybrid of them, or a hybrid plus a small learned reranker - and does whole-document versus word-chunk indexing change the answer, measured on a labelled query set rather than on intuition?

Dataset
Academy knowledge base snapshot (glossary terms, insights, lesson core sections) plus a builder-authored labelled query set
Licence
Authored by the academy owner and bundled with the library; no third-party licence applies
Reference run
executed

At a glance

  • Data

    Academy knowledge base snapshot (glossary terms, insights, lesson core sections) plus a builder-authored labelled query set (bundled-with-library) · Recorded reference run: 567 documents (56 glossary terms, 500 insights, 11 lesson concept texts; 85,543 words); 40 labelled queries (16 dev / 24 held-out)

    Details on data

  • Baseline

    BM25 over record-level text (k1 = 1.2, b = 0.75), lowercase word tokenisation with a fixed stop-word list, implemented with scipy sparse matrices so no extra package is…

    Details on baseline

  • Validation

    Split queries about 40/60 dev/held-out (16/24 in the reference run), stratified by the kind of the best answer.

    Details on validation

  • Metric

    Recall@5 and @10, MRR@10, nDCG@10 (graded), per configuration and per chunking level, with paired bootstrap intervals against BM25; median and p95 query latency on CPU;

    Details on metric

Success criterion

Ship the simplest configuration whose held-out nDCG@10 is not worse than the best configuration by more than the paired-bootstrap 95% interval allows, provided recall@10 on held-out queries is at least 0.85 (target set in advance, not a known result). A reranker ships only if it improves held-out MRR with an interval that excludes zero and adds no more than 50 ms median latency on a laptop CPU. If no configuration meets the recall target, the deliverable is the failure analysis and a recommendation, not a launch.

Context

Search over a curated knowledge base is the most common first retrieval system a data team builds, and it is usually shipped on anecdotes. This project is the academy's own reference for retrieval evaluation mechanics: CPU-only, no model downloads, no API keys, deterministic. It deliberately stays inside lexical and latent (SVD-based) methods so it runs everywhere; a learner extension adds a neural embedding model and a cross-encoder, which require downloads and are therefore documented, not part of the reference run. The flagship (project-036) reuses these mechanics on a different, external corpus.

Data

Dataset
Academy knowledge base snapshot (glossary terms, insights, lesson core sections) plus a builder-authored labelled query set (bundled-with-library)
Licence
Authored by the academy owner and bundled with the library; no third-party licence applies
Size
Recorded reference run: 567 documents (56 glossary terms, 500 insights, 11 lesson concept texts; 85,543 words); 40 labelled queries (16 dev / 24 held-out)
Verification
unverified

Access

Read from a pinned snapshot of content/glossary, content/insights and the lesson concept texts stored with the reference implementation (data/retrieval_snapshot.json.gz); the run records the content hash and the document counts it found, because draft content changes between commits (--live evaluates the current files and records their hash). No login, no network. The labelled query set (each query mapped to one or more document ids; the reference run has 40 queries) is authored by the academy builder and stored with the reference implementation; it is labelled as builder-authored, and queries were written from learner-style questions before the retrieval run.

Grain

Retrieval unit: one record (glossary term, insight) or one lesson concept text (whole-document vs word-chunk indexing is the chunking ablation). Evaluation unit: one query with graded relevant ids (2 = best explanation, 1 = useful related record).

ColumnMeaning
doc_idRecord id (e.g. glossary slug, insight id) or lesson id plus section key
kindglossary, insight or lesson-section; used for slicing
dayCurriculum day of the record, for checking whether search favours early or late material
textConcatenated searchable fields (title/term, short definition or trigger/action, section prose)
query_id / queryBuilder-authored learner-style question
relevantdoc_id:grade pairs; queries with no relevant record are kept as no-answer cases
splitdev or heldout, stratified by the kind of the best answer

EDA and data-quality rules

  • Record the content commit hash and the counts per kind; fail the run if any labelled relevant id is missing from the snapshot (content drifted).

  • Drop or flag records whose searchable text is shorter than a minimum length (e.g. a glossary stub) and report how many.

  • Check duplicate or near-duplicate records (same term defined in two batches) because they split relevance credit.

  • Query/answer overlap check: flag queries whose word Jaccard with the best answer's title exceeds 0.5; keep paraphrased queries in the majority.

Baseline

BM25 over record-level text (k1 = 1.2, b = 0.75), lowercase word tokenisation with a fixed stop-word list, implemented with scipy sparse matrices so no extra package is needed. TF-IDF cosine (scikit-learn TfidfVectorizer, fitted on the corpus only) is reported alongside as a second lexical reference.

Features

Lexical: word unigrams, optional bigrams for TF-IDF, sublinear tf. Latent: LSA = truncated SVD of the TF-IDF matrix (component count chosen on dev from a small grid, e.g. 64/128/256). Hybrid: reciprocal rank fusion of BM25 and LSA rankings (k = 60) and, as a contrast, a min-max-normalised weighted score sum to show why score fusion needs calibration. Reranker features per (query, candidate): BM25 score, TF-IDF cosine, LSA cosine, title-term overlap, record kind. Chunking ablation: whole documents vs 240/120/60-word chunks scored at document level (the corpus has only a few lesson texts, so a lesson-section ablation is left to the learner).

Candidate methods

  • LSA (TF-IDF + truncated SVD) dense ranking

    Latent dimensions can match paraphrases that share no words; tests whether 'semantic' helps on this vocabulary before paying for a neural model.

  • Hybrid RRF of BM25 and LSA

    Lexical precision for exact terms (e.g. 'MASE', 'LoRA') plus latent recall for paraphrases; RRF needs no score calibration.

  • Hybrid + logistic-regression reranker over the top 30

    A tiny learning-to-rank model trained on dev queries with grouped cross-validation; shows reranking gains and the overfitting risk of learning from a few dozen queries (16 in the reference run).

Validation design

Split queries about 40/60 dev/held-out (16/24 in the reference run), stratified by the kind of the best answer. Grid choices (SVD components, fusion weights, reranker regularisation) use dev only; the reranker uses grouped cross-validation by query on dev so no query contributes candidates to both its training and validation folds. Held-out is evaluated once per final configuration. Paired bootstrap (1,000 resamples over queries, seed 7) for every comparison against BM25.

Leakage checks

Reranker features must be computed per query without looking at labels; grouped CV by query prevents a query's positives leaking into its own validation fold. Queries are written before retrieval runs, and the held-out file is hashed. The SVD and TF-IDF vocabularies are fitted on the corpus, never on queries.

Bias and representativeness

Records written later in the course may be longer and dominate BM25 length normalisation; slice results by record kind and day. Builder-authored queries share the authors' vocabulary, so real learner phrasing (typos, mixed languages) is under-represented; say so.

Robustness checks

Perturb ten queries (typos, synonym swaps, word-order changes) and report the rank change of the best answer; rerun with a second content commit to show sensitivity to content drift.

Technical metrics

Recall@5 and @10, MRR@10, nDCG@10 (graded), per configuration and per chunking level, with paired bootstrap intervals against BM25; median and p95 query latency on CPU; index build time and memory.

Business interpretation

Share of learner questions whose best explanation appears in the first five results (the search box shows five); share of no-answer queries that return a confident wrong record; latency within an interactive budget.

Error slices

  • Best-answer kind: glossary definitions vs insights vs lesson sections - short glossary records behave differently under BM25 length normalisation.

  • Query style: acronym or exact-term queries vs paraphrased problem descriptions.

  • Chunking level: questions answered by one lesson paragraph vs by a whole record.

Implementation steps

  • Snapshot loader: read the pinned content, build documents at whole-document and chunk level, write counts and the content hash to the run report.

  • Query set: load, validate ids against the snapshot, split, hash held-out.

  • Metrics module with hand-computed unit tests (recall@k, MRR, nDCG with graded labels, no-answer handling).

  • Rankers: BM25 (scipy sparse), TF-IDF cosine, LSA, RRF and weighted fusion; reranker with grouped CV.

  • Chunking ablation: rerun the best configurations on word chunks (and on lesson sections if you add them); compare with paired bootstrap.

  • Failure analysis: list the 15 worst held-out queries with retrieved vs labelled ids and a failure category.

  • Write the report (results table, intervals, latency, decision, limitations) and a small smoke test for CI.

Reproduction

Reference implementation at python/reference/r08_retrieval_eval (status: implemented; recorded results in its RESULTS.md). Command: cd python && uv run python -m reference.r08_retrieval_eval.run. Outputs in outputs/: metrics.json, per_query.csv, failures.csv and chunking_ablation.csv; dependencies are those already pinned in python/pyproject.toml (numpy, scipy, scikit-learn, pandas, pyyaml); the recorded run took about 11 seconds on a laptop CPU. The recorded numbers describe the pinned snapshot and 40 builder-authored queries only; your learner version should report its own.

Limitations

  • Forty builder-authored queries (24 held-out in the reference run) cannot represent real learner search behaviour; treat results as a method demonstration.

  • The corpus is small and in flux while content is drafted; conclusions are tied to the recorded commit.

  • LSA is a weak stand-in for neural embeddings; a finding that 'latent does not help' does not transfer to modern embedding models.

  • The reranker is trained on very few queries and may not generalise; its result is mainly a lesson about overfitting in learning-to-rank.

Production and monitoring

Rebuild the index on each content release and run the labelled query set as a regression gate. Log queries (without user identifiers) and clicks to grow the labelled set; clicks are biased by rank position, so use them to propose labels, not as labels. Monitor zero-result and no-click rates. Keep the BM25 path as the fallback if the latent index fails to build.

Interview questions

  • Why use reciprocal rank fusion instead of adding BM25 and cosine scores?

  • Your reranker improves dev MRR a lot and held-out MRR barely. What happened and how would you check?

  • When is section-level chunking worse than record-level chunking for search, and how would you detect it in your metrics?

Postmortem and alternatives

Discuss whether the decision depended on a handful of queries (inspect the bootstrap distribution, not only its interval), whether a field-weighted BM25 (title boost) would have matched the hybrid at lower complexity, and what a neural embedding extension would need to show to justify its download and serving cost.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r08_retrieval_eval.run

r08 results - retrieval evaluation over the academy's own text

Retrieval-evaluation mode: there is no generator. These results say how well rankers return the labelled documents. They say nothing about answer quality, groundedness or faithfulness, and this is not a RAG assistant. Numbers are copied from outputs/metrics.json, outputs/per_query.csv, outputs/failures.csv and outputs/chunking_ablation.csv, written by cd python && uv run python -m reference.r08_retrieval_eval.run.

ItemValue
Content snapshot (pinned)content_hash = 4a92a8556c029a52 (corpus and queries agree); copy in data/retrieval_snapshot.json.gz
Corpus567 documents: 56 glossary terms, 500 practical insights, 11 lesson concept texts; 85,543 words; median 152 words per document (min 40, max 478)
Queries40 labelled, binary relevance (1 to 5 relevant documents each; median 2); 28 keyword-style and 12 paraphrase-style; 16 dev / 24 held-out (held-out list hash 4fb2710bd7d4070f)
Chunker checkRe-derived 120-word chunk ids equal the corpus builder's list (1,113 chunks): yes
EnvironmentPython 3.12.13, numpy 2.5.3, scipy 1.18.1, scikit-learn 1.9.1
Runtime of the recorded run10.8 s wall, 9.0 s CPU, all configurations, chunking ablation, bootstrap and reranker

Content is still changing: if public/data/retrieval-corpus.json carries a different hash, these numbers describe the pinned copy, not the new content. run.py --live evaluates the current files and records their hash.

Data checks

All labelled ids exist in the snapshot. No document is shorter than 30 words (shortest: 40). No near-duplicate pair (TF-IDF cosine at least 0.9). Six queries (q10, q11, q21, q23, q25, q35) have a word Jaccard above 0.5 with the title of their first relevant document, all keyword-style; they are easy by construction. There is no query without a relevant document, so no metric was undefined in this run (the code reports undefined values as null with a count).

Results (held-out 24 queries; all 40 in brackets)

Configurationhit@5recall@5recall@10MRR@10nDCG@10
BM25 (k1 1.2, b 0.75)0.833 (0.875)0.673 (0.689)0.726 (0.744)0.840 (0.850)0.719 (0.723)
TF-IDF cosine0.833 (0.875)0.654 (0.695)0.744 (0.755)0.839 (0.845)0.725 (0.724)
LSA, 256 dimensions (chosen on dev)0.833 (0.900)0.719 (0.755)0.775 (0.807)0.846 (0.862)0.763 (0.774)
RRF(BM25, LSA), k = 600.833 (0.875)0.692 (0.705)0.740 (0.777)0.839 (0.839)0.738 (0.739)
RRF(BM25, TF-IDF, LSA)0.833 (0.875)0.683 (0.700)0.740 (0.777)0.838 (0.852)0.736 (0.741)
Weighted score sum (0.25 BM25 + 0.75 LSA)0.833 (0.900)0.719 (0.746)0.754 (0.794)0.840 (0.865)0.755 (0.768)
Reranker (logistic regression, top 30 of RRF)0.917 (0.925)0.731 (0.740)0.789 (0.807)0.865 (0.865)0.769 (0.759)

The reranker's all-queries column mixes grouped-CV predictions for dev queries with a dev-trained model applied to held-out queries.

Findings

  1. No configuration reaches the recall target, so by the pre-set rule there is no launch. The rule required held-out recall@10 of at least 0.85 (set in advance). The best held-out recall@10 is 0.789 (reranker); LSA has 0.775; BM25 0.726. On all 40 queries the best is 0.807 (LSA and the reranker). The deliverable is the failure analysis below, not a recommendation to ship. For orientation only: the share of queries whose best explanation is in the first five results (hit@5, what the search box shows) is 0.833 for BM25 and 0.833 to 0.917 for the others on held-out, i.e. 20 vs 20 to 22 of 24 queries.
  2. LSA is better than BM25 on nDCG@10, and that is the only robust difference. LSA minus BM25 on nDCG@10: +0.044 held-out (95% bootstrap interval 0.007 to 0.080) and +0.051 on all 40 (0.013 to 0.093). On recall@10 the difference is +0.049 held-out (-0.028 to +0.125) and +0.062 on all 40 (0.000 to 0.142). TF-IDF is indistinguishable from BM25 (+0.006, -0.034 to +0.042). MRR and hit@5 do not separate any configuration from BM25; the reranker's +0.083 hit@5 (two queries of 24; interval 0.000 to 0.208) comes closest.
  3. The hybrids do not beat the best single method. RRF(BM25, LSA) and RRF(BM25, TF-IDF, LSA) land between BM25 and LSA (held-out nDCG@10 0.738 and 0.736); neither is distinguishable from BM25 (+0.019, -0.011 to +0.049). The min-max weighted sum with 0.75 weight on LSA, which is near LSA alone, is +0.036 over BM25 (0.000 to 0.072) on held- out. With a weaker lexical ranker and a stronger latent ranker, equal-weight rank fusion pulls the better list toward the worse one; the weight grid selected the LSA-heavy end (0.25 BM25), which says the same thing.
  4. The reranker is not shown to help. Held-out MRR@10 +0.024 over BM25 (interval -0.014 to +0.076), so the ship criterion (interval excludes zero) is not met; its nDCG@10 gain over BM25 is +0.050 (0.005 to 0.096) held-out but it was trained on 16 queries, and on dev its grouped-CV nDCG@10 (0.745) is below plain LSA (0.790). Its learned weights are dominated by BM25, LSA and title overlap (coefficients 2.5, 2.5, 2.5; TF-IDF 1.0) with the kind indicators small (-0.2 to +0.5). About 9.5% of its top 10 lie outside the union of BM25, TF-IDF and LSA top 10 (the pool the labels came from), so it is the configuration most exposed to unjudged documents.
  5. The dev choice of the LSA dimension is fragile. On the 16 dev queries nDCG@10 rises 0.577, 0.675, 0.766, 0.777, 0.7905 for 16, 32, 64, 128, 256 dimensions and falls to 0.783 at 384, so 256 was chosen. On held-out the same table is 0.592, 0.803, 0.814, 0.811, 0.763, 0.747 and on all 40 queries 0.586, 0.752, 0.795, 0.797, 0.774, 0.761 (diagnostic only; not used for selection). A 16-query dev set picked a dimension that the other 24 queries do not favour; with 64 or 128 dimensions LSA would have scored higher on held-out nDCG@10 (0.81 vs 0.763).
  6. Paraphrase queries are the problem; keyword queries are nearly solved. nDCG@10 on the 28 keyword-style queries is 0.854 to 0.881 for every configuration. On the 12 paraphrase-style queries it is 0.417 (BM25), 0.416 (TF-IDF), 0.538 (LSA), 0.447 (RRF), 0.535 (weighted sum), 0.475 (reranker); hit@5 is 8 of 12 for BM25 and 9 of 12 for LSA. By kind of the best answer (queries whose first relevant document is a glossary term: 7; insight: 33), nDCG@10 is 0.661 vs 0.736 for BM25 and 0.739 vs 0.781 for LSA; no lesson-text query exists.
  7. Chunking did not help; whole documents are as good or better. Documents are short (median 152 words), so a 120-word chunk splits most of them in two. On all 40 queries, nDCG@10 for whole documents vs 240 / 120 / 60-word chunks (scored at document level by max over chunks; the LSA dimension is fixed at 256 for every unit size, not re-tuned): BM25 0.723 / 0.726 / 0.717 / 0.719; LSA 0.774 / 0.786 / 0.751 / 0.740; RRF 0.739 / 0.743 / 0.733 / 0.754. 120-word chunks minus whole documents: BM25 nDCG@10 -0.006 (-0.027 to +0.009), RRF -0.006 (-0.028 to +0.013). Chunk-level ranking also fills the top five passages with fewer distinct documents (BM25 4.2, LSA 3.6 at 120 words; 3.7 and 3.2 at 60 words).
  8. Failure analysis (reranker, the configuration with the best held-out nDCG@10; 15 worst queries, failures.csv). Ten are partial recall: the first hit is fine but other relevant documents (queries have up to five) are missing from the top 10. Two are not in the top 10 at all: q02 ("the automatic grader keeps preferring whichever answer it sees first", relevant insight about judging pairs in both orders; first relevant at rank 97) and q14 ("the assistant made up a booking reference that does not exist", first relevant at rank 12); both are paraphrases with a vocabulary gap (grader vs judge and first vs order for q02; "made up" vs verify-the-identifier / hallucination wording for q14). One has its first relevant document at rank 9, below the five shown, and two are ordering-only. 10 of the 15 are paraphrase-style queries.
  9. Perturbation (ten queries). Corrupting the two longest words of each query (adjacent letters swapped) lowers nDCG@10 for BM25 from 0.825 to 0.744, for LSA from 0.874 to 0.795 and for RRF from 0.857 to 0.790. Reversing word order changes nothing, because every ranker here is bag-of-words.
  10. Latency and cost (indicative). Document-level index, per query: BM25 median 0.26 ms (p95 0.43), TF-IDF 0.80 (1.38), LSA 0.51 (1.63), RRF of BM25 and LSA 1.60 (6.30). Index build 0.06 s for BM25 and TF-IDF, 0.49 s for 256-dimension LSA; memory 0.9 MB sparse weights plus 1.2 MB latent matrix.

What this does not prove

  • Judgments are pooled from these very systems. The labelling note says relevant documents were found by pooling the top 10 of BM25, TF-IDF and LSA (whole documents and 120-word chunks) and reading them; documents outside the pool were never judged and count as non-relevant, and content added after labelling was never judged. Recall is therefore a lower bound for any system that ranks unjudged documents highly, and LSA, which contributed to the pool, may look better than an unjudged-but-good competitor would. The judged-coverage proxy above (fraction of each top 10 inside the union of this run's BM25/TF-IDF/LSA top 10) is 1.0 for BM25, TF-IDF and LSA, 0.98 to 0.99 for the fusions and 0.905 for the reranker; the original pool's own LSA setting is not recorded, so this is only a proxy.
  • Forty builder-authored queries, 24 held-out. The intervals are wide, close configurations can swap order with a different split (see finding 5), and "one query" is 4 points of hit@5. Queries share the author's vocabulary; real learner phrasing (typos, mixed languages) is under-represented.
  • LSA is not an embedding model. Nothing here says how a neural embedding or cross-encoder would behave; it needs downloads and is outside this reference run.
  • No generated answer was produced or judged. Retrieval success is not a grounded answer.
  • The failure categories are token-overlap heuristics, not a human error analysis; the latency numbers come from one process on a shared laptop; the held-out set was used for the decision rule's interval comparisons (as the record specifies), so it is not an untouched final test.
  • The reranker training signal is 16 queries; its held-out result is a lesson about learning to rank on little data.

Reproduce

cd python && uv run python -m reference.r08_retrieval_eval.run
cd python && uv run pytest tests/test_r08_retrieval_eval.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric3 criteria

CriterionMeetsExceeds
Labelled query setQueries mapped to graded ids, split and hashed before experiments, with an overlap check.Explains how builder-authored queries could bias results and adds a perturbation set.
Ablation disciplineEach configuration differs from the previous by one change; paired intervals against BM25 reported.Shows score-fusion vs rank-fusion contrast and the reranker's dev vs held-out gap with an explanation.
Decision and costPicks a configuration using the pre-set rule, with latency and build cost.States what result would change the choice and what to measure after launch.

References8