Concept map: Retrieval metrics

Taught and used in

RAG and retrieval, Validation and metrics

Insights 20

• Compare chunk sizes at the same context token budget, not the same k
• Fuse lexical and vector results by rank, not by adding raw scores
• Choose the embedding model on your own labelled queries, not a public leaderboard
• Retrieve wide with a cheap retriever, then rerank narrow with a cross-encoder
• Stop adding retrieved passages once answer quality stops rising
• Report recall@k at the k you actually pass to the model
• Use MRR when one right passage exists and how high it ranks matters
• Use nDCG when some retrieved passages are partly relevant

Projects 10

• Flagship: passenger-rights document Q&A with a retrieval baseline, one controlled improvement and an honest generation mode
• Hybrid lexical/latent retrieval and lightweight reranking ablation over the academy's own knowledge base
• Chunking experiment on long technical papers: evidence recall at a fixed context budget
• LLM-as-judge reliability study: agreement with human pairwise labels, position bias and length bias
• Cost and latency routing study: small model, large model, router and cascade on checkable answers
• Summary faithfulness evaluation for legislative bills: human claim labels vs automatic checks