LLMs, GenAI, RAG, agents, evaluation · insight-0618

Use nDCG when some retrieved passages are partly relevant

Situation
Labellers say some passages fully answer the question, some are related background, and some are outdated versions of the right rule.
Action
Label graded relevance (for example 2 = answers, 1 = helpful, 0 = not relevant) and report nDCG@k, which discounts gains by rank and normalises by the ideal ordering.
Reason
Järvelin and Kekäläinen designed cumulated-gain measures for graded relevance, so a ranking that puts fully relevant passages first scores higher than one that leads with marginal ones.
Boundary
Search over policies with versions, product search, multi-aspect questions.

Symptom or risk

Binary relevance forces a choice, so either outdated passages count as hits or useful background counts as misses.

When every question has exactly one relevant passage, nDCG adds little over MRR.

Graded labels take longer and need a written grading guide to stay consistent between labellers.

Comparing nDCG values computed with different gain scales or cutoffs.

Discounted cumulated gain and its normalised form were defined to evaluate rankings under graded relevance.
Cumulated gain-based evaluation of IR techniques