LLMs, GenAI, RAG, agents, evaluation · insight-0621
Label the evidence passages in your golden set, not only the expected answers
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You are building the first evaluation set for a document assistant from real user questions.
Action
For each question store the expected answer, the ids or text spans of the evidence that supports it, the question type, and whether it is answerable from the corpus.
Reason
Evidence labels let you compute retrieval metrics cheaply and repeatedly, and they make the retrieval-versus-answer diagnosis possible.
Boundary
Any RAG system before tuning (day 43's eval-set-first rule applied to retrieval).
Symptom or risk
With only reference answers you can score outputs but cannot tell whether a failure came from retrieval or generation, and every retriever change needs a full re-judging.
Exceptions
Pure generation tasks with no corpus.
Trade-offs
Evidence labelling takes longer per item, so start with 100 to 200 stratified questions rather than thousands of unlabelled ones.
Pitfalls
Labelling by chunk id: re-chunking breaks the labels; label document spans that can be mapped onto any chunking.