LLMs, GenAI, RAG, agents, evaluation · insight-0619
Locate RAG failures with a retrieval-by-answer two-way table
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
Action
For each golden question record two labels, evidence retrieved in the passed context (yes or no) and answer correct and supported (yes or no), and count the four cells.
Reason
Retrieval success and a grounded answer are different events. Evidence missing points to ingestion, chunking or ranking; evidence present but answer wrong points to context assembly or generation.
Boundary
Every RAG debugging session; the core move of days 48 and 51.
Symptom or risk
Teams tune the wrong stage because one end-to-end accuracy number mixes retrieval misses with generation errors.
Exceptions
Questions whose answer needs no retrieval; tag them and keep them out of the table.
Trade-offs
Needs evidence labels, not just reference answers, which costs labelling time up front.
Pitfalls
Ignoring the "evidence missing, answer correct" cell; it often means the model answered from memory, which will fail on fresher facts.