LLMs, GenAI, RAG, agents, evaluation · insight-0615
Score citation recall and citation precision separately
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- public-sector, cross-industry
Summary: situation, action, reason, boundary
Situation
You report a single "citation accuracy" number for a cited-answer system.
Action
On a sampled set, label whether each answer sentence is fully supported by its citations (citation recall) and whether each citation supports its sentence (citation precision); report both.
Reason
Liu, Zhang and Liang define verifiability through these two quantities, and the ALCE benchmark evaluates citation quality as its own dimension next to correctness and fluency.
Boundary
Day 57 document Q&A projects, audit-facing assistants, research tools.
Symptom or risk
A system that cites rarely but correctly and a system that cites everything loosely can receive the same score, hiding opposite failure modes.
Exceptions
Short single-fact answers with one citation, where the two measures collapse into one check.
Trade-offs
Human support labelling is slow; automated entailment checks scale but need their own agreement check against people.
Pitfalls
Rewarding citation count; more citations can lower precision without improving recall.
Lessons and concepts
Claims and their sources
Verifiability can be measured by citation recall (statements fully supported by citations) and citation precision (citations that support their statement).
ALCE evaluates generated text with citations along fluency, correctness and citation quality, and found even the best models lacked complete citation support about half the time on ELI5.