LLMs, GenAI, RAG, agents, evaluation · insight-0514
Score long answers claim by claim instead of with one overall verdict
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A disruption-report generator writes 300-word summaries and reviewers mark each summary acceptable or not.
Action
Split outputs into atomic claims, check each against the source of truth, and report the share of supported claims plus the number of unsupported claims per output.
Reason
FActScore showed that whole-output binary judgments suit long text poorly, because generations mix supported and unsupported facts; atomic-fact precision measures this directly.
Boundary
Summaries, reports, biographies, any answer containing several factual statements.
Symptom or risk
A summary with eight correct facts and one invented rebooking count passes, and the error rate per fact stays unknown.
Exceptions
Short single-fact answers can be scored whole.
Trade-offs
Decomposition costs annotation time or an extra model call, and automated decomposition needs spot checks.
Pitfalls
Measuring only precision; a fully supported summary can still omit the fact that matters.
Lessons and concepts
Claims and their sources
Min et al. argue binary judgments are inadequate for long-form text that mixes supported and unsupported claims, and define FActScore as the share of atomic facts supported by a knowledge source.