LLMs, GenAI, RAG, agents, evaluation · insight-0620
Measure faithfulness and correctness separately, because faithful answers can be wrong
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
Your RAG evaluation uses a faithfulness score (is every claim supported by the retrieved context?) as the main quality metric.
Action
Track faithfulness to context and correctness against a reference answer as two metrics, and investigate questions that are faithful but incorrect as retrieval or corpus problems.
Reason
Faithfulness, as defined in RAGAS, checks support by the provided context, not truth in the world; it cannot detect stale or wrong source documents.
Boundary
Corpora that change over time: prices, schedules, policies, regulations.
Symptom or risk
An answer faithfully restates a superseded fee table and scores perfectly, while the customer is quoted the wrong price.
Exceptions
Closed, curated corpora that are correct by construction; there, faithfulness is a reasonable proxy.
Trade-offs
Correctness needs reference answers that must be refreshed when documents change.
Pitfalls
Reporting high faithfulness as "no hallucinations" to stakeholders.
Lessons and concepts
Claims and their sources
RAGAS defines reference-free metrics including faithfulness (claims in the answer supported by the retrieved context), answer relevance and context relevance.