LLMs, GenAI, RAG, agents, evaluation · insight-0574
Report quality and coverage separately when evaluating generated samples
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- retail, manufacturing
Summary: situation, action, reason, boundary
Situation
Two image generators are compared by a single FID score, and the lower score wins.
Action
Report precision (share of samples that look like real data) and recall (share of real variety the generator covers) alongside FID, compute every metric with the same sample count and feature extractor, and add task-based checks.
Reason
FID compares feature statistics of real and generated sets in one value, while precision and recall separate sample quality from coverage.
Boundary
Image and other generative model evaluation, synthetic data acceptance.
Symptom or risk
The winner may produce excellent samples of only a few kinds (missing modes), or varied samples of poor quality, and the single number cannot tell which.
Exceptions
For text generation, use task evals, human ratings and diversity measures instead.
Trade-offs
More metrics to explain to stakeholders.
Pitfalls
Comparing FID values computed with different sample sizes or implementations.
Lessons and concepts
Claims and their sources
Heusel et al. introduce the Fréchet Inception Distance, which compares statistics of real and generated images in Inception feature space.
Kynkäänniemi et al. measure sample quality (precision) and coverage (recall) separately, which single-number metrics such as FID conflate.