LLMs, GenAI, RAG, agents, evaluation · insight-0515

Use disagreement across samples as a hallucination flag when no reference exists

Situation
An assistant drafts airport transit guidance for which no structured source of truth exists yet.
Action
Sample several responses to the same prompt and flag sentences that the other samples do not support, so reviewers check those first or the system removes them.
Reason
SelfCheckGPT showed that facts a model knows tend to recur across samples while hallucinated facts diverge, giving a black-box signal without external data.
Boundary
Black-box APIs, cold-start domains, prioritising human review.

Symptom or risk

Invented details look exactly like correct ones, and reviewing every answer by hand is unaffordable.

Where a source of truth exists, verify against it instead.

Several extra generations per answer.

Treating agreement as truth; widely repeated misconceptions are consistent across samples.

Manakul et al. found that sampled responses agree when a model knows a fact and diverge or contradict each other for hallucinated facts.
SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models