LLMs, GenAI, RAG, agents, evaluation · insight-0517

Give the model a scored way to abstain, or your eval rewards confident guessing

Situation
An eval for a policy assistant scores answers only as right or wrong, so a reply of insufficient information counts as wrong.
Action
Add unanswerable items, allow an explicit insufficient-information reply, and score three outcomes (correct, abstained, wrong), penalising wrong answers more than abstentions in line with business cost.
Reason
Recent analysis argues that grading which penalises uncertainty rewards guessing, which helps explain why hallucinations persist.
Boundary
Policy, compliance, finance, health and customer-facing Q&A.

Symptom or risk

Prompts and models that guess more win the comparison, and in production the assistant invents policy when evidence is missing.

Tasks that must always return an answer, such as ranking, need calibrated confidence instead.

Too high a penalty yields an assistant that abstains on answerable questions; track abstention on answerable items.

Building the eval set only from answerable questions.

Kalai et al. (2025 preprint, evolving research) argue that hallucinations persist because training and evaluation reward guessing over acknowledging uncertainty, and propose scoring benchmarks so abstaining is not penalised.
Why Language Models Hallucinate