LLMs, GenAI, RAG, agents, evaluation · insight-0519
Seed eval sets with questions that invite common misconceptions
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist, Data analyst
- Industries
- aviation, finance
Summary: situation, action, reason, boundary
Situation
A revenue-management help bot is evaluated only on questions written by the people who built it.
Action
Ask domain experts to write items whose most common human answer is wrong, drawn from real misunderstandings, and score truthfulness separately from helpfulness.
Reason
TruthfulQA found models produced many answers that mimic popular misconceptions, and the largest models tested were generally the least truthful.
Boundary
Domains with widespread folk beliefs, such as finance, health, statistics and aviation operations.
Symptom or risk
Questions with a popular but wrong answer, such as treating a high load factor as proof that pricing worked, are missing, so the bot's tendency to repeat folk beliefs is never measured.
Exceptions
Purely extractive tasks over supplied documents are less exposed.
Trade-offs
Expert-written items are slow to produce.
Pitfalls
Assuming a newer or larger model has fixed this without measuring.
Lessons and concepts
Claims and their sources
Lin et al. found that models generated many false answers that mimic popular misconceptions and that the largest models in their study were generally the least truthful.