LLMs, GenAI, RAG, agents, evaluation · insight-1547
Do not build the retrieval eval set only from questions generated out of the chunks
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
To get 500 labelled queries quickly, an LLM writes a question for each chunk, and that chunk becomes the gold answer.
Action
Mix generated questions with sampled real queries or hand-written ones, tag the source of each, report metrics per source, and measure word overlap between query and gold chunk to see how easy the set is.
Reason
The question is conditioned on the passage, so it is correlated with it more strongly than a user's information need would be; such sets tend to overstate quality.
Boundary
Bootstrapping an eval set for a new corpus before real traffic exists.
Symptom or risk
Generated questions reuse the chunk's own words, so lexical and fine-tuned retrievers look excellent, and the set misses vague, misspelled or multi-chunk questions real users ask.
Exceptions
Regression tests for ingestion, where "does this chunk remain findable" is the question.
Trade-offs
Real queries need logs, privacy review and labelling effort.
Pitfalls
Training or tuning retrieval on the same generated questions and then evaluating on them.