LLMs, GenAI, RAG, agents, evaluation · insight-1577
Build the golden set from stratified production traffic, not from questions developers imagine
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Engineers wrote the eval questions themselves, and the assistant scores well offline but users complain about requests nobody anticipated.
Action
Sample real (redacted) requests stratified by intent, channel and difficulty, keep misspellings and multi-part asks, and add developer-written items only for rare, high-stakes cases that traffic does not yet contain.
Reason
An eval estimates performance on the distribution it samples. If the set is not drawn from the deployment distribution, its score is an answer to a different question.
Boundary
Any assistant with logged traffic or a predecessor system to sample from.
Symptom or risk
Developer-written questions are clean, well-formed and close to the documentation wording, so they are easier than real traffic and cover only the intents the authors thought of.
Exceptions
Pre-launch systems have no traffic; use support tickets, search logs or pilot users and relabel once real traffic exists.
Trade-offs
Production samples need privacy review and label effort; developer items are cheaper but biased.
Pitfalls
Sampling only from the happy path: users who gave up or escalated are missing from many logs.