LLMs, GenAI, RAG, agents, evaluation · insight-1576
Tie every golden item to its source document and effective date, then re-verify on a schedule
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- aviation, insurance
Summary: situation, action, reason, boundary
Situation
A golden set of policy or product questions was written months ago and the underlying rules, prices or documents have since been revised.
Action
Store with each item the source document id, its version and effective date; re-check items against the system of record when the source changes or every quarter, and retire or rewrite the ones that no longer match.
Reason
A gold answer is a claim about the world at a point in time. Without provenance nobody can tell a model regression from a gold label that expired.
Boundary
Assistants answering from policies, tariffs, regulations or catalogues that change.
Symptom or risk
The set keeps rewarding answers that were right at authoring time, so a correct, up-to-date assistant is marked wrong and a stale one passes.
Exceptions
Timeless tasks such as arithmetic or grammar correction need no refresh schedule.
Trade-offs
Re-verification is review effort; link items to documents so only the affected ones are reopened.
Pitfalls
Updating gold answers silently, which breaks comparison with earlier runs; keep a versioned set and note the version in every report.
Example
Hypothetical: an item's gold answer says a 23 kg checked-bag allowance; the fare rule was revised to 20 kg last month. The item now fails correct answers.