LLMs, GenAI, RAG, agents, evaluation · insight-1595
Shadow a candidate on mirrored production requests before it serves any user
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Applied AI
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A candidate prompt or model beats the baseline offline, but offline items differ from live traffic in length, language mix and context.
Action
Send a copy of live requests to the candidate, discard its output to users, log latency, cost, errors and refusals, and have a judge or reviewers compare a sample of candidate and live answers side by side.
Reason
Shadow traffic has the true input distribution and reveals operational problems with no user-visible risk.
Boundary
Read-only assistants and answer generation.
Symptom or risk
Offline wins fail to transfer, and discovering that through user exposure costs trust.
Exceptions
Do not mirror requests that trigger side effects such as bookings or emails, and honour data-retention limits.
Trade-offs
Double inference cost during the shadow period, and no signal on user reactions.
Pitfalls
Comparing the shadow's output to the live system's as if it were gold; both can be wrong.