LLMs, GenAI, RAG, agents, evaluation · insight-1593
Reweight offline slice scores to the production traffic mix before forecasting online quality
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- aviation, e-commerce
Summary: situation, action, reason, boundary
Situation
The offline score is 85% but production satisfaction looks different, and the golden set over-represents hard or rare cases.
Action
Score by slice, take production request shares from logs, and report the share-weighted score next to the unweighted one; flag slices where offline share and traffic share differ most.
Reason
Overall performance is a weighted average of slice performance, and the weights belong to the deployment, not the eval.
Boundary
Golden sets deliberately enriched with hard or high-risk cases.
Symptom or risk
An unweighted offline mean is an answer for the golden set's mix, so a model can win offline by improving rare slices and still do little for the bulk of users.
Exceptions
For safety slices, the worst-slice score matters regardless of traffic share.
Trade-offs
Weights from logs assume the slice labels are reliable and traffic is stable.
Pitfalls
Reweighting thin slices with a handful of items, which amplifies noise.
Example
Hypothetical: slices score 0.95 (easy) and 0.70 (hard); a 60/40 offline mix gives 0.85, while a 85/15 traffic mix gives 0.9125.