LLMs, GenAI, RAG, agents, evaluation · insight-1635
Reserve a small random slice of routed traffic for the other model so you can measure what routing hides
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer, Applied AI
- Industries
- e-commerce, saas
Summary: situation, action, reason, boundary
Situation
A router sends easy requests to a small model and the team reports quality from the traffic the router chose.
Action
Randomly send about 1 to 3 percent of requests the router would send cheap to the strong model as well (shadow, not shown), score both with the same checks or judge, and estimate the cheap path's quality gap on the population it actually serves.
Reason
A model that influences its own data selection biases what you observe, and Sculley et al. note that some randomisation mitigates such direct feedback loops; routers choose between a strong and a weak model per query, as RouteLLM does.
Boundary
Cascades and learned routers in production with a scorable quality signal.
Symptom or risk
Quality on the cheap path looks excellent because the router only sends it easy cases, and drift in the router goes unnoticed.
Exceptions
Very low volume features, where a periodic offline replay of logged traffic on both models is cheaper.
Trade-offs
The slice costs strong-model calls and extra evaluation; keep it small and fixed.
Pitfalls
Choosing the slice by a rule correlated with difficulty instead of true randomisation.
Lessons and concepts
Claims and their sources
Sculley et al. say a model can directly influence the selection of its own future training data, and that using some amount of randomisation, or isolating parts of the data, mitigates the effect.
RouteLLM trains a router that chooses between a stronger and a weaker model per query and evaluates it as a cost-quality curve.