Engineering, production, governance, cost, communication · insight-1265
Keep shadow deployments free of side effects, quotas and shared state
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- banking, e-commerce, cross-industry
Summary: situation, action, reason, boundary
Situation
A candidate model is deployed in shadow mode and receives a copy of production requests, but it shares the production feature store, rate limits and an outbound notification client.
Action
Give the shadow its own write targets and flags, stub or drop external actions, mark logs with the model version, cap its share of dependency calls, and prove with a test that it cannot trigger an action.
Reason
Shadow evaluation is only safe if responses are discarded and the candidate's work does not change the system it observes.
Boundary
Shadow, mirrored and replay deployments of decision services.
Symptom or risk
The shadow doubles load on shared dependencies, writes rows or sends messages as if it were live, and pollutes the feature cache and the logs used for evaluation.
Exceptions
Read-only models over a replicated snapshot need only the load cap.
Trade-offs
Isolation costs extra infrastructure and can hide interaction effects that exist only on a live path.
Pitfalls
Mirroring requests that contain idempotency keys or tokens, so the real downstream rejects the live request as a duplicate.