Engineering, production, governance, cost, communication · insight-1302
Keep a small permanent randomised control share to measure what the model earns
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- retail, telecom, saas
Summary: situation, action, reason, boundary
Situation
Six months after launch, finance asks what the retention model earned, and the only evidence is before-and-after revenue (hypothetical).
Action
From launch route a small random share of eligible decisions, such as 2 to 5 percent, to the previous rule or no action. Log assignment, and report the model-versus-control difference with an interval each quarter. Size the share from the effect to detect and the value forgone.
Reason
Random assignment makes the groups comparable, so their difference estimates the effect of the model-driven policy without relying on before-and-after adjustments.
Boundary
Targeting, pricing, prioritisation and recommendation models with repeatable per-customer decisions.
Symptom or risk
Before and after mixes seasonality, price changes and the model's effect, so the value claim is overstated or collapses under challenge and budgets follow.
Exceptions
Where withholding the action is unsafe or unlawful, use a staggered rollout or a quasi-experiment and say so.
Trade-offs
The control share forgoes some value; it is the price of a defensible number.
Pitfalls
Letting control cases leak into the model's training data without flagging them.