Statistics, experimentation, causal reasoning · insight-1012
Evaluate an uplift-targeting rule on a randomised holdout, not by the outcome of the people it selected
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- retail, aviation, telecom, banking
Summary: situation, action, reason, boundary
Situation
A model ranks customers by predicted response to a discount, the top decile converts at 14%, and the slide says the rule "earns" 14%.
Action
Randomly withhold the offer inside the selected group, compare treated with withheld, and report incremental conversions and margin after discount. Plot an uplift curve on the holdout, not on training data.
Reason
The value of targeting is a causal contrast; a high predicted purchase rate is not a high predicted change from the offer.
Boundary
Discount, retention and upsell targeting.
Symptom or risk
That 14% includes customers who would buy anyway; the incremental value may be small or negative after the discount cost.
Exceptions
Offers with zero marginal cost, where any non-negative effect is worth sending.
Trade-offs
Withholding the offer from some likely buyers costs short-term revenue.
Pitfalls
Evaluating on customers whose treatment was assigned by the previous model.
Example
Hypothetical: top decile 14.0% with the offer versus 13.2% withheld is a 0.8-point lift, which a 15% discount can easily erase.