Statistics, experimentation, causal reasoning · insight-1010
Validate a short-term proxy against the long-term outcome in past experiments before deciding with it
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, e-commerce, saas, banking
Summary: situation, action, reason, boundary
Situation
Rebooking within 30 days is used as a proxy for annual customer value and changes ship when the proxy rises.
Action
Collect past randomised tests where both were measured and check that effects on the proxy predict effects on the long-term outcome. Combine several short-term outcomes into one predictive index rather than relying on a single one, and re-validate after product changes.
Reason
The effect on a surrogate index equals the long-term effect only when the long-term outcome is independent of treatment given the surrogates, which fails when treatment works through paths the proxy misses.
Boundary
Outcomes that take months or years to observe: retention, lifetime value, loyalty.
Symptom or risk
The proxy can move, for example through discounts, without the long-term outcome following, so the team optimises a number that does not carry value.
Exceptions
Decisions that are cheap to reverse once long-term data arrive.
Trade-offs
Requires an archive of comparable experiments with long-term follow-up.
Pitfalls
Validating by correlation across customers instead of across randomised treatment effects.
Lessons and concepts
Claims and their sources
The treatment effect on a surrogate index equals the effect on the long-term outcome if the long-term outcome is independent of treatment conditional on the surrogates, and the authors characterise the bias when this fails.