Statistics, experimentation, causal reasoning · insight-1031
Define outcome metrics on matured cohorts when cancellations, refunds or no-shows arrive late
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- aviation, e-commerce, insurance, banking
Summary: situation, action, reason, boundary
Situation
A booking-flow test reports net bookings after two weeks, but cancellations and refunds keep arriving for up to a month.
Action
Estimate the delay curve from past cohorts, define the decision metric on bookings after the maturation period, or model expected final outcome from cohort age, and report which version each number is.
Reason
Outcomes recorded at different ages are not comparable, and arms that differ in mix of early or late events diverge in their recorded values only because of timing.
Boundary
Cancellations, refunds, chargebacks, claims and returns.
Symptom or risk
The arm that sells more at first looks better, then loses the advantage as its cancellations land, and the decision uses an immature number.
Exceptions
Metrics that are final at observation time.
Trade-offs
Waiting for maturation slows the decision; a modelled estimate adds assumptions.
Pitfalls
Comparing a recent cohort's net revenue with an older cohort's mature value.