Engineering, production, governance, cost, communication · insight-1270
Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- banking, insurance, e-commerce
Summary: situation, action, reason, boundary
Situation
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
Action
Report each metric by cohort age with the label-maturity curve beside it, compare only cohorts at the same age, and for training either wait past the maturity point or model the delay explicitly.
Reason
A missing outcome for a young cohort is not a negative; treating it as one biases rates downward, and delay models decide whether such a record is negative or should be excluded.
Boundary
Fraud chargebacks, refunds, churn, loan default, claims, conversions with long attribution windows.
Symptom or risk
Recent cohorts always look better than old ones, so performance appears to improve after a release and then 'degrades' as the labels complete.
Exceptions
If the delay is short relative to the reporting period, a fixed lag is enough.
Trade-offs
Waiting for maturity delays feedback; delay models add assumptions that need monitoring.
Pitfalls
Comparing this week at age 3 days with last month at age 45 days.
Example
Hypothetical: if 60% of refunds are filed within 10 days, a cohort at day 10 will show 40% fewer refunds than its final rate even when nothing changed.
Lessons and concepts
Claims and their sources
Chapelle's KDD 2014 paper on delayed feedback in display advertising adds a probabilistic model of the conversion delay so that a not-yet-converted example is treated as negative or excluded depending on elapsed time.