Time series and forecasting · insight-0249
Evaluate an intervention against a counterfactual forecast with control series
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, Data analyst
- Industries
- aviation, retail, telecom
Summary: situation, action, reason, boundary
Situation
A loyalty campaign launched on one market, and leadership wants to know whether bookings rose because of it.
Action
Build a counterfactual forecast of the treated market from pre-period data and unaffected control series (structural time-series or synthetic-control style), then compare post-period actuals with the counterfactual and its interval.
Reason
A counterfactual built from controls that track the treated series before the intervention can estimate what would have happened without it.
Boundary
Marketing launches, policy changes, price changes on one region.
Symptom or risk
Comparing before and after attributes seasonality, trend and market shocks to the campaign.
Exceptions
If controls were also affected by the intervention or a simultaneous shock hit only the treated unit, the estimate is biased.
Trade-offs
Structural models give intervals but depend on the stability of the control relationship.
Pitfalls
Selecting control series after seeing which ones make the effect look biggest.
Lessons and concepts
Claims and their sources
CausalImpact infers intervention effects with Bayesian structural time-series models that use control series to predict the counterfactual.
Synthetic control builds a weighted combination of control units to approximate the treated unit's counterfactual.