Statistics, experimentation, causal reasoning · insight-1020
Test whether a difference-in-differences result survives a change of scale before reporting it
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- aviation, retail, public-sector
Summary: situation, action, reason, boundary
Situation
A difference-in-differences on passenger counts says treated routes grew less than controls, while the same data in logs says the opposite.
Action
Choose the scale from the decision (absolute passengers or percentage change), argue parallel trends on that scale, rerun on the alternative, and report both. If they disagree, the design is not credible without an explicit argument for one scale.
Reason
Parallel trends holds under every monotone transformation of the outcome only under a stronger condition, which in practice requires effectively random treatment in a subgroup and a stable remainder.
Boundary
Difference-in-differences with positive outcomes such as counts, revenue and prices.
Symptom or risk
Parallel trends in levels and in logs are different assumptions, so sign and size depend on an arbitrary scale choice.
Exceptions
Treated and control units with similar baseline levels, where scales agree.
Trade-offs
Reporting two answers is less tidy than one.
Pitfalls
Taking logs "to be safe" and reading the percentage effect as the level effect.
Lessons and concepts
Claims and their sources
Parallel trends holds under every strictly monotonic transformation of the outcome only if a stronger condition holds for the distributions of untreated potential outcomes, which in practice holds essentially only when treatment is effectively random in a subgroup and the remainder is stable; falsification tests are proposed.