Statistics, experimentation, causal reasoning · insight-1005

Count markets, not passengers, when price or capacity is randomised at the route level

Situation
A pricing change can only be applied per route, 14 routes are available, and the analyst computes p-values on millions of passengers.
Action
Use route-period as the analysis unit. Pair routes on pre-period level and trend, randomise within pairs, adjust for the pre-period at market level, and print the number of markets beside every interval.
Reason
Assignment happens at the market, and shocks are shared within it, so passengers on one route are not independent evidence.
Boundary
Geo experiments, route pricing and capacity changes, regional campaigns.

Symptom or risk

Intervals are far too narrow because the effective sample is 14 markets, and route-wide shocks such as weather or events pass as effects.

Hundreds of comparable markets, where market-level inference is stable.

Wide intervals and large minimum detectable effects; the cost of honest inference.

Re-randomising routes until the groups look balanced and then analysing as if the assignment were a single draw.

Hypothetical: 14 routes make 7 pairs, so the test can only detect effects large relative to between-route noise, whatever the passenger count.

In geo experiments, non-overlapping geographic regions are randomly assigned to control or treatment and each region receives its condition through geo-targeted ads.
Measuring Ad Effectiveness Using Geo Experiments