Statistics, experimentation, causal reasoning · insight-1004
Leave a washout gap between switchback periods when a pricing or ranking change lingers
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- aviation, logistics, e-commerce
Summary: situation, action, reason, boundary
Situation
A fare-ranking algorithm is global, so the team alternates it on and off by hour and compares the two states.
Action
Estimate how long the change persists (typical shopping-to-booking lag), make blocks longer than that or drop a washout interval after each switch, randomise the order within day-of-week blocks, and analyse by block rather than by session.
Reason
A switchback gives each unit both conditions over time, so it identifies an effect only if earlier treatment does not affect later outcomes; optimal designs depend on how long carryover lasts.
Boundary
Global algorithms, market-level prices and any change that cannot be split by user.
Symptom or risk
Customers who saw one state still book in the next, so the effect leaks across periods, biasing the contrast and inflating variance.
Exceptions
Changes with no memory, such as a purely instantaneous display rule.
Trade-offs
Washout periods and long blocks cut the number of usable blocks and slow the test.
Pitfalls
Five-minute blocks for customers who compare fares across several days.
Example
Hypothetical: shoppers typically book 3 days after first search, so use blocks of a week or more and discard the first 3 days after each switch.
Lessons and concepts
Claims and their sources
In switchback experiments the same unit is repeatedly exposed to randomly assigned treatments over time, carryover is the lingering impact of a treatment on later outcomes, and the optimal design depends on how long carryover lasts.